{section: Read the Docs Overview} In 2019 we migrated the HTCondor Manual from our old LaTeX format into the Sphinx documentation generator using the reStructuredText format. We will also be hosting it on Read the Docs. The manual can be found here: http://htcondor.readthedocs.io {section: Prerequisites} 1: Install Python. Any version is fine. *::https://www.python.org/downloads 1: Install Sphinx and related support packages. The recommended way to install Sphinx is via pip: {code} sudo pip install sphinx sphinx_rtd_theme nbsphinx {endcode} On RHEL and CentOS *we do not support* installing Sphinx from yum or RPM. The available package is an old version which does not support all our extensions. For other Linux distributions, Windows and macOS, you can find instructions here: *::https://www.sphinx-doc.org/en/master/usage/installation.html 1: Install the HTCondor Python bindings. This could be via =pip= (make sure you get the right version, e.g. =pip install htcondor==8.8.1=), or by editing your =PYTHONPATH= environment variable to point at the =.so= files inside your HTCondor install. Note: on RTD itself, the docs are built using the =htcondor= version read out of =CMakeLists.txt= 1: Install the Python pathlib module (if you aren't on Python >= 3.4): {code} sudo pip install pathlib {endcode} 1: [Optional] Install https://pypi.org/project/sphinx-autobuild/. This is useful if you're making lots of quick edits to the manual and expect to build it many times. {code} sudo pip install sphinx-autobuild {endcode} See below for instructions on using =sphinx-autobuild=. {section: Building} {subsection: Building and previewing the manual locally} *:: The manual pages exist in our git repository under the =/docs= folder *:: Each top-level section of the manual has a corresponding subdirectory with the same name. For example, all the content for "Overview" is under =/docs/overview=, the content for "User's Manual" is under =/docs/users-manual=, and so on. *:: Each page of the manual has a corresponding file with the same name and a .rst extension. For example, the content for the "Overview > Exceptional Features" page is in =/docs/overview/exceptional-features.rst= *:: To make a local build of the manual, go to your =/docs= folder and run: {code} make html {endcode} *:: To preview your build, open a web browser and go to the following URL: =file:////docs/_build/html/index.html= *:: To clean your local build, go to your =/docs= folder and run: {code} make clean {endcode} {subsection: Using sphinx-autobuild to build and preview} Instead of the instructions in the previous section, go to =/docs= and run {code} sphinx-autobuild . _build/html {endcode} You will see a log of the Sphinx build running, and eventually will be provided a link to a =localhost= webserver hosting the docs. Leave this program running: =sphinx-autobuild= will watch the docs source tree for changes and rebuild when it detects changes (you will still need to manually reload the page inside your web browser). {subsection: Different versions of the manual} *:: Read the Docs allows us to host multiple versions of the manual. We'll have two separate versions: "latest" (equivalent to Development Release) and "stable" (equivalent to Stable Release). *:: To make edits to the latest version, make your changes on the *master* branch. *:: To make edits to the stable version, make your changes to the *V8_8-branch* (or the appropriate *V8_8_X-branch* if making your changes after code freeze) {section: Editing} The manual now uses the reStructuredText (rST) format, which is similar to Markdown markup but considerably more powerful. A helpful reference to reStructuredText is available here: http://docutils.sourceforge.net/docs/user/rst/quickref.html In addition, the new manual also uses the Sphinx documentation generator. Sphinx does many useful things such as: *:: Converts reStructuredText files into HTML websites, PDF, EPUB and man pages. *:: Extends rST to provide more complex inline widgets such as tables of contents, syntax-colored code blocks, internal links and references, *:: Automatic indices, search and language-specific module indices *:: Provides a powerful API for writing custom extensions A full Sphinx reference is available here: http://www.sphinx-doc.org/en/master/contents.html This section provides some markup style guidelines, as well as information about how we use both built-in and custom tools. {subsection: Section Titles} Section titles are very fluid in rST and there are many different ways to make them. To keep things as consistent as possible, try to use the following: {code} Page titles get underlined with the = symbol ============================================ Section titles get underlined with the - symbol ----------------------------------------------- Subsection titles get underlined with the ' symbol '''''''''''''''''''''''''''''''''''''''''''''''''' {endcode} {subsection: Indentation} The rST format is very sensitive to indentation. Paragraphs and other blocks of text are expected to be left-aligned. Indenting a block by any amount of whitespace (compared to the preceding block) causes it to get indented. {code} This is a top-level block of text. It will appear aligned to the left-most side of the page. This paragraph is indented by one space. Even though it's only a single space, it will render as a full first-level indent. This paragraph is indented by one more space than the one above it. As a result it will render as a second-level indent. This time I've indented a block by two more spaces the one above it. It doesn't matter that this is inconsistent with the single-space indents above. This block will render as a third-level indent. Back to the top level! This block is indented by 12 spaces. However, as with the previous examples, the amount of whitespace doesn't matter. Because it's the first indented block compared to the preceding block, it will only render as a first-level indent. {endcode} {subsection: Linking to gittrac tickets} Use the following syntax to automatically link to a gittrac ticket, where #### is the number of the ticket: {code} :ticket:`####` {endcode} {subsection: Adding index entries} To add a basic index entry, use the following syntax: {code} :index:`Name of index entry` {endcode} If you want your index entry to appear under a parent entry, the syntax is a little more complicated: {code} :index:`Name of index entry ` {endcode} {subsection: Linking to internal documents} To add a link to an internal document, the syntax looks like =:doc:`/path/to/page-title`=. For example, to link to the Overview > Exceptional Features section, add the following: {code} :doc:`/overview/exceptional-features` {endcode} By default, the link text will be the name of the page. If you want to add custom text, it looks something like the following: {code} :doc:`Here is my custom text link ` {endcode} {subsection: Documenting Python Objects} Python "objects" (classes, methods, free functions, enums, anything) are documented via =sphinx-autodoc= (https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html). "Docstrings" for these objects are written directly into the Python bindings C++ source code, are embedded into the Python library during the HTCondor build process, and are then read by Sphinx during the manual build. As an example, a method declaration (adding a method to a class) looks like this in the C++ source code: {code} .def("queue_with_itemdata", &Submit::queue_from_iter, R"C0ND0R( Submit the current object to a remote queue. :param txn: An active transaction object (see :meth:`Schedd.transaction`). :type txn: :class:`Transaction` :param int count: A queue count for each item from the iterator, defaults to 1. :param from: an iterator of strings or dictionaries containing the itemdata for each job as in ``queue in`` or ``queue from``. :return: a :class:`SubmitResult`, containing the cluster ID, cluster ClassAd and range of Job ids Cluster ID of the submitted job(s). :rtype: :class:`SubmitResult` :raises RuntimeError: if the submission fails. )C0ND0R", (boost::python::arg("self"), boost::python::arg("txn"), boost::python::arg("count")=1, boost::python::arg("itemdata")=boost::python::object()) ) {endcode} Note the use of a raw string delimited by =C0ND0R= for the docstring itself. The syntax is described here: https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html The corresponding =.rst= source that would embed this docstring, prettily-formatted, in the manual looks like {code} .. autoclass:: Submit .. automethod:: queue .. automethod:: queue_with_itemdata .. automethod:: expand .. automethod:: jobs .. automethod:: procs .. automethod:: itemdata .. automethod:: getQArgs .. automethod:: setQArgs .. automethod:: from_dag {endcode} The =.. auto::= are Sphinx directives provided by =sphinx-autodoc= which are replaced by the autodoc-formatted descriptions. So this block produces all of the documentation for the =Submit= object. When documenting a new thing in the bindings, you must also add an appropriate =.. auto::= in the appropriate =.rst=. {section: Publishing} {subsection: Publishing the manual onto Read the Docs} *:: Currently Mark or Josh has to login to Read the Docs to manually push any changes. {subsection: Different versions of the manual} *:: To make changes to the devel/latest version of the manual, put all your changes in the master branch. *:: To make changes to the stable version of the manual, you need to add a tag called "stable" to the commit in V8_8-branch that contains the changes you want. To update the stable tag to a new commit, run the following: {code} git tag --delete stable git push --delete origin tag stable git tag stable git push origin stable {endcode} {subsection: Generating man pages} *:: The files in =/docs/man-pages= will be the official source for our man pages going forward. *:: Sphinx can generate the man pages automatically. They will get output by default to the =/docs/_build/man= folder. From the =/docs= folder, run: {code} make man {endcode}