It is currently Mon Feb 24, 2020 6:01 pm


All times are UTC




Post new topic Reply to topic  [ 16 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Documentation viewer and generator
PostPosted: Thu Jul 01, 2010 1:52 pm 
Developer
User avatar

Joined: Sun May 11, 2008 4:29 pm
Posts: 198
Location: UK
Qt Help

I've just checked in support for generating Qt Assistant compatible help files to PolyVox. Up until now, PolyVox required Doxygen to create the API docs. Now, as of the version in SVN, it will also require the qcollectiongenerator tool which comes with Qt Assistant. This tool takes the HTML output of Doxygen and compiles it into a file which can then be opened with assistant. The file is output as <build_dir>/library/doc/html/polyvox.qhc and can be opened like
Code:
assistant -collectionFile polyvox.qhc

This allows fast searching of PolyVox documentation in a shell showing nothing but the PolyVox API docs:
Attachment:
File comment: PolyVox API docs showing in a Qt Assistant shell.
polyvox_doc_assistant.png
polyvox_doc_assistant.png [ 115.62 KiB | Viewed 6021 times ]

If either qcollectiongenerator or Doxygen are not available, the doc build target will simply not be available. I am planning on adding the ability to fall back to plain Doxygen docs if qcollectiongenerator is not found.

Sphinx

Further to this and in a similar vein I've also been looking into a decent tool for generating end-user documentation (a manual) for Thermite. Dave, remember we spoke before about what tool would be best and I suggested DocBook and others but in the mean time you moved on and started writing something up in a word-processor. Well, since then I've come across Sphinx. Sphinx is the tool (written in Python) created for generating the new Python documentation but has since grown to be a much more general purpose tool for documentation and manuals. It uses the reStructuredText syntax for markup (you may have seen me using this for PolyVox's INSTALL.txt) which keeps the files readable as plain text while being easily convertible to other formats. Sphinx also provides a number of output formats including HTML, QtHelp and LaTeX.

I'm only mentioning this as a matter of interest. There's no need to convert everything now and if you prefer any other format then that's cool. I've done a test conversion of the current docs to Sphinx and it was very easy so I can do it again at any time in the future. The downside to using Sphinx is that it introduces a build dependency on Python but this shouldn't matter for the Thermite SDK since we will build the docs ourselves and bundle them up.

Just putting down my thoughts etc. on what I've been working on the last few evenings.

_________________
Matt Williams
Linux/CMake guy


Last edited by milliams on Thu Jul 01, 2010 2:02 pm, edited 1 time in total.
Smaller screenshot


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Thu Jul 01, 2010 8:58 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
Hey Matt, that's cool stuff!

milliams wrote:
Qt Help

I've just checked in support for generating Qt Assistant compatible help files to PolyVox. Up until now, PolyVox required Doxygen to create the API docs. Now, as of the version in SVN, it will also require the qcollectiongenerator tool which comes with Qt Assistant. This tool takes the HTML output of Doxygen and compiles it into a file which can then be opened with assistant.


Interesting, so is this the same process as is used by Nokia to produce the Qt documentation? I always assumed they has some custom tool to handle Qt specific things such as signals/slots. But they are really using Doxygen as a backend?

milliams wrote:
If either qcollectiongenerator or Doxygen are not available, the doc build target will simply not be available. I am planning on adding the ability to fall back to plain Doxygen docs if qcollectiongenerator is not found.


Unfortunalty this is what happens to me :-( I just reverted back a few revisions, and it seems that I do get the 'doc' target for revision 1112 but not 1113. Of course, I can try to investigate myself why this happens.

But yes, I like it. Though I do think it would be good to fall back on just Doxygen if necessary, because Qt is quite heavy whereas Doxygen is quite standard for this.

I guess we are also permitted to to ship a copy of Qt Assistant with the PolyVox SDK, if we wanted to?

milliams wrote:
size=150]Sphinx[/size]

Further to this and in a similar vein I've also been looking into a decent tool for generating end-user documentation (a manual) for Thermite. Dave, remember we spoke before about what tool would be best and I suggested DocBook and others but in the mean time you moved on and started writing something up in a word-processor. Well, since then I've come across Sphinx. Sphinx is the tool (written in Python) created for generating the new Python documentation but has since grown to be a much more general purpose tool for documentation and manuals. It uses the reStructuredText syntax for markup (you may have seen me using this for PolyVox's INSTALL.txt) which keeps the files readable as plain text while being easily convertible to other formats. Sphinx also provides a number of output formats including HTML, QtHelp and LaTeX.


It looks good, and I think a tool like this is appropriate for the end user documentation. Steve Streeting also had an interesting blog post a few months ago about the documentation tool chain for Ogre:

http://www.stevestreeting.com/2010/03/0 ... #more-2592

It's a bit heavy for what we want but the asciidoc (http://www.methods.co.nz/asciidoc/) tool looks very similar to Sphinx. I have no preference between them at all as I haven't done any research.

milliams wrote:
I'm only mentioning this as a matter of interest. There's no need to convert everything now and if you prefer any other format then that's cool. I've done a test conversion of the current docs to Sphinx and it was very easy so I can do it again at any time in the future. The downside to using Sphinx is that it introduces a build dependency on Python but this shouldn't matter for the Thermite SDK since we will build the docs ourselves and bundle them up.


There is also the option of doing the end user documentation using Doxygen? I know you already added some basic stuff here like the skeleton of a tutorial. How does this then integrate with Qt Assistant? How do Nokia handle their end user documentation (tutorials, examples, etc)?

milliams wrote:
Just putting down my thoughts etc. on what I've been working on the last few evenings.


Yep, that's great :-) Unfortunatly it's not so good from a timing point of view as I have Celine coming for the summer and am unlikely to have much time for PolyVox/Thermite. But I'll work on it when I can :-)


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Fri Jul 02, 2010 12:47 pm 
Developer
User avatar

Joined: Sun May 11, 2008 4:29 pm
Posts: 198
Location: UK
David Williams wrote:
Interesting, so is this the same process as is used by Nokia to produce the Qt documentation? I always assumed they has some custom tool to handle Qt specific things such as signals/slots. But they are really using Doxygen as a backend?
Well, this is really a two stage process. The first is some tool (Doxygen in our case) which scans the source code and creates a bunch of HTML files. The second is a tool which takes all those HTML files (regardless of where they came from) and bundles them up to be displayed in Qt Assistant.

The second step is the same as what Nokia do (this qcollectiongenerator tool is the result of them open-sourcing one of their internal tools I believe). The first step however, is probably different. They have some sort of secret internal Doxygen-like tool (in fact Doxygen was created as a reverse engineered clone of this tool -- see answer 19 at Doxygen's FAQ) which they use to create the HTML API docs. It is this secret tool which handles the signals and slots.

In addition to the auto-generated API docs, Nokia also create a lot of other HTML files (either by hand or with some other tool) for things like tutorials which they then bundle together into Assistant.

David Williams wrote:
milliams wrote:
If either qcollectiongenerator or Doxygen are not available, the doc build target will simply not be available. I am planning on adding the ability to fall back to plain Doxygen docs if qcollectiongenerator is not found.
Unfortunalty this is what happens to me :-( I just reverted back a few revisions, and it seems that I do get the 'doc' target for revision 1112 but not 1113. Of course, I can try to investigate myself why this happens.

But yes, I like it. Though I do think it would be good to fall back on just Doxygen if necessary, because Qt is quite heavy whereas Doxygen is quite standard for this.
Ok, I've just checked in some code (r1118) which provides this fall-back. Doxygen docs will always be created but if qcollectiongenerator is available then they will also be bundled up into a Qt Help Collection.

David Williams wrote:
I guess we are also permitted to to ship a copy of Qt Assistant with the PolyVox SDK, if we wanted to?
Yes, I believe we can. It's LGPL3 but that doesn't affect the licence of PolyVox itself. The way it would work is that in the SDK we would include the assistant.exe executable, the required DLLs and plugins (only a few), polyvox.qch and polyvox.qhc. To make it run we should probably include a Windows shortcut file (or whatever they are) to run the assistant binary with the correct arguments.

David Williams wrote:
milliams wrote:
<snip> ...some stuff about Sphinx
It looks good, and I think a tool like this is appropriate for the end user documentation. Steve Streeting also had an interesting blog post a few months ago about the documentation tool chain for Ogre:

http://www.stevestreeting.com/2010/03/0 ... #more-2592

It's a bit heavy for what we want but the asciidoc (http://www.methods.co.nz/asciidoc/) tool looks very similar to Sphinx. I have no preference between them at all as I haven't done any research.
Well I like Sphinx because it provides all the outputs we need including Qt Help. Plus the syntax makes it possible to read the source files as if they were pretty much plain text (though I guess asciidoc is like that too). From what I can tell Sphinx is a bit more lightweight and it's working. If I have time, I'll play around with creating a simple theme for it.

David Williams wrote:
There is also the option of doing the end user documentation using Doxygen? I know you already added some basic stuff here like the skeleton of a tutorial. How does this then integrate with Qt Assistant? How do Nokia handle their end user documentation (tutorials, examples, etc)?
It would probably be possible to create a the manual in Doxygen but it's not really what it's designed for. It's quite common (e.g. OGRE do it) to use a different tool for API docs and the manual. Each tool has it's own strengths and so it makes sense to work with those. It should even be possible to get Sphinx to link to the Doxygen docs automatically etc. (I'd have to write this myself but it's Python so wouldn't be too hard).

The way Nokia do it (at least this is what I guess they do) is to use their Doxygen-like tool to create just the API docs and create all the rest of the pages using some sort of tool which pumps out HTML files. The two tools will have some sort of weak linking (i.e. the manual pages can link to the class definitions easily) but most of the cohesiveness comes from consistent theming. They then stick them all in the same directory. Of course it is possible they use one tool for doing all of that but it's almost certainly not as nice a tool as Sphinx (or whatever).

David Williams wrote:
milliams wrote:
Just putting down my thoughts etc. on what I've been working on the last few evenings.
Yep, that's great :-) Unfortunatly it's not so good from a timing point of view as I have Celine coming for the summer and am unlikely to have much time for PolyVox/Thermite. But I'll work on it when I can :-)
That's understandable. I'll keep poking away at these things, probably mostly on the documentation but I'm also planning on cleaning up the PolyVox SDK creation to ensure that it's all working as it should (especially with the new docs).

_________________
Matt Williams
Linux/CMake guy


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Sat Jul 03, 2010 10:22 am 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
milliams wrote:
Ok, I've just checked in some code (r1118) which provides this fall-back. Doxygen docs will always be created but if qcollectiongenerator is available then they will also be bundled up into a Qt Help Collection.


It works nicely :-) And actually the qcollectiongenerator is working as well. I think the problem before was with the 'doc' target rather than finding the qcollectiongenerator executable, because actually the build summary said everything was fine. Anyway, it's all working now and the generated documentation looks great :-)

milliams wrote:
Yes, I believe we can. It's LGPL3 but that doesn't affect the licence of PolyVox itself. The way it would work is that in the SDK we would include the assistant.exe executable, the required DLLs and plugins (only a few), polyvox.qch and polyvox.qhc. To make it run we should probably include a Windows shortcut file (or whatever they are) to run the assistant binary with the correct arguments.


Well, I imagine we can just include a static build of assistant. And a batch file which applies the correct arguments, this can be run directly or from our start menu entry.

milliams wrote:
Well I like Sphinx because it provides all the outputs we need including Qt Help. Plus the syntax makes it possible to read the source files as if they were pretty much plain text (though I guess asciidoc is like that too). From what I can tell Sphinx is a bit more lightweight and it's working. If I have time, I'll play around with creating a simple theme for it.


Sphinx is fine by me. Over the next couple of weeks I'll try to introduce a new very basic OpenGL example and write the first tutorial about it. Then we can start to see how all this ties together.

milliams wrote:
That's understandable. I'll keep poking away at these things, probably mostly on the documentation but I'm also planning on cleaning up the PolyVox SDK creation to ensure that it's all working as it should (especially with the new docs).


Great, you may have noticed that in another thread I provided WuTz with a zip of the source code. Maybe in the future I can try to provide an SDK instead :-) I'm wondering whether we should switch to a static library instead though. PolyVox is very small and most of it is template code (which goes directly into the executable anyway). So it might be nice to avoid the .dll dependancy.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Mon Jul 05, 2010 9:38 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
I'm just in the process of preparing a very basic OpenGL example because the existing one is actually a little complex. This will be the basis for 'Tutorial 1'. But I'm wondering, if I write the tutorial in Sphinx is there any capability for embedding source code snippets? Syntax highlighting would be a bonus but is not that important.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Tue Jul 06, 2010 10:27 am 
Developer
User avatar

Joined: Sun May 11, 2008 4:29 pm
Posts: 198
Location: UK
David Williams wrote:
It works nicely :-) And actually the qcollectiongenerator is working as well. I think the problem before was with the 'doc' target rather than finding the qcollectiongenerator executable, because actually the build summary said everything was fine. Anyway, it's all working now and the generated documentation looks great :-)

Excellent, I'm glad it's working.

David Williams wrote:
Well, I imagine we can just include a static build of assistant. And a batch file which applies the correct arguments, this can be run directly or from our start menu entry.

Yeah, that makes sense. It shouldn't need to be anything complicated.

David Williams wrote:
Great, you may have noticed that in another thread I provided WuTz with a zip of the source code. Maybe in the future I can try to provide an SDK instead :-) I'm wondering whether we should switch to a static library instead though. PolyVox is very small and most of it is template code (which goes directly into the executable anyway). So it might be nice to avoid the .dll dependancy.

Yeah, there's no reason why we can't provide both. For distributing on Linux, dynamic libraries are preferred but I can definitely see that a static library makes a lot of sense. I'll add a feature to create both a static and a dynamic library and certainly install them both on Windows.

David Williams wrote:
I'm just in the process of preparing a very basic OpenGL example because the existing one is actually a little complex. This will be the basis for 'Tutorial 1'. But I'm wondering, if I write the tutorial in Sphinx is there any capability for embedding source code snippets? Syntax highlighting would be a bonus but is not that important.

Yeah, see the info in the Sphinx manual for that. It uses Pygments for highlighting so most languages are covered. So, do something like:
Code:
.. code-block:: c++

   const uint16_t g_uVolumeSideLength = 128;
   Volume<uint8_t> volData(g_uVolumeSideLength,
                           g_uVolumeSideLength,
                           g_uVolumeSideLength);
   
   //Note: Deliberately go past each edge by one
   //      to test if the bounds checking works.
   for (uint16_t z = 0; z < g_uVolumeSideLength + 1; z++)
   {
You can use anything for indenting (as long as it's semi-consistent). I've been using tabs since that fits in with everything else in the code. You can also use the .. literalinclude:: command to grab code directly from source files but that might not be what you want most of the time.

[edit] Sorry, just realised it's code-block not highlight. highlight is for setting the highlighting language.

_________________
Matt Williams
Linux/CMake guy


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Thu Jul 08, 2010 7:40 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
milliams wrote:
Yeah, there's no reason why we can't provide both. For distributing on Linux, dynamic libraries are preferred but I can definitely see that a static library makes a lot of sense. I'll add a feature to create both a static and a dynamic library and certainly install them both on Windows.


Actually I don't think it's simple to have both on Windows. If you have a static library then it is all contained within the .lib file, whereas if you want dynamic linking then the code is in a .dll but you still need a .lib which acts as a kind of proxy or stub. So the .lib either contains the code, or it links to the .dll, but not both.

But it's no big deal, we can just have a CMake flag for static/dynamic/both, and default to static on Windows and a different option (both?) on Linux.

milliams wrote:
Yeah, see the info in the Sphinx manual for that. It uses Pygments for highlighting so most languages are covered.


Great, I'll give it a try when I get a chance.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Wed Jul 14, 2010 10:32 am 
Developer
User avatar

Joined: Sun May 11, 2008 4:29 pm
Posts: 198
Location: UK
I see that you've got this started. Good work, I'm glad that Sphinx is working well. I've got a few patches for it for build-systemy things like making Sphinx buildable from the 'manual' target (i.e. alongside the 'doc' target. We can make these all under one target but for now having them separate is easier) and automatically filling out various paths in the conf.py (I'm passing it through a CMake configure_file() command).

I've changed the file extension to rst rather than txt since this stops it seeing the CMakeLists.txt as a source file (and also enables the reST syntax-highlighting in Kate :)).

Finally, I've written a small Sphinx extension called doxylink which allows you to link to Doxygen generated documentation from within Sphinx. This means that you can write something like:
Code:
A :polyvox:`Volume <PolyVox::Volume>` is made up of a number of :polyvox:`Regions <PolyVox::Region>` which are...
This requires that the Doxygen documentation has been built since it scans the PolyVox.tag file output by Doxygen.

Any questions about it and I'll be glad to help.

_________________
Matt Williams
Linux/CMake guy


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Thu Jul 15, 2010 10:11 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
milliams wrote:
Finally, I've written a small Sphinx extension called doxylink which allows you to link to Doxygen generated documentation from within Sphinx. This means that you can write something like:
Code:
A :polyvox:`Volume <PolyVox::Volume>` is made up of a number of :polyvox:`Regions <PolyVox::Region>` which are...
This requires that the Doxygen documentation has been built since it scans the PolyVox.tag file output by Doxygen.


Cool, it seems to work nicely!

I plan initially on having three documents:

1) Install guide (I see you've done this).
2) Principles of PolyVox
3) Basic tutorial

Intermediate and advanced tutorials may come later. I se what you've done with the ..include so maybe I can do something similar to create seperate 'chapters'?


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Documentation viewer and generator
PostPosted: Fri Jul 16, 2010 9:34 am 
Developer
User avatar

Joined: Sun May 11, 2008 4:29 pm
Posts: 198
Location: UK
David Williams wrote:

Cool, it seems to work nicely!

I plan initially on having three documents:

1) Install guide (I see you've done this).
2) Principles of PolyVox
3) Basic tutoria

Intermediate and advanced tutorials may come later. I se what you've done with the ..include so maybe I can do something similar to create seperate 'chapters'?

The Install guide is the same install guide that has been in the base directory for a few months now but since I'd been writing it with the reStrucxtured Text markup all along I could simple import it directly and have it render correctly.

I think the best way to structure this is to have an index page which acts mostly as a table of contents (like on http://docs.python.org). Then each of the different documents will be a separate .rst file. Each .rst file will map to one .html file in the end.

The only reason I'm using the ".. include::" bit in install.rst is so that I can just pull in the existing document from the base directory without having to (re)move it from there (people expect to find an INSTALL.txt file there).

If you want a multi-page tutorial then you will have to make it as multiple .rst files. However, if you want to group them together, you can put them into a subfolder. For example you might end up with:
Code:
documentation/
  ├-index.rst
  ├-tutorial/
  |   ├-index.rst
  |   ├-begin.rst
  |   └-continue.rst
  ├-install.rst
  └-principles.rst

Then, tutorial/index.rst would act as an intermediate table of contents to link to the sub pages and would contain:
Code:
Tutorial
========

.. toctree::

   begin
   continue
This is the same as they do with the Sphinx documentation at http://sphinx.pocoo.org/contents.html leading to the markup subfolder at http://sphinx.pocoo.org/markup/index.html. You'd also need to edit the toctree in the base index.rst to read something like:
Code:
.. toctree::
   :maxdepth: 2
   
   principles
   tutorial/index
   install
The tutorial is looking good so far though.

_________________
Matt Williams
Linux/CMake guy


Top
Offline Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 16 posts ]  Go to page 1, 2  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Theme created StylerBB.net