Volumes Of Fun
http://www.volumesoffun.com/phpBB3/

Big, breaking changes to PolyVox (merging RLE/Paging work)
http://www.volumesoffun.com/phpBB3/viewtopic.php?f=14&t=188		 Page 1 of 2
Author:  David Williams [ Tue Apr 05, 2011 10:08 pm ]
Post subject:  Big, breaking changes to PolyVox (merging RLE/Paging work)
Those of you who who follow PolyVox development closely will know that we have been working for the last couple of months on improving the Volume class in PolyVox. There are primarily two new features:

  • Support for compression of the data in volumes. Each block in the volume can now be RLE encoded to reduce the memory footprint. A small cache of recently used blocks are kept uncompressed for fast access.
  • Support for paging volumes. When the compression isn't enough, blocks can now be paged out of memory to allow for huge volumes. This support ws largely contributed by ker.

I have just finished merging this work from it's own branch back into the trunk, so you will get it when you update SVN. Before you do this however, there are a number of breaking changes which you will need to be aware of. In no particular order:

  • The SWIG bindings are currently broken, so you cannot access PolyVox from C#. Java, etc. They will need updating to match the new code. I won't be doing this myself though will gladly help anyone who wishes to do so.
  • You should not attempt to access the volume from multiple threads. PolyVox was not thread-safe before but you may have been getting lucky - you probably won't get lucky anymore.
  • The new Volume class will probably be slower than the old one as blocks need to be looked up in a map rather than being accessed directly. We will aim to improve this in the future once we do some profiling and see how the new system is working out.
  • The Region class has changed. In particular the width(), height(), and depth() functions have been deprecated. If you want these values you should compute them yourself from getLowerCorner() and getUpperCorner().
  • The serialisation code in PolyVoxUtils is probably broken.
  • The voxel classes (Density8, Material8, etc) no longer provide the operator<.
  • A lot of int16_t and uint16_t have been replaced by int32_t.

I have also updated some documentation. In particular:


You may also find useful information in the development threads:


The download page has been updated with a snapshot of the new version, and I'm also leaving up the previous shapshot for those who have difficulties with the new system.


Good luck porting to the new system, post here with any problems you encounter.
Author:  ape [ Wed Apr 06, 2011 6:35 am ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
I've been waiting for this stuff to come to fruition to update the C# bindings, I'll try and get on that soon. Great work, guys.
Author:  David Williams [ Wed Apr 06, 2011 8:52 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
Great, but be aware that I commented out the bindings line in one of the CMakeLists.txt files, so youll need to put that back in before you can build them. I think the biggest challenge will be the use of std::function (for the paging) but hopefully you can find a way to bind to that.
Author:  milliams [ Thu Apr 07, 2011 5:10 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
It's great that this is finally merged and especially that you've found time to update the documentation.

I'm currently preparing for the conversion to Git so it shouldn't be too long before it's ready.

I do however see one of the tests are now failing (http://my.cdash.org/testDetails.php?test=5629657&build=175711). Given that it's the VolumeSizeTest I can believe that this is a consequence of the code changes and so it may be the test that's wrong but I don't know for sure
Code:
********* Start testing of TestVolume *********
Config: Using QTest library 4.7.3, Qt 4.7.3
PASS   : TestVolume::initTestCase()
testvolume: /home/matt/PolyVox/library/PolyVoxCore/include/Volume.inl:312: bool PolyVox::Volume<VoxelType>::setVoxelAt(int32_t, int32_t, int32_t, VoxelType) [with VoxelType = unsigned char, int32_t = int]: Assertion `m_regValidRegion.containsPoint(Vector3DInt32(uXPos, uYPos, uZPos))' failed.
This reminds me that I really should get the nightly builds working again (I submitted this failing test using make Experimental -- anyone can do that if they have a compilation error or a failing test).
Author:  David Williams [ Thu Apr 07, 2011 5:54 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
It would indeed be nice to get the CDash stuff working again. I think we can fix the failing test after the Git merge though. As you say, I imagine it just needs updating, and I would like to add some more tests anyway.
Author:  DefiniteIntegral [ Sat Apr 09, 2011 12:41 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
By the way on the download page you listed version 1440 as March 05, but it's actually April 05 ;)
Author:  Zoxc [ Sat Apr 09, 2011 2:14 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
cstdlib should be included in Volume.inl because it now uses abort.
Author:  David Williams [ Sun Apr 10, 2011 6:31 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
DefiniteIntegral wrote:
By the way on the download page you listed version 1440 as March 05, but it's actually April 05 ;)


Thanks, I've fixed it.

Zoxc wrote:
cstdlib should be included in Volume.inl because it now uses abort.


Also fixed (in Git, but not in SVN).
Author:  kattle87 [ Thu Apr 14, 2011 8:24 am ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
This looks like some awesome work! Congratulation to everyone :)
Just one question: how much difficult would it be to make polyvox thread-safe when using OpenMP?
OpenMP is a compiler-directive based API for shared-memory parallelism. Just to give you an idea, parallelizing a for-loop requires ONE #pragma line, and the compiler will take care of doing everything else. No libraries nor anything else needed (just a compiler directive and a flag while compiling!)
Having a section marked as "critical" (meaning it can be accessed only by one thread at a time) also requires just one line of code alike. (see https://secure.wikimedia.org/wikipedia/ ... C.2B.2B.29 there's the "critical" clause).
I only took a look at the new doc but the functions that manage the block cache could probably get some critical sections and be ready to be used with OpenMP.
Possible objections:
-we don't need parallelism
-we don't like parallelism
-we want the users being able to choose and tweak the library with their own parallel paradigm, we can't make such choices for them.

I'm doing some heavy computations, and I can't say how much of parallelism I will need (surely it will help). But for sure this is something a some people would like. (EG: extracting different surfaces at the same time might be nice for some people!).
Author:  David Williams [ Thu Apr 14, 2011 8:29 pm ]
Post subject:  Re: Big, breaking changes to PolyVox (merging RLE/Paging wor
I think in general we would like to see (parts of) PolyVox become thread-safe. The main reason it has not happened is that I believe it is a lot of work, and to be honest I don't have significant experience in this area. Of course I've written code which takes advantage of threading, but I think that actually designing thread-safe data structures (while keeping them efficient) is a lot more difficult.

If we do eventually go down this path then OpenMP may well be an option - it seems pretty well supported. The new C++0X standard also includes threading primitives so this is a strong candidate too (in the spirit of avoiding dependancies). However, at the moment PolyVox is storing its blocks in an std::map, and while a threading library may make it easy to ensure that only one thread can access the map at a time, this in itself will not give performance benefits. We would instead want to replace the std::map with a structure which allows concurrent access from different threads.

I think initially the most likely outcome would be a new volume class which sits along side the existing one. Perhaps we could have PagedVolume and SimpleVolume, someting like that. SimpleVolume could just store raw data and so be made thread safe much more easily, but would not handle large amounts of data (perhaps up to 1Gb). Then maybe add some utility classes to copy data between them. I have no immediate plans to do this though.
Page 1 of 2 All times are UTC
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/