Thanks for clarifying that. My head has been around ogre paging and listening to DragonM talk about converting heightmap formats to and from this and that, sorry for the slight derail, was hoping to help provide a nudge in the right direction with my inane babblings, carry on gentlemen!

Ok, yesterday I planned finally working on my VoxelMesh and VoxelEditor class but I am still using quite an old version (well, few weeks old), I didn't start yesterday.

I am wondering, should I bother to start now or will I need to completely re-work it with the new system you are working on and therefore not to bother yet?

as long as you do not access the blocks directly you should be fine

you'll need to get rid of getEnclosingRegion (by manually creating a region)
and of all calls that have to do with the size of the volume (just insert the value from some variable you created yourself)



I created a solution that works, and imo should not be noticably slower than the previous VolumeSampler (didn't test it, but if anyone feels up to it, there's a #define that can easily be used to change the method back or to something new)

please tell me about any issues, I know things like the serialization or the changetracker cannot work at all, but serialization is imo obsolete anyway and the changetracker should be changeable to the new Volume with some effort

How does it work then? How do you tell it to get rid of data and insert new data?

once the buffer is full, a function is called with a region as an argument.
you can then extract all the points inside the region through the getVoxel function

once data is requested that is not in memory, a function is called that is supposed to set all voxels in given region.

these functions by default do nothing, but you can use boost::bind or std::bind to call any function you like.

Ok sounds good and simple.

How do you set how large of region to prepare each time?

How do you unload a region?

Also what do you mean "once the buffer is full"? which buffer? when does it fill?

a region is set on volume creation, check the (now only, before it was the 4th) parameter. uBlockSideLength
the regions that are loaded are always of size uBlockSideLength^3

you cannot unload a region manually (yet?)
once the buffer is full, regions will automatically start unloading.

there is a function to set the buffer size.
if you want a maximum of 1024x1024x1024 voxels to be loaded before unloading begins, and your uBlockSideLength is 32 (the default), then you need to set a buffer size of (1024/32)^3 = 32768

due to RLE, you can have much more than 1024^3 voxels in memory at one time so the line blurs somewhere as to what are the ideal settings.

the buffer size goes up by one for each region that is loaded.

Great, it seems to work There are quite a few things I will want to tidy up though and some things I'll want to think about some more.

Having seen that it causes some problems in Thermite, the main thing I'm thinking is that I'd still like the option of giving a volume a fixed size (Thermite currently expects this). I think we could keep the concept of the volume having width, height, and depth but if the user does not specify these then they can default to being the biggest possible values. Maybe if the user does specify these then the streaming should be automatically disabled. I still need to think about this a bit.

My main concern is not breaking too much backwards compatability - an approach like I describe should allow the serialization and VolumeChangeTracker to keep on working. A lot of people have already got in place a system with different volumes for different pages and it would be nice if they could transition only when they are ready.

Is the streaming now working well enough for your own project? If so I think I'll commit your patch unchanged and then go over it making the changes above and generally keeping everything tidy. Then in a week or so I'll hand it back to you. Then we repeat until it's working well for both of us?

I'll start playing with it as soon as you commit and try to help weed out bugs if they occur. I know I'm overzealous, but this is too awesome to pass up!

This sounds like a pretty big change to the core of PolyVox. If I were you I'd consider forking the current codebase and running with a new framework if that is more beneficial. I just don't see the reason behind trying to stay backwards compatible with projects that could just stay on the current codebase.