Wow a lot of changes have been made since I last worked on my project. It appears all for the better. I was deeply involved in writing a paging engine, but it seems Polyvox beat me to it.

A have a few questions concerning the upper bounds of this new volume.

First, What is the upper bound on the number of voxels this new volume can contain? I would like to implement 'endless' terrain in my engine and need to know how far this volume could get me.

Second, What is the upper bound on the memory requirements for this new volume, and for the simple volume? Or in other words maximum file size. If I end up writing my own paging engine, I will try to reduce file count by combining multiple 'chunks' in one file, but I need to know how large the chunks might be to give them enough room.

I hope those questions made sense. They do in my head If they are already answered elsewhere in the forum, I apologize. I looked but couldn't find anything.

One last question, Have the serialization features been changed to work with these new volumes yet? Or do I need to write my own function to save the data?

Thank you for all your help, this forum has been great. And this library has been fantastic so far. I am interested in testing the changes to the Meshdecimator. Are there any plans to add LOD support in the future?

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--

See these functions in LargeVolume:

void setCompressionEnabled(bool bCompressionEnabled);
void setMaxNumberOfUncompressedBlocks(uint16_t uMaxNumberOfUncompressedBlocks);
void setMaxNumberOfBlocksInMemory(uint16_t uMaxNumberOfBlocksInMemory);

Using the default block size of 32, the maximum amount of memory that can be used is approximately 2GB, if max blocks in memory is 65535 and compression is disabled. This would be a cube approximately 1290^3 voxels.

For larger cases it might be necessary to change to uint32_t.....

Edit: The above assumes 1-byte per voxel.

It's a good thing you posted this, I just realized I am probably going to need more than 65535 blocks in memory (although I'll have almost all of them compressed)

Is there any reason one might not use compression? Also how does block size affect memory use?

It appears I might have to stay with simpleVolume and write my own paging engine. I hope to use many more than 65535 voxels. My current goal is four voxels a meter, and I hope to allow the landscape to go as far as the computer can support. Its possible I could use multiple largeVolumes though.

I had a few other questions in my first post I hope get answered, but I do have a couple more.

Is there any upper bound to the size when compression is enabled? If not is there any average size, so I could see what to expect?

Thank you for your help!

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--

I can't think of any good reason not to use compression at all. That was just an example for a maximum memory use case.

The best approach would probably be to use a high number of uncompressed blocks during large-scale save/load operations, and change to a small number of uncompressed blocks for normal mesh extraction operations (since extraction would only access volume intermittently).

Personally I use LargeVolume as it has unrestricted bounds (unlike SimpleVolume), but I manage all paging externally. This can be done by passing NULL function pointers to the LargeVolume constructor. To compile successfully it is necessary to comment out the deprecated constructor, however. To manage paging yourself, you can allocate a block by just accessing a voxel within it - this will create an empty block. Then use void flush(Region regFlush) to deallocate blocks.

So a paging endless terrain using external page management is possible, I do this with my code already.

Of course to have more than 65535 blocks in memory, it will be necessary to make modiciations to LargeVolume to change from uint16_t to uint32_t. Shouldn't be that hard to do I imagine.

Block size affects memory use in relation to setting the maximum number of blocks in memory. That is because it is number of BLOCKS in memory (not voxels). So the larger each block, the more memory used for the same number of blocks. Eg using the example from my first post, a 64-voxel block size would increase memory use from 2 to 8GB if all 65535 were allocated.

With compression enabled, there is no way to really say how high memory use would go. It will depend on how many compressed/uncompressed blocks you are using, and what sort of data you were storing. I really have no idea about the compression but I believe David has mentioned for Minecraft style terrain the compression would be something like 1/100th size?

As for saving the data, chunk size is up to you. If you are saving per-block using standard block size, an uncompressed block is 32KB. Compressing this using ZLIB with repeating but mildly noisy data reduced it to about 4KB in my testing. Minecraft style terrain data should compress much more. How much to save in 1 file is up to you, it is unrelated to block size. Just remember that the larger the region you save, the better ZLIB compression you will get (within reasonable limits relating to randomness of the data).

I am not certain about the serialization features at the moment. Possibly broken? Best to ask David about that.

I hope this long ranting post helps somewhat....

DefiniteIntegral has already addressed some of these but I'll run through them quickly...



The theoretical limit is the size of an int32. I.e. with enough memory you can go +/- 2 billion voxels in each directions.



It is supposed to be simply however much memory you have in your PC. However, it seems that the maximum number of blocks in memory is a uint16 which limits it to 65535 blocks. This is an oversight and it should be changed to a uint32. It's still using an std::map internally so I don't know when this gets to slow, we should still test with a std::hash_map at some point.



Seralisation is currently broken. It used to fill a whole volume from a file stream but this doesn't make sense now that the volumes are infinite. I will change it to accept a Region with in a volume. However, it will still be fairly basic (doesn't handle 'chunks' as such) and I think most people will want custom solutions.



Are you using the smooth extractor or the cubic one? The cubic version was massively improved a month ago and now decimates as much as it can without adding any visual artifacts. I don't plan to take this any further becuase I think any visual artifacts will be very obvious in this case.

The smooth one doesn't have LOD at the moment (beyond what the MeshDecimator provides, which isn't great). I do plan on adding utility classes to resize one volume into another one. This will be a big step in allowing people to perform LOD themselves.

Ok, thank you for your answers. I'm still not sure whether to go with the large volume, or write my own paging engine for the smaller volume.

In the latter case, would it be better to update to the latest git version, or stay with the old snapshot I am using?

Also if serialization is broken, how do you save volume information to disk?

Thanks

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--

Overall I think you should go with the existing LargeVolume, at least until you have a reason not too. The only downside to the LargeVolume which I can think of is that it is not threadsafe.

If you go your own route you should probably write your own volume class, rather than devise a system which places multiple volumes next to each other. If you try to use multiple volumes you will find you get various problems at the borders.



I guess that in either case you should look to upgrade, as PolyVox is constantly improving. I'll try to make another stable snapshot in the next week or so.



If you mean how do I do it, then actually I don't as I'm using procedural generation. If you mean how can it be done, well you just have call call getVoxelAt() on all the voxels you want to save and put the results into a file stream.

Ok so the theory behind the serialization code is alright, the code itself doesn't work anymore.



By not threadsafe, do you mean the library does not protect itself, and a programmer would have to use his own mutexs and semaphores to protect data? Or is it unsafe regardless of any protection taken?

I suppose for now, there is no way I will ever exceed the bounds of uint32. I am mainly making a basic procedural terrain generation engine for now anyways. However in the future I would like to have the terrain effectively limitless. My ideals right now call for 4 Voxels/meter with 256 meters to the 'bottom' of the earth (I want to generate caves too) with height (possibly) unbounded and all directions unbounded. I imagine at 4 voxels a meter this will add up to an incredible amount of voxels in little time. I think 4 voxels a meter will give a fairly smooth looking terrain though.

This leads to another question which could sound crazy. If the engine can be changed to a uint32, what about uint64? Also how involved would it be to change out the stl_map for a hash_map? I'm obviously not as familiar with this library as I would like to be, but maybe I could make the change and test it out.

One last question. I'm sure at some point the use of the large volume was covered in another forum post. Could someone either point me to this post (as I can't seem to find it) or some other source so I can figure out how to use this new paging feature. I assume you add new regions as you need more space, but I don't know.

Thanks again for your help.

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--

Yes, that's correct.



It means you should only access the volume from one thread at a time. You can have one thread which (for example) loads data into the volume, and another thread which (for example) runs the surface extractors, but you should never let the two threads access the volume at the same time. You will need to take responsibility for that by providing your own thread syncronisation if you use multiple threads.

At leat that is my understanding, I haven't actually used the LargeVolume class much myself.



I don't think it would be a big problem, though I don't anticipate we would need to do it any time soon.



I don't think it's hard, I expect that std::map and std::hash_map have very similar interfaces. But you should do some profiling first - you only need to make this change if you determine that the std::map lookups are becoming a bottleneck.


The most relevant forum posts are here:
http://www.thermite3d.org/phpBB3/viewtopic.php?f=2&t=145
http://www.thermite3d.org/phpBB3/viewtopic.php?f=14&t=197
http://www.thermite3d.org/phpBB3/viewtopic.php?f=14&t=188

There is also some class documentation here:
http://thermite3d.org/resources/documentation/polyvox/library/doc/html/class_poly_vox_1_1_large_volume.html#_details

Note that in the lastest code the class name has changed from 'Volume' to 'LargeVolume'. I'll try to get a snapshot with these changes out soon.

These functions were pointed out when talking about the size max. Are these the only places where I would replace uint16?

Thanks for those links I will study those soon!

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--