It is currently Mon Dec 16, 2019 12:33 am


All times are UTC




Post new topic Reply to topic  [ 24 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject: Re: New Volume class with improved compression
PostPosted: Wed Feb 16, 2011 2:58 am 

Joined: Sun Jan 23, 2011 6:06 am
Posts: 92
I am also attempting paging across multiple volumes with multiple threads.

However I don't really want to lock the entire system every time I need to get/set a voxel, since different threads may be accessing voxels a large distance apart.

So I am just using smaller volumes 1024^3 which I have wrapped in a class containing a mutex. That way I only have to lock the volume containing the voxel a particular thread is accessing at the time.

This is where built in thread support in Polyvox would concern me - I don't want to switch to a super large volume only to have it lock the entire thing each time it is accessed.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Wed Feb 16, 2011 8:00 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
DefiniteIntegral wrote:
I am also attempting paging across multiple volumes with multiple threads.

However I don't really want to lock the entire system every time I need to get/set a voxel, since different threads may be accessing voxels a large distance apart.

So I am just using smaller volumes 1024^3 which I have wrapped in a class containing a mutex. That way I only have to lock the volume containing the voxel a particular thread is accessing at the time.


It sounds like you are already accessing the volumes in a thread safe way (by having a mutex per volume) and so you should have no problem with switching to the new system but keeping the multiple volumes. You should then get a significant amount of compression on each 1024^3 volume so hopefully you'll see some memory savings as well.

DefiniteIntegral wrote:
This is where built in thread support in Polyvox would concern me - I don't want to switch to a super large volume only to have it lock the entire thing each time it is accessed.


Indeed, but if I do make the Volume class thread safe it will not be done by locking the whole volume each time a voxel is accessed. Instead PolyVox would just lock the affected block, and even then it would probably only need to lock it if it needs to be compressed/uncompressed.

So I think you can safely switch to the new Volume class, but you will need to keep several of them as you are doing at the moment.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Wed Feb 16, 2011 9:03 pm 

Joined: Sat Sep 18, 2010 9:45 pm
Posts: 189
I may try something like that eventually. My plan is to instead of using my task-based thread system to generate chunks, chunks that may map to the same volume, I will instead just divide my world into several large volumes, then make it so only one thread can work on it at a time. The trick is to still use the task-based thread system, just now when a task is done, it will mark that volume as free. This way basically still generate manageble chunks at a time, still use the task based threads, still use have interactivity, have larger chunks with compression, and without using locks. I'm just not sure whether this will work the way I hope it will work, especially WRT to performance.

Anyway, I did some quick timing of my code with various settings of shared block size and cache size and also polyvox without compression. I'm not sure whether the tests are valid or not, it's just quick test. Also, I don't think I will worry too much about this for now. I don't have any game play. I still don't have a clear picture of how my final game will work. Without knowing that it's hard to say how (and how big) I should make my world. I need to finish doing that first.

Here are some results (timing are done with Win32 QueryPerformanceCounter; time are in seconds.):

Code:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

World Map config:
on-screen map dimensions: 1024*2
unload radius: 544
Outputing Volume Map stats:
Shared Block Size: 32
Uncompressed Cache Size: 4
avgCompression: 0.64402
total volume size in kb: 189423
average graphics and physics upload time: 0.00851222
average generation time: 0.129216
average extraction time: 0.120105
average elapsed time: 0.257834

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

World Map config:
on-screen map dimensions: 1024^2
unload radius: 544
Outputing Volume Map stats:
Shared Block Size: 16
Uncompressed Cache Size: 8
avgCompression: 0.64402
total volume size in kb: 189423
average graphics and physics upload time: 0.00850891
average generation time: 0.117844
average extraction time: 0.107218
average elapsed time: 0.233571


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++



Without compression (polyvox trunk)

One chunk One volume (32 by 32 by 256 sized volumes)

Code:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

World Map config:
on-screen map dimensions: 1024^2
unload radius: 544
Outputing Volume Map stats:
Shared Block Size: 32
Uncompressed Cache Size: 1
avgCompression: 0
total volume size in kb: 0
average graphics and physics upload time: 0.00849916
average generation time: 0.0419583
average extraction time: 0.071266
average elapsed time: 0.121723

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Multipe pages per volume. Each volume is 1024x1024 in this case.

Code:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

World Map config:
on-screen map dimensions: 1024^2
unload radius: 544
Outputing Volume Map stats:
Shared Block Size: 32
Uncompressed Cache Size: 1
avgCompression: 0
total volume size in kb: 0
average graphics and physics upload time: 0.00849916
average generation time: 0.0419583
average extraction time: 0.071266
average elapsed time: 0.121723

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++



Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Wed Feb 16, 2011 10:34 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
beyzend wrote:
Anyway, I did some quick timing of my code with various settings of shared block size and cache size and also polyvox without compression. I'm not sure whether the tests are valid or not, it's just quick test. Also, I don't think I will worry too much about this for now. I don't have any game play. I still don't have a clear picture of how my final game will work. Without knowing that it's hard to say how (and how big) I should make my world. I need to finish doing that first.


Yep, I think that's sensible. I'm mostly concerned with making sure I won't completly break somebodys project by switching to the new Volume class, and it seems that in general we're going to be ok. From your figures it looks like the new class is a little slower, but that's hardly suprising with the compression that is being done. I'm sure there's some scope for optimisation here anyway.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Thu Feb 17, 2011 2:54 am 

Joined: Sun Jan 23, 2011 6:06 am
Posts: 92
David Williams wrote:
Indeed, but if I do make the Volume class thread safe it will not be done by locking the whole volume each time a voxel is accessed. Instead PolyVox would just lock the affected block, and even then it would probably only need to lock it if it needs to be compressed/uncompressed.


That's good to hear. Locking individual blocks should work out better than locking the entire volume in a wrapper class.

Wouldn't you still need to lock for get/set on a cached block to prevent access violation? Maybe a R/W mutex....


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Thu Feb 17, 2011 10:01 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
DefiniteIntegral wrote:
Wouldn't you still need to lock for get/set on a cached block to prevent access violation? Maybe a R/W mutex....

I'm not sure exactly, but I optimistic a solution can be found. Maybe there needs to be a (smaller) block cache per thread, or maybe the volume allows several threads to be reading at a time but only one to be writing. I'm not going to jump to any conclusions about this until I give it some proper thought in the future.

But I'll only implement something in PolyVox if it is better than just locking the whole volume, because otherwise it's best left in the hands of the user.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Sat Feb 19, 2011 8:28 pm 

Joined: Fri Feb 18, 2011 8:41 pm
Posts: 173
I think a good idea would be to allow streaming of regions.

Allow to create a whatever size volume but only have it in RAM per request. That's what I did in my height map used for the GPU Clip Maps.

In mine it worked by setting a chunk size (you can use it as constant region size), so for example the world could have been the size of 50,000 * 50,000 and each chunk was 1,000*1,000. When walking around the world I'd always have the 9 squares at the center of my position handled by the renderer and the parts surrounding those ready in RAM so if I move away from a chunk's bounding region they are ready to replace the ones no longer needed. Then I start loading the new required chunks and unload the ones that are no longer in my second layer surrounding after some time.

Of course if you do so, don't bother with the threats etc, just make a function to load region (or region index if constant size) from stream and to unload and let the programmer handle it in his own threads so it is easier for you to create. This is pretty fast and flexible system to allow for supposedly "infinite" terrains. It should also handle the problem of holes and mis-matching geometry quite nicely and allow for very large worlds (I think morrowind or oblivion were 20 kilometers in size, you can easily get to be larger with this kind of system)


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Sat Feb 19, 2011 10:46 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
In general I agree, it would be nice to support streaming. If implemented it will probably provide some kind of callback funtion to handle missing regions, so that the user can choose to load it from disk, generate it procedually, etc. Such infinite streaming volumes are pretty low priority for me though.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Sun Feb 20, 2011 8:19 am 

Joined: Fri Feb 18, 2011 8:41 pm
Posts: 173
Maybe I will look into adding that functionality to the source code myself when I have some time and it becomes more urgent for me, if it will turn good enough you could add it to the svn if you like. But might take me some time to get free time for this.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: New Volume class with improved compression
PostPosted: Sun Feb 20, 2011 3:03 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
Sure, that'd be great. On a related note I'm going to hold off merging this into the trunk until next weekend, because I have a busy week and may need time to deal with any problems which occur.


Top
Offline Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 24 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Theme created StylerBB.net