It is currently Sat Aug 22, 2020 4:29 am


All times are UTC




Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: LargeVolume rather slow & wasting space?
PostPosted: Thu Jan 03, 2013 3:51 pm 

Joined: Thu Oct 06, 2011 2:26 pm
Posts: 46
Location: Berlin
Hey there,

I'm using PolyVox to try out some terrain generation stuff with large terrain (2000x1000x2000 per terrain page). Basically, I want to to use PolyVox as a "mesh generator" (for Ogre3D) only, and forget about it once I have the mesh. Pretty mean, I know ;)

First thing that bugged me was that it does not seem to be possible to create voxels of less than 1 byte size. Which is 8 times as much as I actually need.
I actually just need solid/air information, everything else I want to store somewhere else where it takes less space than if it was stored within each voxel. I tried LargeVolume<bool> of course, but that did not decrease the memory used at all.
Is there a way to make this possible? It really feels like I'm wasting space here :(

On the other hand, even when using BYTE directly, the Volume doesn't even take up 100MB. How is it even possible that 2000x1000x2000 voxels of 1 byte take up that few space? :shock:

My actual problem is the speed, though...
Here is the code that applies a heightmap to the Volume, by simply making the upmost voxel at one coordinate solid:
Code:
int y = 0;
int midLevel = _height / 2;
float range = ((float)_height) / 2.0f;
double value = 0.0;
for (int x = 0; x < _width; ++x)
{
   for (int z = 0; z < _depth; ++z)
   {
      // Get the correct y coordinate
      value = _heightMap[z * _depth + x];
      y = int(midLevel + value * range + 0.5f);
         
      // Apply to top voxel
                // _voxelVolume is a PolyVox::LargeVolume<BYTE>*
      _voxelVolume->setVoxelAt(x, y, z, 255);
   }
}

That's not that much, IMO, yet takes 20 seconds to complete for a 2000x2000 heightmap.
Creating the heightmap itself barely takes 0.15 seconds. And yes, it is release mode.
I am sure I'm doing something very, very wrong here, but what is it?

Btw... although I'm already sure that something I'm doing is fishy, I continued and tried to generate a mesh from that LargeVolume. I think I started the process roughly 5 minutes ago and it is still not finished. Okay, I'm in Debug mode right now, but still ;)

Here is the code I'm using for that (more or less taken from a tutorial):
Code:
void TerrainCreator::applyVoxelsToMesh(Ogre::ManualObject* p_manualMesh)
{
   PolyVox::SurfaceMesh<PolyVox::PositionMaterialNormal> mesh;
   PolyVox::MarchingCubesSurfaceExtractor<PolyVox::LargeVolume<BYTE> >
      suf(_voxelVolume, _voxelVolume->getEnclosingRegion(), &mesh);
   suf.execute();

   p_manualMesh->begin("MySimple", Ogre::RenderOperation::OT_TRIANGLE_LIST);
   {
      const std::vector<PolyVox::PositionMaterialNormal>& vecVertices = mesh.getVertices();
      const std::vector<uint32_t>& vecIndices = mesh.getIndices();
      unsigned int uLodLevel = 0;
      int beginIndex = mesh.m_vecLodRecords[uLodLevel].beginIndex;
      int endIndex = mesh.m_vecLodRecords[uLodLevel].endIndex;

      for(int index = beginIndex; index < endIndex; ++index)
      {
         const PolyVox::PositionMaterialNormal& vertex = vecVertices[vecIndices[index]];
         const PolyVox::Vector3DFloat& v3dVertexPos = vertex.getPosition();
         const PolyVox::Vector3DFloat& v3dVertexNormal = vertex.getNormal();
         const PolyVox::Vector3DFloat v3dFinalVertexPos =
            v3dVertexPos + static_cast<PolyVox::Vector3DFloat>(mesh.m_Region.getLowerCorner());

         p_manualMesh->position(v3dFinalVertexPos.getX() - 25, v3dFinalVertexPos.getY()
                     - 25, v3dFinalVertexPos.getZ() - 25);
         p_manualMesh->normal(v3dVertexNormal.getX(), v3dVertexNormal.getY(),
                     v3dVertexNormal.getZ());

         p_manualMesh->colour(1.0f, 0.0f, 1.0f);
      }
   }
   p_manualMesh->end();
}

_________________
My site! - Have a look :)
Also on Twitter - with more ketchup


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Thu Jan 03, 2013 7:03 pm 

Joined: Fri Sep 14, 2012 10:54 pm
Posts: 15
TheSHEEEP wrote:
Hey there,

I'm using PolyVox to try out some terrain generation stuff with large terrain (2000x1000x2000 per terrain page). Basically, I want to to use PolyVox as a "mesh generator" (for Ogre3D) only, and forget about it once I have the mesh. Pretty mean, I know ;)

As far as i got it, thats what PolyVox is made for. Storing Volumedata and extracting meshes.

Quote:
First thing that bugged me was that it does not seem to be possible to create voxels of less than 1 byte size. Which is 8 times as much as I actually need.
I actually just need solid/air information, everything else I want to store somewhere else where it takes less space than if it was stored within each voxel. I tried LargeVolume<bool> of course, but that did not decrease the memory used at all.
Is there a way to make this possible? It really feels like I'm wasting space here :(

Well... Thats tricky. The only option i could imagine is using Bitfields. But meh... Thats a lot of shifting going around and some position claculating. I dont know how fast that would be despite the horrific speed of bitshifting. :D
bool itself is one byte of size, too. If I'm correct. That is because the CPU can only compute packs of 1 byte size minimum.

Quote:
On the other hand, even when using BYTE directly, the Volume doesn't even take up 100MB. How is it even possible that 2000x1000x2000 voxels of 1 byte take up that few space? :shock:

The largevolume stores a big deal of its data with runtimelengthencoding. Only a few of the voxels are stored uncompressed for faster access. Some wicked stuff goin on there, believe it! :P

Quote:
My actual problem is the speed, though...
Here is the code that applies a heightmap to the Volume, by simply making the upmost voxel at one coordinate solid:
Code:
...

That's not that much, IMO, yet takes 20 seconds to complete for a 2000x2000 heightmap.
Creating the heightmap itself barely takes 0.15 seconds. And yes, it is release mode.
I am sure I'm doing something very, very wrong here, but what is it?

Ye. Thats quite right, i noticed that myself. I got data of 64x64x64 stored and generated ( perlin noise ) in 3seconds but extracted in 0.03 seconds. I guess that is because of the forementioned RLE encoding going on. THe newest voxels are stored uncompressed while the others get compressed. Thats a lot of packing, unpacking etc going on.
Just guessing tho.

Quote:
Btw... although I'm already sure that something I'm doing is fishy, I continued and tried to generate a mesh from that LargeVolume. I think I started the process roughly 5 minutes ago and it is still not finished. Okay, I'm in Debug mode right now, but still ;)

That rly sounds fishy. oô
Extracting is SOOOOO frickin fast with PolyVox!



Just my 50pence. Every info without validation or warranty. :P


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Thu Jan 03, 2013 7:32 pm 

Joined: Thu Oct 06, 2011 2:26 pm
Posts: 46
Location: Berlin
I just tested everything with SimpleVolume. That does work for smaller volumes but crashes when creating a volume of size 2000x1000x2000 with a bad_alloc. Not that surprising, I guess, as memory is limited for each process (at least on Windows). That was also the initial reason why I was trying to create a volume that takes only 1 bit per voxel.

KuroSei wrote:
Well... Thats tricky. The only option i could imagine is using Bitfields. But meh... Thats a lot of shifting going around and some position claculating. I dont know how fast that would be despite the horrific speed of bitshifting. :D
bool itself is one byte of size, too. If I'm correct. That is because the CPU can only compute packs of 1 byte size minimum.

You're right about bool being 1 byte always. But I don't think that putting a Bitfield into PolyVox will help me, here. Guess I'm stuck with either writing my own voxel -> mesh library (which I certainly won't do ;))
or somehow finding a way to speed up LargeVolume (that would be optimal!)
or writing my own operating system so I can use large SimpleVolumes (unlikely)
or some other possibilities I will have to come up with (likely)

KuroSei wrote:
The largevolume stores a big deal of its data with runtimelengthencoding. Only a few of the voxels are stored uncompressed for faster access. Some wicked stuff goin on there, believe it! :P

I do!

KuroSei wrote:
Ye. Thats quite right, i noticed that myself. I got data of 64x64x64 stored and generated ( perlin noise ) in 3seconds but extracted in 0.03 seconds. I guess that is because of the forementioned RLE encoding going on. THe newest voxels are stored uncompressed while the others get compressed. Thats a lot of packing, unpacking etc going on.
Just guessing tho.

Possibly, but then it would be great if you could disable that packing for a while.
Like:
1. Apply a 100x100 block of the heightmap.
2. Pack.
3. Apply the next block of the heightmap. Go to 2 until finished.
5. ???
6. Profit.

KuroSei wrote:
That rly sounds fishy. oô
Extracting is SOOOOO frickin fast with PolyVox!

I noticed when I had a SimpleVolume of a size that did not crash the application (barely).
But have you tried with a 2000x1000x2000 LargeVolume? :)

So, if someone could tell me a way to make LargeVolume faster or has a better idea for creating a really big terrain (think Minecraft, but real 3D instead of just blocks), then I'd be very thankful :)

_________________
My site! - Have a look :)
Also on Twitter - with more ketchup


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Thu Jan 03, 2013 8:00 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
Improving the volume classes is our main focus for the next release of PolyVox, and if you look at BitBucket you can see that yesterday I was adding a large number of unit tests to ensure they all behave as expected. We're currently working on our next engine built on PolyVox and I expect the LargeVolume to be at the core of this, but currently it does have a couple of issues.

The LargeVolume is able to store data in a small amount of space because it makes use of RLE compression (as KuroSei said) which should be effective for Minecraft style data but probably not so much for smooth data. Yesterday I opened an issue to replace this with a proper compression library because I believe this will be faster and better. See https://bitbucket.org/volumesoffun/poly ... on-library

The LargeVolume also performs paging, and between this and the compression there is quite a lot of logic required when accessing data. This means that the LargeVolume is slower than the other volume types but I haven't yet determined how much. The unit tests I just added will be used for benchmarking and over the next week or two I expect to make some improvements here. It's really what I'm working on at the moment.

I think that use of the LargeVolume is the most likely cause of your poor performance, but you should also make sure that your mesh is not too large. If you are generating a single mesh from that 2000x2000 region then that will be too much, and you should instead use a larger number of smaller meshes. Start small, get everything working, and then scale it up.

As for storing one bit per voxel I'm afraid we don't support this, and we've got no plans to. A good compression algorithm will probably notice it though and give you a good compression rate.

If you're going to much work with LargeVolume I'd recommend using the 'develop' branch in our Git repo (you can download a zip of the source) because I do expect to see some improvements happening here (but check the changelog.txt regarding the new wrapping modes). Also keep an eye on this issue: https://bitbucket.org/volumesoffun/poly ... me-classes

Oh, one other thing, it's possible to increase the number of blocks which are used before paging and compression kick in. Maybe this helps your performance:

http://www.volumesoffun.com/polyvox/doc ... 5096a1e1ce

http://www.volumesoffun.com/polyvox/doc ... a51584c2b5


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Thu Jan 03, 2013 8:28 pm 

Joined: Thu Oct 06, 2011 2:26 pm
Posts: 46
Location: Berlin
It seems I have a very good timing, then. ;)
Great to see you are working on those improvements!

As I am currently just testing some things with terrain, I don't need the speed improvement right now, but in the future (half a year from now, perhaps), I will. And your post does sound as if something better will be ready by then.
Right now, I'm more about testing noise for terrain generation, etc. I was just a bit surprised when performance went very slow in a place where I did not expect it :)

Also thanks for all the links!
David Williams wrote:
I think that use of the LargeVolume is the most likely cause of your poor performance, but you should also make sure that your mesh is not too large. If you are generating a single mesh from that 2000x2000 region then that will be too much, and you should instead use a larger number of smaller meshes. Start small, get everything working, and then scale it up.

Would it really be faster to create a lot of small volumes (and meshes) than to create one large voume & mesh?

David Williams wrote:
Oh, one other thing, it's possible to increase the number of blocks which are used before paging and compression kick in. Maybe this helps your performance:
http://www.volumesoffun.com/polyvox/doc ... 5096a1e1ce
http://www.volumesoffun.com/polyvox/doc ... a51584c2b5

It did, thank you!
All in all, it is still slower than creating a SimpleVolume of equal size and then a mesh from it.
If I'd have to create multiple meshes anyway, I'd probably go with SimpleVolumes for now, at least until the LargeVolume performance improves :)

_________________
My site! - Have a look :)
Also on Twitter - with more ketchup


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Fri Jan 04, 2013 9:35 am 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
TheSHEEEP wrote:
Would it really be faster to create a lot of small volumes (and meshes) than to create one large voume & mesh?

In general you should have one volume regardless of how big your terrain is. Have a read of this: http://www.volumesoffun.com/polyvox/doc ... aller-ones

Note that you should still generate multiple (probably many) separate meshes. Smaller meshes are faster to generate, easier to update, and can be culled against the view frustum for faster rendering.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Fri Jan 04, 2013 11:41 am 

Joined: Thu Oct 06, 2011 2:26 pm
Posts: 46
Location: Berlin
Well, 2000x1000x2000 already is a "smaller" page of the whole terrain.
To be able to create more interesting terrain, 1 unit = 0.5 meters. Everything less "dense" would look way too block-like.

So said terrain would be 1km/1km big. That's not that big when it comes to viewing distance.
Imagine standing on a mountain, looking into a flat terrain. How much would you need? 5 kilometers? 10? more?

Also, there won't be any terrain modification going on, which is why I really only need PolyVox when a terrain page is requested for the first time. For each additional request, the pre-created (and saved to disk) mesh will be used.

I guess I could split them a bit, anyway. Probably to 1000x500x1000. But I want to keep the number of SceneNodes in Ogre as low as possible (as there will be more then enough once all the trees/grass, etc. are in).

_________________
My site! - Have a look :)
Also on Twitter - with more ketchup


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Fri Jan 04, 2013 1:26 pm 

Joined: Thu Oct 06, 2011 2:26 pm
Posts: 46
Location: Berlin
Can't believe this whole thread is without pictures! ;)

Image

This is a 600x400x600 terrain (well, part of it), which is created in ~7 seconds. Which IMO is too much, but I have not done any threading or other optimization, so it is fine for the moment.

What bugs me, though, are the shadow "artifacts". You can clearly see each "line" of (previously) voxels in the mesh on the steep sides of mountains.
I guess that comes from me only applying height noise and no real 3D noise (yet), but does anyone know a faster way around that than also applying 3D noise? :)

The only idea I have to improve this would be to also apply some minor noise during surface extraction, which basically means writing my own surface extractor, which really is above my skill level with PolyVox and extraction algorithms.
I tried applying some minor noise during mesh creation, but that does not work as PolyVox outputs triangle lists and no strips or fans. Why is that, btw.? Triangle lists afaik are the slowest of them all, as all vertices are repeated multiple times.

Should I probably make a new topic for all of this? Seeing as this is not really related to LargeVolume any more.

_________________
My site! - Have a look :)
Also on Twitter - with more ketchup


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Fri Jan 04, 2013 3:24 pm 

Joined: Fri Sep 14, 2012 10:54 pm
Posts: 15
Those aren't artifacts.
The shadows are working the way they should. :)

Its more the type of mesh extracted by the Smoothsurface extractor. It extracts this steppy kind of terrain. You could iterate over the triangles and smooth them by averaging out all adjacent vertices i guess... Could smooth things out. But then a simplifaction of the mesh as a first step would be good. Those meshes are so freakin dense!


Top
Offline Profile  
Reply with quote  
 Post subject: Re: LargeVolume rather slow & wasting space?
PostPosted: Fri Jan 04, 2013 5:03 pm 

Joined: Thu Oct 06, 2011 2:26 pm
Posts: 46
Location: Berlin
KuroSei wrote:
Those aren't artifacts.
The shadows are working the way they should. :)

I know, that's why I put the "" around the artifacts ;)

KuroSei wrote:
Its more the type of mesh extracted by the Smoothsurface extractor. It extracts this steppy kind of terrain. You could iterate over the triangles and smooth them by averaging out all adjacent vertices i guess...

I could of course go over the vertices given by the surface extractor, but then I would also have to smooth the normals. That would also mean doing double work. The SurfaceExtractor already iterates over all the voxels, so it would be better to smooth things out there than to iterate over all vertices again once they are "out".

KuroSei wrote:
But then a simplifaction of the mesh as a first step would be good. Those meshes are so freakin dense!

You mean with a MeshDecimator?
Good idea, I could try that out.

_________________
My site! - Have a look :)
Also on Twitter - with more ketchup


Top
Offline Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Theme created StylerBB.net