Threads and LargeVolume Paging

GM_Riscvul · **Joined:** Tue Feb 22, 2011 8:04 am **Posts:** 101

I've been working on a multi-threaded engine using polyvox.

I realize polyvox is not a thread safe library and so I have been designing my engine so only one operation should be using the volume at a time.

It worked beautifully for mesh decimation, however I have been struggling moving mesh extraction to the thread.

I have been getting access errors when I execute the surface extractor, which I can only assume is because I am somehow trying to open empty areas of the volume. They occur inside of LargeVolumeSampler.inl.

I was under the assumption that if an extraction was called and the area was not loaded that extraction would wait for the pageLoad function to complete its task before continuing. This doesn't seem to be happening. I never reach the breakpoint for my pageLoad code anymore. I am trying to make sure I understand so I can design this correctly.

Here is the section in question.

Code:

         worldLock.lock();
         MaterialAll = new SurfaceMesh<PositionMaterialNormal>();
         MaterialAll = extractMesh(MaterialAll, reg);

         Ogre::SceneNode* mManualObjectNode = mSystemDevices.mSceneMgr->getRootSceneNode()->createChildSceneNode(chunkID, Ogre::Vector3(0.0f, 0.0f, 0.0f));

         Ogre::ManualObject* manual = createManObj(MaterialAll,chunkID);

         mManualObjectNode->setScale(1.0f, 1.0f, 1.0f);
         mManualObjectNode->setVisible(true);
         mManualObjectNode->setPosition(reg.getLowerCorner().getX(),reg.getLowerCorner().getY(),reg.getLowerCorner().getZ());
         mManualObjectNode->attachObject(manual);

         requestDecimation(chunkID, MaterialAll);
         worldLock.unlock();

And the extraction function that is called.

Code:

inline SurfaceMesh<PositionMaterialNormal>* MeshManager::extractMesh(SurfaceMesh<PositionMaterialNormal>* mesh, Region reg)
{
   SurfaceExtractor<LargeVolume, MaterialDensityPair44> surfaceExtractor(mWorldVoxels, reg, mesh);

   surfaceExtractor.execute();

   return mesh;
}

I don't know if any code will help, but I'm getting rather stumped.

ker · **Posted:** Sun Oct 16, 2011 9:39 am

it looks right to me, even though i wonder where the MaterialAll object gets used and deleted

(inside createManObj ? or somewhere else, too?)

can you show a backtrace of the access errors?
or are they random?

did you try this in the main thread, too? would be interesting what happens if it's only in the main thread.

yes your function callback should be called.

GM_Riscvul · **Joined:** Tue Feb 22, 2011 8:04 am **Posts:** 101

MaterialAll gets used in createManObj and is deleted at the end of this function call.

I would show a backtrace, but I'm not sure how. The error is always in the same place in LargeVolumeSample.inl. I assume the code is working there, because it always has in the past. It must have something to do with the multi-threading.

The thing that confuses me is that I have a version that does pretty much the same thing for decimation that works perfectly. I just wanted to add extraction and things stopped working.

I used to run extraction in the main thread, and decimation in the second thread and it worked wonderfully. I wanted to add extraction since that caused a small delay as well, and that is when things stopped working.

Does that help clarify? How do I find a backtrace? I'm using Visual Studio 2010 if that helps.

David Williams · **Joined:** Sun May 04, 2008 6:35 pm **Posts:** 1827

I guess the first question is to make sure you have all the necessary mutex locking in you code. You've shown in the snippet above that you have worldLock.lock()/unlock() calls around the extraction code - do you also have them around around other places where you access the volume? Such as the loading or modification code?

If you do have the other calls then you are only ever touching the volume from one thread, and so I don't see how this could be threading related.

GM_Riscvul wrote:

I was under the assumption that if an extraction was called and the area was not loaded that extraction would wait for the pageLoad function to complete its task before continuing. This doesn't seem to be happening. I never reach the breakpoint for my pageLoad code anymore. I am trying to make sure I understand so I can design this correctly.

So your pageLoad function now never gets called? Can you work out why? I mean, it should be called from here:

Code:

// create the new block
LoadedBlock newBlock(m_uBlockSideLength);
itBlock = m_pBlocks.insert(std::make_pair(v3dBlockPos, newBlock)).first;

//We have created the new block. If paging is enabled it should be used to
//fill in the required data. Otherwise it is just left in the default state.
if(m_bPagingEnabled)
{
   if(m_funcDataRequiredHandler)
   {
      // "load" will actually call setVoxel, which will in turn call this function again but the block will be found
      // so this if(itBlock == m_pBlocks.end()) never is entered      
      //FIXME - can we pass the block around so that we don't have to find  it again when we recursively call this function?
      Vector3DInt32 v3dLower(v3dBlockPos.getX() << m_uBlockSideLengthPower, v3dBlockPos.getY() << m_uBlockSideLengthPower, v3dBlockPos.getZ() << m_uBlockSideLengthPower);
      Vector3DInt32 v3dUpper = v3dLower + Vector3DInt32(m_uBlockSideLength-1, m_uBlockSideLength-1, m_uBlockSideLength-1);
      Region reg(v3dLower, v3dUpper);
      ConstVolumeProxy<VoxelType> ConstVolumeProxy(*this, reg);
      m_funcDataRequiredHandler(ConstVolumeProxy, reg);
   }
}

So what if you set a breakpoint in there? Can you see why the function isn't being called?

Next up, there is a known bug with the LargeVolume class and it's possible this is affecting you. I don't quite see how, and I haven't actually seen this bug in practice (though it should be easy to reproduce). Probably I just have to fix the bug to eliminate this as a possibility.

GM_Riscvul wrote:

The thing that confuses me is that I have a version that does pretty much the same thing for decimation that works perfectly.

Right, but the decimation doesn't involve touching the volume.

GM_Riscvul wrote:

How do I find a backtrace? I'm using Visual Studio 2010 if that helps.

Look at the callstack window. This shows the function you have crashed in, and the functions you called to get there.

GM_Riscvul · **Joined:** Tue Feb 22, 2011 8:04 am **Posts:** 101

I've looked over my code and I have locks on the two places that can perform extractions, but they should never take place at the same time since they are part of the same thread. I don't have mutexs anywhere else but I can't think of any way my program would touch the volume except for extraction. I do not have any modification at the moment, just extracting and rendering meshes.

There is also the loadChunk function, but it should be called by the extraction operation shouldn't it?

Here is the call stack.

Code:

>   WorldCraft.exe!PolyVox::LargeVolume<PolyVox::MaterialDensityPair<unsigned char,4,4> >::Sampler::setPosition(int xPos, int yPos, int zPos)  Line 144 + 0x5 bytes   C++
    WorldCraft.exe!PolyVox::SurfaceExtractor<PolyVox::LargeVolume,PolyVox::MaterialDensityPair<unsigned char,4,4> >::computeBitmaskForSlice<0>(const PolyVox::Array<2,unsigned char> & pPreviousBitmask, PolyVox::Array<2,unsigned char> & pCurrentBitmask)  Line 153   C++
    WorldCraft.exe!PolyVox::SurfaceExtractor<PolyVox::LargeVolume,PolyVox::MaterialDensityPair<unsigned char,4,4> >::execute()  Line 76   C++
    WorldCraft.exe!MeshManager::extractMesh(PolyVox::SurfaceMesh<PolyVox::PositionMaterialNormal> * mesh, PolyVox::Region reg)  Line 211   C++
    WorldCraft.exe!MeshManager::operator()()  Line 50 + 0x2c bytes   C++
    WorldCraft.exe!boost::detail::thread_data<MeshManager>::run()  Line 57   C++
    WorldCraft.exe!boost::`anonymous namespace'::thread_start_function(void * param)  Line 168   C++

Here is the section of code it breaks on.

Code:

      
const int32_t uXBlock = mXPosInVolume >> mVolume->m_uBlockSideLengthPower;
const int32_t uYBlock = mYPosInVolume >> mVolume->m_uBlockSideLengthPower;
const int32_t uZBlock = mZPosInVolume >> mVolume->m_uBlockSideLengthPower;

Also its possible the load function is not being called because it was previously loaded. I extract meshes slightly larger than the area so vertices will be generated for the edges of the area I want displayed. This causes Polyvox to load more chunks than necessary. It might have already loaded the area being extracted. however I would think it would still need to call my datahandler function. I haven't tried putting a breakpoint in your code yet, but it isn't breaking inside my function.

Where is it called? I can't find the code in SurfaceExtractor.inl or LargeVolumeSampler.inl

Quote:

Right, but the decimation doesn't involve touching the volume.

Haha duh. I feel intelligent. :roll:

I hope we are able to trace this down. Thank you for the help so far.

ker · **Posted:** Mon Oct 17, 2011 6:23 am

GM_Riscvul wrote:

Also its possible the load function is not being called because it was previously loaded. I extract meshes slightly larger than the area so vertices will be generated for the edges of the area I want displayed. This causes Polyvox to load more chunks than necessary. It might have already loaded the area being extracted. however I would think it would still need to call my datahandler function.

extraction works before the crash? how could it be preloaded?
i thought your datahandler was never called.

question: does the main thread create the extraction thread before it creates the volume? (sorry if that was stupid, just making sure, i've def done it before

)
or is the volume created inside the extraction thread?
the crash looks more like a crash on an uninitialized object.

i do not see how the crash could happen. bitshift cant be it, so it has to be the volume pointer or the this pointer. and i do not see how either could be corrupted

GM_Riscvul · **Joined:** Tue Feb 22, 2011 8:04 am **Posts:** 101

This is how the scene is started.

Code:

   mWorldManager = new WorldManager(mKeyDevices);
   mWorldManager->initialize(4);
   mWorldManager->createTestWorld();

The constructor stores the devices pointer and creates the second thread class.

Code:

   WorldManager(device_info pSystem)
   {
      mSystemDevices = pSystem;
      mDecimator = new MeshManager(pSystem, mWorldVoxels);
   }

The constructor for the thread class takes a pointer to the unitialized world. This could be a problem, but I thought the pointer would contain the address and when the world is initialized this pointer would be pointing to the initialized value.

Code:

MeshManager::MeshManager(device_info pSystem, LargeVolume<MaterialDensityPair44>*& pWorldVoxels, list<string>* pMeshPool)
{
   mSystemDevices = pSystem;
   programStop = false;
   mWorldVoxels = pWorldVoxels;
   mMeshPool = pMeshPool;
}

Then Initialize is called. This sets some test values, binds the paging functions, initializes the LargeVolume, and then starts the second thread. At this point I expect the pointer in the thread to be pointing to this now initialized LargeVolume.

Code:

void WorldManager::initialize(int worldSize)
{
   //Currently does not load player location from disc
   mPlayerLocation.x = 5;
   mPlayerLocation.y = 30;
   mPlayerLocation.z = 5;
   mViewSize = worldSize;

   mSystemDevices.mSceneMgr->getCamera("FirstPerson")->setPosition(mPlayerLocation);
   oldCameraPosition.x = 0;
   oldCameraPosition.y = 0;
   oldCameraPosition.z = 0;

   //These functions are used by PolyVox for paging. Need to be bound to avoid object issues.
   boost::function<void (const ConstVolumeProxy<MaterialDensityPair44>&, const Region&)> polySaver  (boost::bind( &WorldManager::saveChunk, this, _1, _2 ));
   boost::function<void (const ConstVolumeProxy<MaterialDensityPair44>&, const Region&)> polyLoader (boost::bind( &WorldManager::loadChunk, this, _1, _2 ));

   mWorldVoxels = new LargeVolume<MaterialDensityPair44>(polyLoader,polySaver,32);
   //Start Decimator Thread
   boost::thread mMeshManager(*mDecimator);
}

Lastly the create world function is called. This is why I have some areas loaded. As I have not finished the extractor thread I have left the test code in for building the initial area. It calls a function that extracts an area and creates a manual object from it. I am moving this functionality to the thread once it is working, but I have left this here as it is stable and helps testing. My load function is called when generating these pieces of terrain.

Code:

void WorldManager::createTestWorld()
{
   //Create a world of 5 blocks for testing purposes
   createManObj(*(new Region(*(new Vector3DInt32(0,0,0)), *(new Vector3DInt32(32,31,32)))), "0x0y0z");
   meshPool.push_back("0x0y0z");
   createManObj(*(new Region(*(new Vector3DInt32(0,0,-32)), *(new Vector3DInt32(32,31,0)))), "0x0y-1z");
   meshPool.push_back("0x0y-1z");
   createManObj(*(new Region(*(new Vector3DInt32(31,0,0)), *(new Vector3DInt32(64,31,32)))), "1x0y0z");
   meshPool.push_back("1x0y0z");
   createManObj(*(new Region(*(new Vector3DInt32(-32,0,0)), *(new Vector3DInt32(0,31,32)))), "-1x0y0z");
   meshPool.push_back("-1x0y0z");
   createManObj(*(new Region(*(new Vector3DInt32(0,0,31)), *(new Vector3DInt32(32,31,64)))), "0x0y1z");
   meshPool.push_back("0x0y1z");
}

the createManObj function they are calling is what I based the code in my thread off of. It shouldn't be any different. I can post it if someone thinks it would help.

I hope this answered all the questions. Could this be related to my initialization?, or did I pass the pointer incorrectly? I thought I used this pattern successfully before, but perhaps not.

I'm at a loss as to whats wrong. The code looks good (to me) in theory. If this ends up having nothing to do with Polyvox I'm sorry, but I'm trying to rule out ignorance of some LargeVolume aspect.

ker · **Posted:** Tue Oct 18, 2011 5:47 am

GM_Riscvul wrote:

This is how the scene is started.

Code:

   WorldManager(device_info pSystem)
   {
      mSystemDevices = pSystem;
      mDecimator = new MeshManager(pSystem, mWorldVoxels);
   }

The constructor for the thread class takes a pointer to the unitialized world. This could be a problem, but I thought the pointer would contain the address and when the world is initialized this pointer would be pointing to the initialized value.

this is very wrong. let me explain.

Code:

int* p1; // contains random address
int* p2 = p1; // p2 contains same random addr
p1 = new int; // p1 gets valid address to memory
*p1 = 5; // assign 5 to that memory
*p2 = 6; // assign 6 to random memory addr (CRASH or very bad random errors)

you cannot do this this way.
why don't you simply make the mWorldVoxels an object instead of a pointer and pass a reference to the object?
or in your case, initialize it before passing it (and if you do, please have a look at std::unique_ptr and similar, it will prevent this kind of error, there are even thread safe pointer classes)

also, c++11 has std::thread and std::function and std::bind which are equivalent to the boost versions.

GM_Riscvul · **Joined:** Tue Feb 22, 2011 8:04 am **Posts:** 101

Thanks ker!

Oh duh! Thats a pointer 101 error.

When you spelled it out for me with your example I felt pretty stupid.
I now initialize the MeshManager inside of the initialize funciton and after I initialize the LargeVolume.

That seems to have fixed the errors regarding the volume but opens up one other error. This doesn't seem related to PolyVox, but perhaps someone has seen this error before. I can't seem to find much information on google about it.

Code:

14:49:50: OGRE EXCEPTION(7:InternalErrorException): Cannot create GL vertex buffer in GLHardwareVertexBuffer::GLHardwareVertexBuffer at ..\..\..\..\..\RenderSystems\GL\src\OgreGLHardwareVertexBuffer.cpp (line 46)

It occurs when I try to call manual->end() in my thread. The code is below. I don't really expect an answer since this isn't about PolyVox at all, but if someone knows what this is about I would appreciate a tip.

Code:

Ogre::ManualObject* MeshManager::createManObj(SurfaceMesh<PositionMaterialNormal>* mesh, string name)
{
   Ogre::ManualObject* manual = mSystemDevices.mSceneMgr->createManualObject(name);

   if(mesh && mesh->getNoOfVertices() > 0 && mesh->getNoOfIndices() > 0)   // make sure mesh is allocated and valid first
   {
      uint32_t numOgreIndices;                              // for recording and reporting # of ogre indices/vertices
      uint32_t numOgreVertices;
      const vector<uint32_t>& indexVector = mesh->getIndices();                     // get a vector of all indices
      const std::vector<PolyVox::PositionMaterialNormal>& vertexVector = mesh->getVertices();   // get a vector of all vertices

      numOgreIndices = mesh->getNoOfIndices();
      numOgreVertices = mesh->getNoOfVertices();

      manual->estimateIndexCount(numOgreIndices);   // set index/vertex counts before starting to speed up allocations
      manual->estimateVertexCount(numOgreVertices);

      manual->begin("Worldcraft/Greengrass", Ogre::RenderOperation::OT_TRIANGLE_LIST);   // start adding triangles 

      PolyVox::Vector3DFloat positionTempVar;
      PolyVox::Vector3DFloat normalTempVar;

      // iterate through vertices
      for(int v = 0 ; v < numOgreVertices ; v++)
      {
         positionTempVar = vertexVector[v].getPosition();   // get position vector from vertex list
         normalTempVar = vertexVector[v].getNormal();

         manual->position((Ogre::Real)(positionTempVar.getX()), (Ogre::Real)(positionTempVar.getY()), (Ogre::Real)(positionTempVar.getZ()));   // add a new vertex to ogre manualobject
         manual->normal((Ogre::Real)(normalTempVar.getX()),(Ogre::Real)(normalTempVar.getY()),(Ogre::Real)(normalTempVar.getZ()));
      }

      // iterate through indices
      for(int i = 0 ; i < numOgreIndices ; i++)
      {
         manual->index((Ogre::uint32)(indexVector[i]));   // set indices in same order as polyvox mesh (if using DirectX render system, may differ for OpenGL)
      }

      manual->end();   // stop adding triangle
   }

   return manual;
}

ker · **Posted:** Wed Oct 19, 2011 7:50 am

just as a failsafe, test whether

Code:

numOgreIndices = mesh->getNoOfIndices();
numOgreVertices = mesh->getNoOfVertices();

are equal to

Code:

indexVector.size();
vertexVector.size();

during development, try to use .at() instead of [] on std::vector
it will make sure you don't access random memory there

try not setting any normals, just positions.

don't use the estimate*count functions during development. they are optimizations.

check whether the number of vertices is below 65000 (2^16), there are some short vs int things inside ogre.

are you sure an index is of type uint32? try no casting, or casting to uint16.

Threads and LargeVolume Paging

Who is online