Volumes Of Fun
http://www.volumesoffun.com/phpBB3/

Dynamic Linking with MinGW and CMake
http://www.volumesoffun.com/phpBB3/viewtopic.php?f=14&t=487		 Page 1 of 1
Author:  holocronweaver [ Thu Feb 21, 2013 5:27 pm ]
Post subject:  Dynamic Linking with MinGW and CMake
Hello all. I am trying to build an Ogre project on Windows which dynamically links to PolyVox (dev branch). My build system is based on CMake and I am using MinGW to compile. I can get my project to compile, but linking fails with the following error:
Quote:
c:/mingw/bin/../lib/gcc/mingw32/4.7.2/../../../../mingw32/bin/ld.exe: cannot find -lPolyVoxCore
c:/mingw/bin/../lib/gcc/mingw32/4.7.2/../../../../mingw32/bin/ld.exe: cannot find -lPolyVoxUtil

Looking at the build.make file generated by CMake, I notice that none of the PolyVox DLL files are listed. Investigating the CMake/PolyVoxConfig.cmake file identifies the cause of the problem: no DLL files are being searched for and included. I can add in the necessary checks, and if no one objects will create a merge request with the modified CMake files, but before I do I thought I would inquire as to why this has not already been done.

In addition, I had to remove this section from the PolyVox CMakeLists file to allow a dynamic bulid on Windows:
Code:
IF(WIN32)
   SET(LIBRARY_TYPE "STATIC")
ENDIF()

Thus it seems dynamic libraries are disabled in Windows. Why is that?
Author:  David Williams [ Fri Feb 22, 2013 9:54 am ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
This has come up a few times before and I always struggle to remember the exact situation. But I looked at the commit comment for the line in question and is had a link to this thread:

http://www.volumesoffun.com/phpBB3/viewtopic.php?p=3203#p3203

In summary, we used to build both static and dynamic versions of PolyVox whenever you did a single build. This was OK on Linux where it generated seperate '.a' and '.so' files, but it was painful on Windows because both static and dynamic versions generated conflicting '.lib' files (as well as a .dll for the dynamic build).

It looks like this is the reason that dynamic libraries were disabled on Windows, though the argument has changed a litle now because you can choose a build type rather than trying to build both.

That thread also links to this one:

http://www.volumesoffun.com/phpBB3/viewtopic.php?p=1584#p1584

In this thread we basically decided that there was no advantage to having a dynamic library on Windows and that it caused some complications, so we decided that PolyVox on Windows should be static-only. On Linux we maintained the dynamic option because Linux users are more used to them and because some package repositories prefer/require them.

So, I guess the real question now is why you need to use a dynamic version of PolyVox rather than the static one. Can you not use the static library instead? If there is a real need for the dynamic version then we can consider bringing it back, but my main concern is that it's somthing extra to test and maintain (as well as the concerns in that thread).

That said, the current situation is clearly confusing as CMake offers the option of a dynamic build and then doesn't actually do it. This even caught me out a few weeks ago when I claimed that both worked on Windows, without realising that the dynamic build was being converted to static behind my back:

http://www.volumesoffun.com/phpBB3/viewtopic.php?p=3629#p3629

So yeah, we should make this clearer at least. I've added an issue so this doesn't get forgotten:

https://bitbucket.org/volumesoffun/poly ... on-windows
Author:  holocronweaver [ Fri Feb 22, 2013 3:52 pm ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
David Williams wrote:
So, I guess the real question now is why you need to use a dynamic version of PolyVox rather than the static one. Can you not use the static library instead? If there is a real need for the dynamic version then we can consider bringing it back, but my main concern is that it's somthing extra to test and maintain (as well as the concerns in that thread).

I generally prefer dynamic linking whenever possible. It is a matter of convenience and simplicity. I would like to have a single repository containing all my library files and use those for every project. It saves disk space, allows me to easily check that library updates cause no problems in my projects without recompiling them, and allows me to update all my projects simultaneously with a single file change. In addition, it allows my users to swap out or patch libraries to their liking. Even if I were to fall off the face of the Earth and my users stranded without source code access, they could at least fix bugs arising from the libraries.

Also, though this is likely a personal foible, the non-uniformity of having all my libraries dynamically linked except PolyVox irks me. :)

I can certainly understand defaulting to static builds on Windows, but disallowing dynamic builds all together seems a bit much. I realize that dynamic linking in Windows can be awry, but I still feel the above mentioned benefits outweigh the costs. Besides, most people wanting to do the dynamic linking are just going to modify the CMake files as I have, so why make them go through the hassle? If you prefer not to support and test dynamic linking, you could include a warning message in the CMake output stating so. (BTW: dynamic linking to the PolyVox dev branch currently works fine on Windows :D )

The only part I do not understand is how dynamic linking effects SWIG (as mentioned in the second link). Could you elaborate a bit? If it does cause problems, this too could be mentioned in a CMake warning. I am guessing this is a Windows-specific issue since I do not recall encountering it on Linux.
Author:  David Williams [ Sat Feb 23, 2013 7:38 am ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
holocronweaver wrote:
...allows me to easily check that library updates cause no problems in my projects without recompiling them, and allows me to update all my projects simultaneously with a single file change. In addition, it allows my users to swap out or patch libraries to their liking. Even if I were to fall off the face of the Earth and my users stranded without source code access, they could at least fix bugs arising from the libraries.


This sounds good in theory but doesn't work in practice. You see, PolyVox is almost entirely header files, and these will be compiled directly into your application regardless of whether you are linking statically or dynamically. Have a look at the number of header files (.inl files are also headers) here:

https://bitbucket.org/volumesoffun/polyvox/src/95f0aa22c12dde4b6e145f6b057a84cee006a5b0/library/PolyVoxCore/include/PolyVoxCore?at=master

versus the number of source files here:

https://bitbucket.org/volumesoffun/polyvox/src/95f0aa22c12dde4b6e145f6b057a84cee006a5b0/library/PolyVoxCore/source?at=master

If you open a few source files you'll see they mostly have very little code. The only significant stuff that gets compiled into a library is the Region class and static data such as the Marching Cubes tables.

All significant code such as volumes and surface extractors are simply in header file which are included by your application and compiled into it. This is done because PolyVox is highly templatised and including definitions in header files is the only practical way to do templates (C++'s Exported Templates feature was never widely supported and is now removed from the standard).

holocronweaver wrote:
I can certainly understand defaulting to static builds on Windows, but disallowing dynamic builds all together seems a bit much. I realize that dynamic linking in Windows can be awry, but I still feel the above mentioned benefits outweigh the costs.


Except it seems those benefits don't really exist (as described above). Updating the .dll will just trick you into thinking you are updating PolyVox. It doesn't update any significant part of PolyVox but may mean you have a mismatch between the code that has been compiled into your executable and the data in your .dll.

holocronweaver wrote:
Besides, most people wanting to do the dynamic linking are just going to modify the CMake files as I have, so why make them go through the hassle? If you prefer not to support and test dynamic linking, you could include a warning message in the CMake output stating so.


I was going to agree with you, but then I started writing the arguments above and convinced myself otherwise :-) It might be bad if users do this because they convince themselves they are getting benefits which they are not really. We could actually prevent it by removing the '__declspec(dllexport)' which Visual C++ requires for building a .dll (though I'm not saying we should... maybe a warning is enough).

holocronweaver wrote:
The only part I do not understand is how dynamic linking effects SWIG (as mentioned in the second link). Could you elaborate a bit? If it does cause problems, this too could be mentioned in a CMake warning. I am guessing this is a Windows-specific issue since I do not recall encountering it on Linux.


I forget the exact details, but there were problems with SWIG and the '__declspec(dllexport)' defined by POLYVOX_API. So each class now has to start with some logic like this:

Code:
#ifdef SWIG
        class Region
#else
        class POLYVOX_API Region
#endif
        {
        public:


Not a huge problem as we've solved it already, but still a bit ugly.
Author:  holocronweaver [ Sat Feb 23, 2013 8:44 pm ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
David Williams wrote:
It might be bad if users do this because they convince themselves they are getting benefits which they are not really. We could actually prevent it by removing the '__declspec(dllexport)' which Visual C++ requires for building a .dll (though I'm not saying we should... maybe a warning is enough).


That is a good point. I think a warning is indeed the way to go, along with static linking being the default method for Windows. If it ever becomes tedious to support dynamic linking on Windows, you could issue a warning that dynamic linking is untested. That way it will be up to the developer whether dynamic linking is really worth the potential extra pain.

I tried switching to a static build of PolyVox out of curiosity, but again no library files were added to my build.make (same problem I mentioned in my first post). I am adding PolyVox to my project using the following lines in my CMakeLists file:

Code:
find_package(PolyVox REQUIRED)

if(NOT PolyVox_FOUND)
  message(SEND_ERROR "Failed to find PolyVox.")
endif()

...

include_directories(
  ${OGRE_INCLUDE_DIRS}
  ${OGRE_SAMPLES_INCLUDEPATH}
  ${OIS_INCLUDE_DIRS}
  ${PolyVox_INCLUDE_DIRS}
)

...

target_link_libraries(
  ${APPLICATION_NAME}
  ${OGRE_LIBRARIES}
  ${OIS_LIBRARIES}
  ${PolyVox_LIBRARIES}
)


Should I be doing something more?
Author:  David Williams [ Mon Feb 25, 2013 9:53 am ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
Right, so I haven't really answered your questions about the MingW build problems yet. Actually I'm not that experienced with CMake (Matt wrote most of the scripts) and I am a little confused. You said that no library files are added to build.make (in both static and dynamic modes) but you also said "dynamic linking to the PolyVox dev branch currently works fine on Windows". So, as I understand it you have been able to work around the build issues but you are wondering why it doesn't happen automatically?

holocronweaver wrote:
I am adding PolyVox to my project using the following lines in my CMakeLists file:
.
.
.
Should I be doing something more?


I'm not sure, but I guess my first question would be whether the PolyVox examples build as expected? We never test on MingW but we go run nightly tests on Linux and it should be similar.

Btw, I broke the dev branch on Friday so watch out for that. I'll try to fix it soon.
Author:  David Williams [ Mon Mar 04, 2013 3:35 pm ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
Hi, I was just wondering what you ended up doing in the end? I ask because I'm having another look at which warnings we disable when building PolyVox, and now the only warning we still disable is:

Code:
warning C4251: ... needs to have dll-interface to be used by clients of class...


This warning happens a lot when building PolyVox as a .dll (except it's never seen because it's disabled). But having done some research I'm increasingly of the opinion that what we currently do is not a good solution. There is some relevant information here:

http://www.unknownroad.com/rtfm/VisualStudio/warningC4251.html

The conclusion is basically that there is no good solution, and crashes can occur if care is not taken. Also this post is useful:

http://stackoverflow.com/a/5664491

The whole area is rather complex, but basically mixing templates with .dlls is messy (especially if your .dll is compiled with a different compiler/version than your application).

Having reviewed this information, I'm now again inclined to remove the dynamic option for PolyVox on Windows, unless anyone can provide a compelling argument otherwise?
Author:  holocronweaver [ Thu Mar 21, 2013 2:05 am ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
Sorry for the belated reply - for some reason I never received an e-mail notification about your last couple posts.

I have tried both static and dynamic linking and they worked equally well in my project so long as I link all my libraries in the same way. However, CMake has trouble generating MinGW makefiles when I try statically linking some libraries and dynamically linking others. As mentioned above, I would prefer to dynamically link my libraries, so currently I switched to pure dynamic linking until I figure out a way around the problems with CMake mixing linking types.

If I understand correctly, the warning you show is for VisualStudio. Does a similar warning occur when you dynamically link PolyVox using gcc or MinGW?

I was not aware of troubles with C++ templates and dynamic linking while using multiple compilers. Good to know! However, I cannot think of an occasion where I would want to use multiple compilers in a single project. That seems like a major headache and I would certainly never do it if I want to dynamically link my libraries. I think this is something the PolyVox user needs to be careful to avoid, so perhaps a warning could be introduced rather than disabling dynamic builds.
Author:  David Williams [ Thu Mar 21, 2013 10:44 am ]
Post subject:  Re: Dynamic Linking with MinGW and CMake
holocronweaver wrote:
If I understand correctly, the warning you show is for VisualStudio. Does a similar warning occur when you dynamically link PolyVox using gcc or MinGW?


I believe it's a Visual Studio issue though I've never tested MinGW. I'm pretty sure it doesn't occur on GCC on Linux, and I'd imagine it was related to the compiler infrastructure rather than the operating system.

Before doing anything too radical here I will do some further investigation to find out what GCC does and how/why it is different to Visual Studio.

holocronweaver wrote:
I was not aware of troubles with C++ templates and dynamic linking while using multiple compilers.


Neither was I, it just goes to show how complex C++ can be sometimes!

holocronweaver wrote:
However, I cannot think of an occasion where I would want to use multiple compilers in a single project. That seems like a major headache and I would certainly never do it if I want to dynamically link my libraries.


Long-term we've talked about providing a prebuilt SDK, and we can't practically build .dlls for every Vuisual Studio version and patch version. But we can avoid this problem by just including static libs in such a prebuilt SDK, and allowing users to build the dynamic version themself if they prefer.

Anyway, it seems I should do some more research to determine what is really going on and what the implications are. It's logged as a PolyVox issue so I'll get to it at some point :-)
Page 1 of 1 All times are UTC
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/