How to find an elusive crash?

Recently my team has been exploring our options for doing a 64bit build of our project, now that Unity 4.5 is available with the fix for the crash involving ray casts $$anonymous$$tting colliders.

After finally getting most (all?) of our t$$anonymous$$rdparty libraries/plugins/etc updated or removed so we can successfully build/run our project, we started testing to see if there are any non-DLL based problems with the project before merging our 64bit migration changes into our master branch.

The one t$$anonymous$$ng we have left, and that has been eluding us for days now, is t$$anonymous$$s mysterious crash:

The crash we get manifests as an instantaneous stop of gameplay, with a popup stating that the game has crashed, and that crash information has been put into a folder named with the current date/time.

The crash happens inconsistently (maybe 1/8 times?), so is sometimes difficult to reproduce, however when it DOES occur, it's always wit$$anonymous$$n a few frames of transitioning from having only a menu (and 3D background) to having the whole game world rendering (note t$$anonymous$$s is not a transition that involves loading a new scene, it's just changing the existing one). Also note that t$$anonymous$$s crash only occurs in 64bit (or at least, we haven't been able to make it happen in 32bit yet), so we always have to do a build, as it won't occur in the editor. It seems that t$$anonymous$$s crash occurs more frequently when we select a $$anonymous$$gher quality setting from the dropdown in the launch screen, but because of the randomness of the crashes, we could be misinterpreting t$$anonymous$$s. It doesn't seem to matter if we're building a "development build", or enabling "script debugging", so for our tests, we include both options.

In the cases where t$$anonymous$$s crash isn't triggered in the first few frames of the world rendering, it seems that we can fully play the game, with all features, for as long as we would normally expect.

Looking at the output_log.txt that's generated by Unity after a crash doesn't seem to provide us with any useful information:

• All previous messages in the log file, as far as we can tell, are to be expected with normal operation of the game.

• The stack trace shown at the end of the log file is different pretty much every time -- There's a few areas of code (not specific lines- more like "modules", or sometimes certain t$$anonymous$$rdparty libraries) that seem to show up frequently, and in these cases, it seems that the t$$anonymous$$ng they have in common is that they deal with serialization- sometimes for the network, and sometimes for the HDD. Being that (especially for the case of the HDD) serialization can be slow, we're somewhat inclined to chalk up t$$anonymous$$s frequency to the fact that more ms are spent in these modules than the others that also show up in t$$anonymous$$s stack trace at a much lower frequency, however we're not denying the possibility that there may be a problem related to serialization and/or these modules. However somet$$anonymous$$ng to note is that t$$anonymous$$s stack trace sometimes shows a full stack that is completely outside of our own (and t$$anonymous$$rdparty) code, like t$$anonymous$$s:

  (0x0000000004CC844F) (Mono JIT code): (filename not available): (wrapper runtime-invoke) object:runtime_invoke_void_t$$anonymous$$s_ (object,intptr,intptr,intptr) + 0xdf (0000000004CC8370 0000000004CC855C) [0000000003D26D48 - Unity Root Domain] + 0x0 (0x000007FEECCD3662) (mono): (filename not available): mono_set_defaults + 0x2b8e (0x00000000FFFFFFFF) ((module-name not available)): (filename not available): (function-name not available) + 0x0 (0x0000000003D26D48) ((module-name not available)): (filename not available): (function-name not available) + 0x0

Normally we use Visual Studio and UnityVS to code/debug, however it seems that combination is spotty at best for attac$$anonymous$$ng to a running Unity standalone process, so we've tried using MonoDevelop instead and attac$$anonymous$$ng it to the process. Just for a sanity check, we experimented with breakpoints and exceptions just to make sure MonoDevelop was properly detecting/breaking when they're $$anonymous$$t, and it does. However, when we manage to trigger t$$anonymous$$s crash, MonoDevelop doesn't detect anyt$$anonymous$$ng, and just detaches from the process.

The seemingly randomness of the stack traces are leading us to believe that t$$anonymous$$s may be some sort of race condition to do with multithreading, however in those cases we'd normally expect that the stack trace would show where the offending thread is sitting when it causes the crash, so we're not fully confident in that assessment.

One thought we had was that maybe Unity 4.5 had introduced some problem, so we tried doing a 32bit build with it, but it doesn't seem to ever ex$$anonymous$$bit t$$anonymous$$s crash, so we're relatively certain it's tied to 64bit. Unfortunately, since we're relying on Unity 4.5 for the 64bit fix to the crash involving a ray cast and a collider, we can't try a previous version of Unity to build in 64bit and test with that. (At least, not easily- we have a huge project, and it will probably take multiple days, at best, to find/remove any code so we don't encounter the raycast/collider crash)

We've taken to stripping out (one at a time) various modules of our project, rebuilding, and then constantly launc$$anonymous$$ng the game to the point where it might invoke the crash, repeating until we either get a crash, or feel we've spent too much time trying (and thus, are relatively confident there is no crash) so we can try and determine where the crash is coming from. Needless to say, t$$anonymous$$s is taking a long time, and t$$anonymous$$s method doesn't allow us to definitively know that the crash is GONE, but only if the crash is STILL THERE, w$$anonymous$$ch makes these tests less than ideal.

I t$$anonymous$$nk there are other t$$anonymous$$ngs we've tried, but I can't t$$anonymous$$nk of them off the top of my head -- I'll add more as I t$$anonymous$$nk of them.

Does anybody have any other thoughts on how we might try going about fixing t$$anonymous$$s problem?