2/3 times pressing a button does nothing.

@robertbu yes, thank you, that did it. I thought that calling it from FixedUpdate would make it more responsive because it would get called more regularly. Do you know why that's not the case?

Check this link , It explains the difference clearly .

Here is my guesstimation of the issue based on my visualization of how this all works (and why I made the suggestion to go to Update()). GetButtonDown() will be true during a single Update() frame. It may not even be the full deltaTime of that frame, just the duration of all the Update() calls for that frame. When the frame rate is high, or if the frame rate is uneven, then the frame that gets the GetButtonDown() can fall between FixedUpdate() calls. The default FixedUpdate() rate is 50 fps. Imagine that your Update() calls are running at 120 fps. Many Update() frames will start and end between FixedUpdate() calls, and these MouseDown events will be lost if you are only checking in FixedUpdate().

@robertbu I thought that I had looked at the page here so many times before, but I must not have ever carefully read it. http://docs.unity3d.com/Documentation/ScriptReference/30_search.html?q=GetButtonDown It specifically says to use Update. Thanks to your insights, I now understand why that is important.

I converted my comment into an answer. @Duck's answer does a through job of explaining. Thanks @Unity_gamer.