Raycast hit or miss in the same situation

I have a player controller and I am sending a ray from the origin of the camera. What I am trying to do is be able to look at a safe and if I press E, a dialogue pops up. But I have a really interesting problem that I have no idea how to fix. It will work sometimes and sometimes it will just not work. The really interesting thing is that even if I don't move the cameras position or rotation AT ALL, there is still no telling whether it will execute or not.

To demonstrate this, I shot a raycast 50 times from the exact same position.

http://i.imgur.com/9ivKTEb.png

Here are my results

http://i.imgur.com/u1mWJQ9.png

And finally here is my code

     void sendRay(){
         Camera camera = cam.gameObject.GetComponent<Camera>();
         Ray ray = cam.gameObject.GetComponent<Camera>().ScreenPointToRay(new Vector3(camera.pixelWidth * 0.5F, camera.pixelHeight * 0.5F, 0));
         RaycastHit hit;
         if(Physics.Raycast(ray, out hit, 3)){
             print("hit");
             if(hit.transform.tag == "safe"){
                 safe.enter();    
             }
         }else{
             print("no hit");
         }
     }