Collider not working...

In my game I have a player, and 4 walls. The player moves around the screen until touching one of the walls. The walls move as well as the player.

I have IsTrigger set on all four walls, and the following code applied to all four walls, but not the player:

 function OnTriggerEnter (other : Collider) {
     Destroy(other.gameObject);
     print("Code ran.");
 }

The player has a rigidbody, but the walls don't.

When the walls hit the player, it sometimes works.

If the player stays still (and hasn't moved throughout the whole game), the wall will just go straight through the player. However if the player moves, the player will be destroyed. What is causing the wall to go straight through the player?