Unity Faux gravity on a cube: reaching some faces is troublesome

My goal is to move a small cube on the surface of a big one, like a big cube shaped planet.

If the small cube moves forward only (or backwards only) it works fine, it goes around. If it starts going left or right, it can go the adjacent side. From there it can't go to any other sides without glitching except where you came from. Something like this, green means moving without a problem: So if I press D, the little cube goes from side 1 to side 3. On side 3 if I press W it starts moving towards side 2. However if reaches the edge, it starts glitching. If I start pressing the buttons it can go to side 2 but it turns 90 degrees there like it was going from side 1 to 2. The same thing happens if I go from side 3 to the bottom only it turns 180 degrees like it's going from side 1 to the bottom through side 2.

I'm using this to move the little cube:

 transform.Translate(horizontal * Time.deltaTime, 0f, vertical * Time.deltaTime);

The horizontal and vertical variables can be 0, 1 or -1. This is what gets the little cube from one side to the other:

 Vector3 direction = (centreOfGravity.position - transform.position).normalized;
     RaycastHit hit;
     if (centreOfGravity.GetComponent<Collider>().Raycast(new Ray(transform.position, direction), out hit, maxDistance))
     {
         transform.rotation = Quaternion.FromToRotation(Vector3.up, hit.normal);
         rb.AddForce(-hit.normal * gravity * rb.mass);
     }

What causes this problem, how can I fix it?