Control character without CharacterController or

"The trouble is CharacterControllers can only rotate on the Y axis and Rigidbodys don't move with their parents."

this doesn't make any sense.

But if colliders are a problem try making sure your translate distances are small enough that you don't skip through objects, or make the objects fatter. You can always try using Raycasting to detect the distance you are from an object as well.

you're essentially circumventing the entire built in physics system. I understand wanting to do that but I don't recommend it. Working with a rigidbody is doable while still circumventing the dumber parts of the physics system.