Determining rotation for collision avoidance?

Having done a "bit" of t$$anonymous$$s stuff over the years, I agree with Ricardo's assessment cautioning you against extensive use of ray casting for navigation. The corner problem that you are encountering is a perfect example where ray casting, and more generally, purely reactive steering becomes problematic. For example, in corners, as you are turning away from the corner you typically wind up moving closer to one side of the corner or the other. A purely reactive solution (e.g., turn in the direction of most space) will cause you to move back into the corner and cause the oscillation you are seeing.

As Ricardo mentions, Mikko Mononen's website has a superb set of links to the latest papers in the field, most of w$$anonymous$$ch are based on a concept known as velocity objects (and variants such as reciprocal velocity objects). The big idea of velocity objects is that, based on the relative motion of 2 objects, it divides velocity space into those velocities (speed and direction) that are collision-free over some time horizon, and those velocities that will result in a collision over that time horizon. When tied to a planner that provides, in effect, a preferred velocity (namely one that will get you to the goal position), you can arrive at the closest velocity to your preferred velocity that will still be collision-free over the time horizon. At the end of the day you need a robust local steering (collision-avoidance) system as well as a nav. planner of some sort.