Networked vehicle physics - to rigidbody or not rigidbody?

I'm trying to find the best approach for networking vehicle physics.
Right now I have a rigidbody on the parent and everything works fine non-networked, but the problem appears when trying to get this to work over a network (using Mirror).
Yes physics and networking do not mix well. So what approach would you use for this?

-Ditch the RB and use raycasts for collision detection? How would you simulate all other physics in general?
-Keep the RB? Then how would you approach collisions on all other players with local authority?

I see people on forums speaking of how ditching Unity's physics and simulating your own is a simple task, yet I haven't found any good examples of anybody doing that.

Thanks for the help -- I'd appreciate any insight