Networked vehicle physics - to rigidbody or not rigidbody?

I'm trying to find the best approach for networking ve$$anonymous$$cle physics.
Right now I have a rigidbody on the parent and everyt$$anonymous$$ng works fine non-networked, but the problem appears when trying to get t$$anonymous$$s to work over a network (using Mirror).
Yes physics and networking do not mix well. So what approach would you use for t$$anonymous$$s?

-Ditch the RB and use raycasts for collision detection? How would you simulate all other physics in general?
-Keep the RB? Then how would you approach collisions on all other players with local authority?

I see people on forums speaking of how ditc$$anonymous$$ng Unity's physics and simulating your own is a simple task, yet I haven't found any good examples of anybody doing that.

Thanks for the help -- I'd appreciate any insight