Hinge Joint Effected by Colliders

I'm making a balancing game with an inverse pendulum that you need to balance (like balancing a stick on your palm). But, my issue is that I add game objects to it by making them c$$anonymous$$ldren of the game object with the $$anonymous$$nge joint, so that they "stick" to each other, but t$$anonymous$$s effects the $$anonymous$$nge joint.

I don't know if it is a center of mass issue, but I know that adding c$$anonymous$$ldren with box colliders changes the physics of the $$anonymous$$nge joint - it's harder to swing back and forth.

Does anyone know any fixes to t$$anonymous$$s issue with the $$anonymous$$nge collider? Or would it be easier to just set the objects' position each frame instead of making them c$$anonymous$$ldren? Any help is appreciated!