HingeJoint with no rotation?

I'm implementing a 2D swinging mechanic, sort of on a rope/grappling hook. Looks like this:

As you can see, the collider is rotating along with the HingeJoint2D which is a problem for me.

I've put the sprite renderer on a seperate gameobject, so the sprite doesn't rotate with the swinging gameobject - which has a rigidbody2d and hingejoint2d that is created at runtime via script.

I tried putting the Box Collider 2D on the gameobject that has the sprite, which makes it not rotate with the swinging, but then it no longer collides with anything

I've also tried checking the 'Fixed Angle' on the RigidBody2D, however this completely disables the HingeJoint2D and the character no longer swings.

What can I do to keep the HingeJoint2D method for swinging, but have it not rotate the collider?