Why is CapsuleCast/Raycast performance effected by rigidbody?

I am doing some big number testing for collision detection with a MMO authoritative simulation server, and am curious as to why I am getting the results I am getting.

Basically I am running a test where I do 100000 capsule casts (which would be a weapon swing) in a scene with 3000 capsule colliders (simulating the hitboxes of the players) When the colliders do not have rigid bodies attached, unity executes it in approx .35 seconds.

When the colliders HAVE rigid bodies, it takes nearly 8 seconds... a huge difference. I have the rigid bodies set as kinematic, and am just really confused as to WHY this is the case considering CapsuleCast(), i believe, doesn't take rigid bodies into account and definitely works fine without them in the scene.

The same is true for raycasting without RigidBodies, RayCastAll() takes .12 seconds, and with them it takes 1.5 seconds to execute.

(just as a disclaimer before someone says it, this is not due to rigid bodies simply being in the scene either, because if I don't do any sort of casting then it takes .02 seconds to play the scene)

Thanks so much for the help!
Richard

--As a follow up, it doesn't effect performance if the rigid body is only in the root of the object, or on ALL the child components (legs, head, torso, etc)... both yield the same execution time. If there is any rigid body attached to the colliders, performance goes down by 1000%... lol, help.