Does collider size affect performance?

It's hard to say if any of those situations individually would cause lag to appear where previously there was none. As for whether it will affect performance, the answer ultimately depends on the underlying implementation of the physics engine. More specifically, it depends on the spatial acceleration strategy/data structure being used. With no spatial acceleration (i.e. n-squared collision detection, where every object checks every other object for collision), it wouldn't matter, but Unity/PhysX certainly does something smarter. But regardless of the exact spatial partitioning strategy used (whether it's an octtree, kd-tree, or otherwise), the fact is that the larger a collider is, the more things there will be that need to check if they are colliding with it. So in that respect, yes, collider size will affect performance. Making the collider use a rigidbody (making it dynamic) will be an additional hit, as will making it a mesh collider.