How big is too big for a Character Rig?

I'm using the Skinning Editor / Character Rig to create a Skeleton Rig for a character, but in my game the user can swap out shirts, pants, shoes, hair etc. Potentially this Photoshop PSB file could end up being made up of 100s of layers.

Before I go too far down this path, I wanted to a ask:

• Will this pose memory/performacne issues if this single sprite sheet has 1000s of objects in it?

• Will unity itself become slow and unusable because of all the layers?

Any insight greatly appreciated!