Texture Memory from Prefabs that aren't in the Scene

I was running the profiler today and noticed that my texture memory was really high. When I ran the detailed profiler it showed a bunch of Sprite Atlases that are being referenced by prefabs that aren't even in the scene. I figured that since the prefab wasn't in the scene that it shouldn't be taking up any texture memory but I might wrong in that line of thinking.

Does anyone know if there is a way to ensure that unused prefabs don't take up texture memory until the prefab itself is instantiated in to the scene?

Also I am using NGUI if that makes a difference.

Thanks!