Find all components of specific type, as fast as possible

Since FindObjectsOfAllType iterates all components, I implement this:

 public class CachedBehaviour<T> : MonoBehaviour where T : CachedBehaviour<T>
 {
     public static readonly HashSet<T> cache = new HashSet<T>();
     protected virtual void Awake() => cache.Add((T)this);
     protected virtual void OnDestroy() => cache.Remove((T)this);
 }

I write a test to iterate 200 objects for 100 times, modifying corresponding gameobjects' transform. there are about 100000 objects in the scene.

• FindObjectsOfType<CustomType> costs nearly 2sec for one frame.
  

• CachedBehaviour<CustomType>.cache costs 4ms per frame.
  

• Change the specific componet from CustomType to builtin SpriteRenderer, FindObjectsOfType uses only up to 0.4ms per frame.

So there is a faster way for built-in components? How does it work? Can I make use of it?