GPU Instacing performance drop

Hey Guys, I implemented GPU instancing, but everything works great when no scripts are attached to the prefab that I am instancing. I can instantiate 20,000 spheres and still get a descent 35FPS, with 10,000 objects I get roughly 65 FPS and with 5,000 92 FPS. but when I attach a very simple script with one line of code to the prefab performance decreases drastically. With 5,000 I get 23FPS with 10,000 I get 9 FPS and with 20,000 3 FPS. When I did the same process with a fairly more complex script... roughly 200 lines of code, the performance dropped even more and when I tried 20,000 objects my computer could hardly instantiate them and eventually It got stuck. Basically I am assuming that there is a lot of behind the scene things unity is doing to compile all the scripts attached to all those prefabs and that is what is causing the performance lag. But interestingly enough when I check the performance in the profiler I only get a big consumption of "Physics", then in "Others" but "Scripts" does not even appear. See attached image as reference.

Is there any way to save performance when more complex scripts are used? These are just small tests I did for a larger project I am developing where I have quite a long set of scripts, and my goal is to at least be able to run 5,000 instances at a descent frame rate, 35-50 would be fine. Any insights would be great!! Thank you