Unity Stop/Freeze when Destroying+reloading a lot of GameObjects, solution ?

Hello, I'm currently building a 2D procedural maze, lets say it got 100x100cells, containings each around 5-10gameObjects in an Array. Currently i make Inactive every Cells and what they contains if they are far from the character, i got around 150fps in the viewport.

I made a function Reset that basically Destroy every GameObject of the Array, and set the array to null, then i relaunch the MapGeneration.

But after 2-3Reset, fps drops around 50fps, and then Unity Freeze completetly. I don't completly understand why, since there is as much as the first generation on the scene. Its a real problem for a proto-game where i need to reset the maze every 5-10minutes :S.

Should i instead destroy the whole scene and recreate it ? Or is there a method i'm missing about it ?

(i heard about pooling but this fit stuff like fire bullets or heavye constant creation of GameObject). I can't poolthousand of cells :S)

Thanks in advance^^.