Best performance for repeatable objects in a City scene.

Greetings. I'm currently working on a project which includes a city/street filled with fully explorable buildings. Each window and door is openable, etc. As I've encountered some performance issues already I guess it's best to ask before doing anything further.

My idea was to make some building elements saved as separate models, which will later allow me to build different buildings from the same parts (I also have heard that reusing same assets helps with performance). Think of it as of Lego blocks. I have things like Floor model, Wall model, Stairs model, Door model, Window model etc.

In my situation, what's the best way to make it work without lagging? What are the steps I should take from the moment of asset import to ensure good performance? Would it actually be better to make whole building models and import them into Unity? Since all doors and windows are interactive I'd have to add them later.

PS. None of the stuff I use is high-poly