Using git with large project: best practice ?

So, I have a large project (around 10 GB asset folder). I use asset store assets extensively. I have organised my project with my own assets in Assets/_Custom/ folder, and all other assets placed where asset store places them on import. (Typically in the root).

Since I don't edit the asset store assets, I don't really NEED them in source control, and since I have a LOT of them, I would like to avoid to have them in source control, in order to get faster pushes and pulls. I have set up LFS and a suitable git ignore, but LFS also bogs down.

What would be good practise ? .gitignoring all folders except my _Custom/ folder ? Will that work ? Would prefabs in _Custom/ referencing files outside that folder be able to correctly find their assets ?