What is the best approach to filter large data sets in Unity?

I don't have time to give a complete answer but check out C# Linq... it provides many DB style functions you can apply to collections and arrays that may be sufficient for what you need to do. If you are more specific about exactly what you want to do I may be able to give more detailed suggestions!

What kind of dataset you are talking about? If it's a big amount of object data generated internally by your application you will need different answers than if you want to read and save to an external storage, process data downloaded on the fly from Internet, etc.

Please, show us a bigger picture.

Sorry about lack of details.

Here is the current flow - the player chooses a generated character, and I construct a scene from a template. The player interacts with the scene, triggering one of four possible resolutions. Depending on the resolution, I either generate a new scene from the template, or new characters to play.

There will be over 1000 scene templates, and the character chosen will inform which template is used. The template chooser needs to evaluate which template is appropriate by first comparing logical semantic data (tag a or b, not c) for the character and meta game, returning a subset that are allowed which can then be picked either by priority or randomly chosen.

That chooser is reflected in other areas - for instance, the objects included in a scene may be assigned in a similar fashion, and certain aspects of character generation.

This is all offline, and I will define the templates and rules for choosing them. I hope that helps - most of my experience is in scripting content, this is my first time trying to do it all!