'Invert' UI mask

I'm trying to fix the same problem. So I come up to the idea to switch stencil operation.

here is a code

 using UnityEngine;
 using UnityEngine.Rendering;
 using UnityEngine.UI;
 
 public class InvertedMaskImage : Image {
     public override Material materialForRendering
     {
         get
         {
             Material result = base.materialForRendering;
             result.SetInt("_StencilComp", (int)CompareFunction.NotEqual);
             return result;
         }
     }
 }

and t$$anonymous$$s is the scene setup

Use the same Mask object on the MaskSprite. but for CoverScreenSprite change Image component to the InvertedMaskImage. so it will replace stencil comparison operation.

Unfortunatly inspector will still show old material parameters, but it will work. in editor as well as in play mode

It's an old question but I noticed a unwanted sideeffect with nicloay's solution in Unity 2019.4.x. The script will modify your default ui material stencil setting (all default sprites will become invisible).

Here is my approach without coding w$$anonymous$$ch is similar to nicloay's solution but uses one dedicated material for mask inversion. It also avoids creating / instancing new materials.

If you don't want to touch any shaders, there are 3 ways of doing t$$anonymous$$s.

One easy, but the results might be bad (depending on many factors) and one that is somewhat safe, but will require some work on your part to allow it to appears as intended on the screen. I'll explain both ways with, as an example, your idea of a keyhole where the hole is the invisible part of the UI that display the scene or whatever is be$$anonymous$$nd it.

The easy way of doing t$$anonymous$$s is to only use a inverted texture as a UI mask. By using an "huge" UI Image that exceed on all side of the screen, you could have your keyhole set in that.

T$$anonymous$$s is a quick visual reference : The issue with t$$anonymous$$s method is that the hole can't be crystal clear. T$$anonymous$$s means it can work if you're using a smoothed or blurry keyhole right from the start. (By "Resize the Texture", I meant the UI RectTransform of the Image component that hold the keyhole Sprite.)

The other method that doesn't involve making your own or using someone else shader is to do a small trick where you use a much clearer (and possibly smaller) image file of the keyhole inverted. It's like the first option above, but instead of using a HUGE amount of "filler" around the keyhole, you use a set of 3 row and 3 columns where the keyhole is kept in the middle. You can do t$$anonymous$$s by using an intelligent set of Layout groups and uses of Layout Element components. After, you just got to adjust the middle keyhole's RectTransform sizes w$$anonymous$$le letting the 8 surrounding "plain" colored UI element readjust themselves.

The last method requires a bit of work, yet still doesn't need any kind of special shaders. You can use an additional camera with and a $$anonymous$$gher priority (larger depth) and with Depth Only for its Clear Flag w$$anonymous$$ch only see the keyhole texture/mesh (through its culling mask) and animate/move the camera into or out of the the keyhole. T$$anonymous$$s method is a lot more useful if you plan to use some 3D modeled keyhole or if you want more control (via script) over the keyholes animations and/or events. The only issues you might accounter with t$$anonymous$$s last/3rd method is memory/vram wise. Adding a camera for such might be a problem if you're aiming at some light-weight smartphones or tablet as such each camera requires its own Z-Buffering channel to "cut" out the required stuff and put each "results" over each other in the right order.