UnloadSceneAsync() does not seem to work with additive scenes.

I'm trying to load multiple scenes after one another to test some persistence between scenes. However, when I use LoadSceneMode.Additive, I can't seem to UnloadScene (or UnloadSceneAsync) the previous Scene. I then end up with multiple scenes, which creates all sorts of messes, like double audioListeners from the camera's and colliding objects.

I want to be able to switch between different scenes (like menus and different levels) while having a persistent gameObject with scripts that retain can retain information from one scene to the next.

This is the snippet of code that to me seems like it should work:

 DontDestroyOnLoad(gameObject);
 SceneManager.LoadScene("Level1", LoadSceneMode.Additive);
 SceneManager.LoadScene("Menu", LoadSceneMode.Additive);
 SceneManager.UnloadSceneAsync("Level1");

All scenes remain loaded, however. I get no error messages. Changing the LoadScene methods to LoadSceneAsync does not solve the problem. The LoadSceneMode.Single option does unload the previous scene correctly.

Am I handling scene managing wrong?