Physically based sky indirect lighting is wrong

Hi, I'm currently using 2021.2 with the HDRP. I'd very much like to use the physically based sky for my game, since it works very well for day/night cycles, but I'm running into an issue I have no idea how to solve. The indirect lighting of the scene when using physically based sky is very off. Here is how I expect the indirect lighting to look (using HDRI sky). https://imgur.com/a/ffab4pW

T$$anonymous$$s is what it looks like with the physically based sky. https://imgur.com/a/tkyNokM

As you can see, the indirect lighting with the physically based sky is just... very off. I tried changing the physically based sky settings, and pretty much every other lighting setting there is, but not$$anonymous$$ng changes the only t$$anonymous$$ng that fixes the issue is simply not using the physically based sky. Any help is greatly appreciated