How to make the skybox in High Def Render Pipeline NOT influence lighting of things in the scene?

I can't really figure this out. there seem to be only 2 relevant settings for the HDRI sky in high def render pipeline and that is exposure and multiplier and no matter what I do to them it's either the skybox is completely dark and doesn't influence things in the scene, or it's bright and it does create ambient light.

What I'd like is for the skybox to be in full brightness, but with all the things in the scene not changed. I checked if it's not the directional light causing it and it's not.