Weird environment lighting in HDRP

I am a beginner at Unity and I just started to explore HDRP. I setup a simple scene, with a terrain, a building and a directional light. I haven't changed any lighting settings, nor have I added any volumes (post processing/sky-fog/global). With the directional light enabled, the scene looks normal, but when it is disabled, I expect the scene to go pitch black, but instead, I get a very dull looking terrain and building, with an extremely bright and glowing sky. I have attached both the images