Terrain sampleHeight jumps in value

Im currently making a game where I have the object follow the mouse until I click and then it gets placed on the terrain. It works but when it is in the follow mouse part of the code the returned Y value from sampleHeight jumps up in value. There is no problem when its placed, only when its following the mouse.

Heres some code: (All in void Update())

     startBuildBut.onClick.AddListener(TaskOnClick);
     mousePos = mainCamera.ScreenPointToRay(Input.mousePosition);
     RaycastHit hit;

     if (startBuild == true) {
         if(Physics.Raycast(mousePos, out hit)){
             Vector3 finalPos = hit.point;
             finalPos.y = Terrain.activeTerrain.SampleHeight(startClones[counter].transform.position) + 2.0f;
             Debug.Log(finalPos.y);
             startClones[counter].transform.position = finalPos;

And here is the console output: You can see the value going from 9 (what it is supposed to be) and 200 (wrong value that makes the object jump and bounce in game)