Network.Instantiate issues - GameObject ownership and network views?

We are some weird behvaiour with Network.Instantiate we are calling it to spawn cars in a racing game for a game jam. The players are assigned a unique ID from the server to indicate their starting position. Network connection all works fine and RPC calls go through perfectly correctly.

 // Start the race
 public void BeginRace() {
     if(!networkManager.IsServer())
     {
         Vector3 carPosition = new Vector3();;
         Quaternion carRotation;
         
         switch(networkPlayerID)
         {
         case 1:
             carPosition = new Vector3(841.8432f,102.1979f,808.0422f);
             break;
         case 2:
             carPosition = new Vector3(841.6416f, 102.1979f, 816.8533f);
             break;
         case 3:
             carPosition = new Vector3(828.7191f, 102.1979f, 807.7421f);
             break;
         case 4:
             carPosition = new Vector3(828.7191f, 102.1979f, 816.1051f);
             break;
         }
         
         // Spawn the car
         GameObject car = (GameObject)Network.Instantiate(car1Prefab, carPosition, new Quaternion(0f, Mathf.PI * 0.5f, 0f, 0f), 0);
         
         // Pair the camera to the car
         GameObject camera = GameObject.FindGameObjectWithTag("CarCamera");
         CarCamera cameraScript = camera.GetComponentInChildren<CarCamera>();
         cameraScript.target = car.transform;
     }
     
     // Start HUD Countdown
     hud.StartCountdown();
 }

We find that if 1 player connects. It spawns on the client and on the server and the camera is paired to the car correctly. However if multiple clients connect we have issues. The correct number of cars spawn in the correct position on each client. The camera is paired to the correct car. However the server (which spawns no cars) fails to show any cars. Also the user controls all 3 cars.

Anyone got any ideas on what's going wrong?