Unity3D Photon movement synchronization

Hello,

I'm making a billiard game and I wanna sync the balls' movement. I'm using Photon network and t$$anonymous$$s is my code so far:

 Vector3 vel;
 Vector3 angularVelocity;
 Vector3 pos;
 
 void Update()
     {
         UpdateMovement();
     }

 void UpdateMovement()
     {
         if (PhotonNetwork.inRoom && !GetComponent<PhotonView>().isMine)
         {
             if (networkManager.playerTwo)
             {
                 if (gameManager.firstPlayerTurn)
                 {
                     rb.position = Vector3.Lerp(rb.position, pos, 0.1f);
                     rb.velocity = Vector3.Lerp(rb.velocity, vel, 0.1f);
                     rb.angularVelocity = Vector3.Lerp(rb.angularVelocity, angularVelocity, 0.1f);
                 }
             }
 
             if (networkManager.playerOne)
             {
                 if (!gameManager.firstPlayerTurn)
                 {
                     rb.position = Vector3.Lerp(rb.position, pos, 0.1f);
                     rb.velocity = Vector3.Lerp(rb.velocity, vel, 0.1f);
                     rb.angularVelocity = Vector3.Lerp(rb.angularVelocity, angularVelocity, 0.1f);
                 }
             }
         }
     }

 void OnPhotonSerializeView(PhotonStream stream, PhotonMessageInfo info)
     {
         if (stream.isWriting)
         {
             stream.SendNext(rb.position);
             stream.SendNext(rb.velocity);
             stream.SendNext(rb.angularVelocity);
         }
 
         else
         {
             pos = (Vector3)stream.ReceiveNext();
             vel = (Vector3)stream.ReceiveNext();
             angularVelocity = (Vector3)stream.ReceiveNext();
         }
     }

The problem is that the movement is kind of sloppy and when it's the second player's turn the balls won't move on any screen, they get back on the original position(the position that's on the first player's screen).

Can anyone help me with t$$anonymous$$s ?

Thank you for your time.