Unity3D Photon movement synchronization

Hello,

I'm making a billiard game and I wanna sync the balls' movement. I'm using Photon network and this is my code so far:

 Vector3 vel;
 Vector3 angularVelocity;
 Vector3 pos;
 
 void Update()
     {
         UpdateMovement();
     }

 void UpdateMovement()
     {
         if (PhotonNetwork.inRoom && !GetComponent<PhotonView>().isMine)
         {
             if (networkManager.playerTwo)
             {
                 if (gameManager.firstPlayerTurn)
                 {
                     rb.position = Vector3.Lerp(rb.position, pos, 0.1f);
                     rb.velocity = Vector3.Lerp(rb.velocity, vel, 0.1f);
                     rb.angularVelocity = Vector3.Lerp(rb.angularVelocity, angularVelocity, 0.1f);
                 }
             }
 
             if (networkManager.playerOne)
             {
                 if (!gameManager.firstPlayerTurn)
                 {
                     rb.position = Vector3.Lerp(rb.position, pos, 0.1f);
                     rb.velocity = Vector3.Lerp(rb.velocity, vel, 0.1f);
                     rb.angularVelocity = Vector3.Lerp(rb.angularVelocity, angularVelocity, 0.1f);
                 }
             }
         }
     }

 void OnPhotonSerializeView(PhotonStream stream, PhotonMessageInfo info)
     {
         if (stream.isWriting)
         {
             stream.SendNext(rb.position);
             stream.SendNext(rb.velocity);
             stream.SendNext(rb.angularVelocity);
         }
 
         else
         {
             pos = (Vector3)stream.ReceiveNext();
             vel = (Vector3)stream.ReceiveNext();
             angularVelocity = (Vector3)stream.ReceiveNext();
         }
     }

The problem is that the movement is kind of sloppy and when it's the second player's turn the balls won't move on any screen, they get back on the original position(the position that's on the first player's screen).

Can anyone help me with this ?

Thank you for your time.