Mirror networking, weird behaviour with local scale

I have a method I use to flip the sprite so my shooting hand is always on the same side as the mouse.

         //Left and Right

         if (delta.x >= 0 && !facingRight)
         { // mouse is on right side of player
             playerBody.transform.localScale = new Vector3(xSpriteSize, ySpriteSize, 1);
             facingRight = true;
         }
         else if (delta.x < 0 && facingRight)
         { // mouse is on left side
             playerBody.transform.localScale = new Vector3(-xSpriteSize, ySpriteSize, 1);
             facingRight = false;
         }

Clientside, it works fine, but when 2 players are playing together, whenever the other player moves their mouse past their player, it generates a weird Paper Mario effect that I'd really like to get rid of. (Mouse can't be seen but it's on the right screen moving across the player, which is causing left screen player to see the weird Paper Mario Effect.) I tested and paused the unity editor a bunch and it seems Mirror is translating xSpriteSize to -xSpriteSize and vice versa, instead of setting them to their values. Does anyone know how to force Mirror to set the value instead of translating it?