Network Unspawn - Only happening host side, client keeps object

On contact, my bullets only UnSpawn the character on the host side. Clients keep the GameObject. this is not the intended purpose.
After making a few changes, it appears that now my GameObject isn’t unspawning at all.

My Collision code:

void OnCollisionEnter(Collision col)
    {
        GameObject hit = col.gameObject; //Was looking at a tutorial and this was suggested
        if (hit.name.Contains("Character") && GameObject.ReferenceEquals(hit,gameObject))
        {
            camera.GetComponent<cameraScaleController2>().m_Targets.Remove(col.gameObject.transform);
            camera.GetComponent<cameraScaleController2>().numTargets--;
            hit.GetComponent<character>().CmdUnSpawn(hit); //CmdUnSpawn is on my player object because I read that was required.
            Destroy(gameObject);
        }
        else { Destroy(gameObject); }
    }

And now CmdUnSpawn:

[Command]
    public void CmdUnSpawn(GameObject toDestroy)
    {
        NetworkServer.UnSpawn(toDestroy);
    }

As far as I can tell, this should work? Where have I gone wrong.

For reference, I am using a different network sync package called “Smooth Sync”, so none of my objects have Network Transforms on them (all have network identity, its required for Smooth Sync).

I needed to give the bullet authority in the network identity.