Need help with a design approach regarding SkinnedMeshRenderer items

I have a dilemma currently, I've been following a tutorial on making an inventory system in unity. It's come along way and I'm at a point where I would like when equipping and unequipping items from inventory it will update the player character graphically. It currently does this fine when equipping the item. However I'm stumped on how to make it work when removing the item from the equipment loadout. At one point it worked (even though it shouldn't have) it would do it without problem no null ref errors at runtime or anything. Now it throws null exceptions, and I get why it does it. I'm writing here today to perhaps find the best way to approach this.

Some background info: When the item is equipped I have a ScriptableObject that holds the information of the EquippableItem object. When the method is called to equip the item it will instantiate the item mesh object on the players body (parent object) doing all the necessary things needed to make sure it deforms accordingly when the player animates and what not. My problem is when it comes to unequip the item from the character. I'm not sure the most effective way to approach this. What needs to happen is when the item is unequipped it should destroy the mesh object from the body. But it needs to be able to get called from the unequip method. This way when swapping items it doesn't mistakenly delete the item being equipped from the swap.

I thought about making a EquipmentHelper script that would check the equipment slot in the equipment manager. If it is null then delete the object assigned there. But i foresee this causing problems with the equipment swapping..

I'm not sure how to effectively do this, in my mind my approach is not as efficient as it could be. Any suggestions would be greatly appreciated!