Best way to sync scene with new connections?

Hello, while working on my game's inventory system I noticed that when players connect after the host grabs an object, it will still appear there which isn't the only problem, I'll explain what's going on. You can see it here (webbuild, Q to open inventory, E to grab items) if you open two tabs.

Basically each player class has its own inventory List which is updated to everyone whenever the player grabs or uses an item (e.g. I grab a sword, I equip it -> other player runs the same methods as the main player's which uses its own inventory, so keeping it updated is important). Although when a player joins after the host modified his inventory, it doesn't sync. Here's what I thought:

Should I send each player's inventory(list) on player join?

But how would that sync grabbed items created after the host grabbed in-scene created objects?

If I had to remake all the inventory/items system for something more efficient I'd gladly do it so please, suggest anything. This is my first time trying to make a game in Unity and asking something here by the way.

If you need scripts let me know and I'll edit the post (I believe I can do that?) thanks for reading.