Joystick ID when connecting and disconnecting

So I added 4 controller inputs in the input manager.
Each set of input just uses joystick 1 through 4.

Now the issue is when disconnecting a controller and then reconnecting, it doesn't take one of the ID's that became unused, it just takes 4 then 5 then 6 etc.

So how do I prevent this, cause obviously I didn't add the input for joystick 5 or I would need to add all possible joystick numbs, which is just stupid.

I just want the 4 controllers to be ID 1, 2, 3, 4 always.