Issues with multiplayer Xbox controllers and players (non-networked)

I am trying to set up a system for a game i'm working on in which the game, which is multiplayer but not online (for now) will accept four different sets of Xbox controller inputs (for now just the left stick, since that's the important bit), and move the correct player's character in the scene. I have arranged it to have a different set of inputs in the InputManager for each controller, both in Horizontal and Vertical directions, and each has been set to accept a numbered joystick, but it isn't working the way I want it to.

The first time I tried to get this working, initially with only two controllers, the Player 1 character could only be moved with the Player 2 controller, whilst the Player 1 controller moved Players 2, 3 and 4. This was annoying, but it got worse when the other two controllers were plugged in, and the Player 1 and 2 controllers did literally nothing. Player 1 could only be moved by the Player 4 controller, and Player 3 moved everyone else.

I have tried to find a way to fix this, but my skill level in regard to inputs isn't what it could be, and I have no idea what to do to move forward. My best option right now is to convince my team to bite the bullet and make the game entirely multiplayer and limit it to one player per client (even though networking, in my mind, is a pain in the ass), to avoid the issue, but that was what we were going to tackle after this step.

Anyone got any ideas? I'd be grateful.

Also, if you need further detail, just ask. :)