Unity Remote 4 blocks USB debugging until restart

I got Unity Remote 4 and I have a very peculiar problem. The steps:

• connect phone via USB to PC;

• phone goes into USB debugging mode and PC recognizes it;

• if I run my Unity project on my PC it streams correctly on the phone;

• if I unplug the cable (no matter if I closed Remote 4 or if I stopped the project), then I can no longer plug the cable back in and get recognized by my PC or go into debugging mode.

I tried force stopping the app on my phone; closing my Unity on my PC; restarting my PC -> nothing works. Only when I restart my phone can I go into debugging mode and be visible to the PC again.