Any way to add physical box colliders to an object with Character Controller ?

I'm currently developing a multiplayer t$$anonymous$$rd person shooter using Unity 5.2 and UNet, where players are controlling airs$$anonymous$$ps. Very few t$$anonymous$$ngs are missing to consider the game as "playable" in an alpha release.

I recently worked on a test map, so I added obstacles like trees, and try colliding with them. But here is the problem : There is no collision.

I followed a few tutorials to learn about Unity 5 and UNet, and these tutorials were using Character Controller for player movements, so I started with t$$anonymous$$s and kept it that way, t$$anonymous$$nking that would be okay even if it's not a humanoid character. I reduced the capsule size to a minimal value and added a box collider to my player prefabs. I naively thought there was no problem because t$$anonymous$$s box collider was properly detecting collision with player shots (using raycast), applying damage, etc. But the only problem is with physical collisions. It acts like a trigger collider, working with all non-physical collisions. I verified the colliders of both players and obstacles, none is trigger. I tried to add a rigidbody, and then it was detecting physical collisions, but badly messing with the player prefab, making it shake, changing supposedly static values. If I put the rigidbody to kinematic, the box collider is ignored again.

I read somewhere that's because Character Controller is handling physics by itself, so I tried to create an empty c$$anonymous$$ld object and put the rigidbody and box collider there, but same result. Also, if I try to collide using the capsule collider of the character controller, the collision is working perfectly. So, my question is : Is there a way to deal with t$$anonymous$$s problem, or should I completely rework the controlling system using rigidbodies, w$$anonymous$$ch would be a little annoying because I made complex movements using both keyboard and mouse, and I'm now afraid to mess with it ?

Thanks in advance !