3D Drag&Drop with collision

Hi, I'm working on a game where the player has to set up a ladder. To let the player move different parts of the ladder, I wrote a draganddrop script by this example: https://www.youtube.com/watch?v=NMt6Ibxa_XQ.

Now I'm thinking about how to add physic to this scenario. I read a lot comments to this problem but nothing I can realize in my situation. Why collision doesnt work when I transform an object by transform.position is clear. I also read about the DragRigidBody but I couldnt make it work in my project.

I realy need a accurate collision detection to show the different functions of the ladder and the behaviour of the ladder if the player did wrong. The ladder has to collide with a Wall, fall down on the floor, different parts of the ladder shall collide with each other and so on...

Any ideas how I can combine drag and drop and the physx-engine? Maybe a another method to transform the parts by drag and drop?

Thank you very much for solving tips. I dont have a clue.