Canvas Inventory closing

So I'm trying to fix a Inventory which is a Child of the Canvas. I'm using the DragHandler (OnBeginDrag, OnDrag and OnEndDrag) to drag & drop Items. When I'm dragging an item, a collision script is checking if the the item Collider is inside the inventory Collider. If not, the inventory gets closed so the Player can basically put the item into somewhere else.

My Problem Right now is when I'm dragging the item (in OnDrag the item follows Input.MousePosition) the Position of the item is for a Moment Pretty far away to the left of the mouseposition, since the canvas is in screenspace - overlay render mode and only if i'm moving the mouse further the item Position is following the mouse Position again. This short Change of the Position is causing the inventory to close as soon as I start dragging the item.

Does anyone know how to fix this kinda Problem? For more Information to the game: It's a 3D game, I tried to set the render mode of the canvas to camera (main camera) but then the inventory is bugging. I hope I Illustrated the Problem enough to understand it, if not ask me anything. The scripts aren't fully programmed by myself (I'm working onward on scripts from another Person).