Drag Item Scaling on Phone

I implemented a drag and drop system that works very well using the mouse using the drag and pointer interfaces (IDragHandler etc.). To determine the amount to move my object on drag, I divide the eventData delta by the scale factor of the canvas. This does not work on a phone, with finger dragging unfortunately. How would I implement something similar for an iphone or android?

This is the gist of the code:

 public void Start()
 {
 //...
 //this is a vector2
 m_scaleFactor = canvas.GetComponent<RectTransform>().localScale;
 }    
 
 public void OnDrag(PointerEventData eventData)
 {
      if (!Active)
        return;
 
      //used to reference canvas.scaleFactor
      m_rectTransform.anchoredPosition += eventData.delta / m_scaleFactor;
      onDragAction?.Invoke(eventData);
             
 }

EDIT: this is actually an aspect ratio issue. I can reproduce by changing the aspect ratio in the editor. Still not sure what to scale by in order to fix though.