RaycastHit2D hit to texture coordinate

I can hit the object, and for test purposes I paint on it, but I can't match the exact position, my calculated position is always off no matter what I do, can someone help with the math? Example of how the position is off in this specific example:

I painted in the middle and moved a bit with the mouse, the resulting red dots are in the corners.

 void Update()
     {
 
         if (!Input.GetMouseButton(0))
             return;
 
         Vector2 mousePos = Camera.main.ScreenToWorldPoint(Input.mousePosition);
         RaycastHit2D hit = Physics2D.Raycast(mousePos, Vector2.zero);
         if (hit.collider != null)
         {
             print(hit.point);
         }
 
         Vector2 pixelUV = Vector2.zero;
         Texture2D tex = hit.transform.GetComponent<Renderer>().material.mainTexture as Texture2D;
 
 
         Vector2 localPos;
         localPos.x = hit.transform.position.x - hit.point.x;
         localPos.y = hit.transform.position.y - hit.point.y;
 
         // Uv = localPos.y / height * textureHeight
 
 
         pixelUV.x = (localPos.x / hit.transform.localScale.x ) * tex.width ;
         pixelUV.y = (localPos.y / hit.transform.localScale.y ) * tex.height;
 
         Debug.Log("transform position = " + hit.transform.position);
         Debug.Log("hit.point = " + hit.point);
         Debug.Log("pixelUV = " + pixelUV);
         Debug.Log("tex " + tex.width);
 
 
         tex.SetPixel((int)pixelUV.x, (int)pixelUV.y, Color.red);
 
         hit.transform.GetComponent<Renderer>().material.mainTexture = tex;
         tex.Apply();
 
     }

Especially these lines:

             Vector2 localPos;
             localPos.x = hit.transform.position.x - hit.point.x;
             localPos.y = hit.transform.position.y - hit.point.y;
 
             pixelUV.x = (localPos.x / hit.transform.localScale.x ) * tex.width ;
             pixelUV.y = (localPos.y / hit.transform.localScale.y ) * tex.height;
 

I did alot of try and error but have no idea how I get the exact positon.