Problem with "pushing" objects with FPS Controller

I found a script that I attached to FPS Character Controller available in standard assets that allows the player to push objects.

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class FizikaObjekt : MonoBehaviour {
 
     public float pushForce = 0.5f;
 
     void OnControllerColliderHit(ControllerColliderHit hit)
     {
         Rigidbody body = hit.collider.attachedRigidbody;
 
         //checking whether rigidbody is either non-existant or kinematic
         if (body == null || body.isKinematic)
             return;
 
         if (hit.moveDirection.y < -.3f)
             return;
 
         //set up push direction for object
         Vector3 pushDirection = new Vector3(hit.moveDirection.x, 0, hit.moveDirection.z);
 
         //apply push force to object
         body.AddRelativeForce(pushForce * pushDirection);
     }
 }

The problem is that no matter what value I put in mass or pushForce variable, the object I push is always pushed at same speed and force. I want to make objects heavier so that pushing them from point A to point B takes a bit more time than just send them flying when touching with the controller.

Any suggestions?