Is there a way to set the mass of a gameobject without using the Rigidbody component??

Well mass = force / acceleration. So simply calculate force and acceleration. Then calculate mass.

Here’s how to calculate acceleration:

 //Java Implementation
 
 var lastVelocity : Vector2;
 
 function Update()
 {
     var acceleration = (velocity - lastVelocity) / Time.deltaTime;
     lastVelocity = velocity;
 }
 
 //C# Implementation
 Vector2 lastVelocity;
 
 void Update()
 {
     float acceleration = (velocity - lastVelocity) / Time.deltaTime;
     lastVelocity = velocity;
 }
 
 

Not to sure how you are implementing force so I can’t give a clear answer for that. Even though that’s a way to calculate acceleration you still need velocity which so happen rigid body simulates. So you have a bit of work ahead of you.

Well, if you don't use Unity's physics system you just have to "store" the mass in one of your variables. Since you want to roll your own physics you have to keep track of an objects properties (for a rigidbody the static infomation is the mass, the center of mass, the inertia tensor as well as any form parameters like drag). Note that i talk about Rigidbodies in general, not Unity's rigidbody component.

Simulating proper newtonian physics can be quite complicated. It's not clear if you actually use another third party physics engine or if you actually want to roll your own. If you use a thrid party physics engine it should have means to set the mass of an object. If you roll your own, just declare a variable like

 public float mass;

And assign a value. Of course in the actual equations when you calculate the required acceleration from a given force you have to use that value.

Since this seems to be "homework" i will move the question into the help room.