Delete certain gameobjects with mouseclicks.

I have an array of gameobjects that fall randomly from the top of a grid. They drop to the bottom and wait to be clicked on, all is good there. When I use GetMouseButton(0) to destroy a gameobject, it destroys all gameobjects that are on the screen. How would I be able to delete just the one block or a group of same colored blocks that are touching one another?

I don't have much code, due to not knowing where to start. I assume a 2d raycast is what I am needing to do, but I don't have a clue how to do that.

Here is the code I have, it is attached to my gameobjects.

 public class Blocks : MonoBehaviour {
  
 // Use this for initialization
 void Start () {
  
 }
  
 void Update ()
 {
     if(Input.GetMouseButton(0))
         Destroy (gameObject);
 }
 }