Is it possible to destroy an object at a specific position?

No, that is not possible. You have to know which GameObject is at that position. If you are making a grid-type game, you can make a jagged array which contains a GameObject at each specific index = position.

So if your grid is, say 5 by 4 you will have a jagged array like this:

 GameObject[][] objectsInMap = new GameObject[5][]

Then, use a for-loop to initialize each index to 4:

 for (int i = 0; i < objectsInMap.Length; i++)
 {
     objectsInMap[i] = new GameObject[4];
 }

Then, when you need to destroy a GameObject at a specific position, use:

 if (objectsInMap[x][y] != null)
 {
     Destroy(objectsInMap[x][y]);
 }