How can I create nested buttons ?

I know you may not like this, but I'm going to encourage you to re-think your design to something a little more traditional. People accurately hit buttons about 95% of the time, which means about 1 out of 20 times someone will take an unintended action with this layout (as opposed to hitting a margin and doing nothing). Having a big start button above the three configure buttons will be a much more user-friendly design than a button-in-button design.

You can disable GUI interaction with the Main-Button when the mouse is over any of the other buttons.

Here is an exemple:

 void OnGUI(){
             
         Event e = Event.current;
 
             Rect r = new Rect (0,0,200,200); // play button rect
             Rect s1 = new Rect (20,80,60,20); // setting 1 button rect
             Rect s2 = new Rect (120,80,60,20);  // setting 2 button rect
 
         // check if mouse is over any of the settings buttons, if so disable the gui for the Play button
         foreach (Rect rect in new Rect[]{s1,s2})
             if (rect.Contains(e.mousePosition))
             GUI.enabled = false;
         
         if (GUI.Button (r,"Play")){ 
                 Debug.Log ("Play was Clicked");
             }
 
         GUI.enabled = true; // re-enabling the gui for the other buttons
 
         if (GUI.Button (s1,"setting 1")){
                 Debug.Log ("setting 1 was Clicked");
         } 
         if (GUI.Button (s2,"setting 2")){
                 Debug.Log ("setting 2 was Clicked");
         }
 
         }