Color change of GUI button in custom property drawer is unresponsive

When I try to make a GUI button in the Unity Editor change colors on hover and click, the color change is successful but horribly delayed. Is there another workaround for making the button change color in a way that is much more responsive?

Here's the minimum code needed to reproduce the example:

 using UnityEngine;
 public class Test : MonoBehaviour
 {
    
 }

 using UnityEngine;
 using UnityEditor;
 
 [CustomEditor(typeof(Test))]
 public class TestEditor : Editor
 {
     public override void OnInspectorGUI()
     {
         base.OnInspectorGUI();
 
         GUIStyle style = new GUIStyle()
         {
             alignment = TextAnchor.MiddleCenter,
             fontStyle = FontStyle.Bold,
 
             normal = new GUIStyleState()
             {
                 background = Texture2D.whiteTexture
             },
             hover = new GUIStyleState()
             {
                 background = Texture2D.grayTexture
             },
             active = new GUIStyleState()
             {
                 background = Texture2D.blackTexture
             }
         };
 
         GUILayout.Button("Hello!", style);
     }
 }

You can then attach the "Test" object to a GameObject in the Unity editor and get this editor:

As you can see in this gif, it takes the button a little while to respond to the presence of the mouse