Custom Editor not Persisting Changes to Prefab

So, I am creating a procedural iso tile map generator. I created a custom editor for tiles. You can attach a sprite to a tile, specify it's z order, and give a list of tile indices that can be next to it in each direction (N, NW, W, SW, S, SE, E, NE). The problem is that the tile indices per direction will not persist in a prefab.

The relevant code for Tile is:

 public class Tile : MonoBehaviour {
   public enum Direction { NORTH, NORTHEAST, EAST, SOUTHEAST, SOUTH, SOUTHWEST, WEST, NORTHWEST};
   //hack
   [System.Serializable]
   public class SillyHashSet : HashSet<int> { }
   [System.Serializable]
   public class SillyDictionary : Dictionary<Direction, SillyHashSet> { }
 
   public int zOrder = 0;
   public SillyDictionary neighborTiles;
   public Sprite sprite;
 
   public void Awake() {
 
   }
 }

I had to use that inheritance hack to get the special instance of templating to serialize into the unity editor. Here is the relevant part of my editor script.

 [CustomEditor(typeof(Tile))]
 public class TileEditor : Editor {
     private Tile.Direction direction = Tile.Direction.NORTH;
     private int addingInt = 0;
     public override void OnInspectorGUI() {
         Tile tile = (Tile)target;
         if(tile.neighborTiles == null) {
             tile.neighborTiles = new Tile.SillyDictionary();
         }
         addingInt = EditorGUILayout.IntField("Add Tile Index As Neighbor", addingInt);
         EditorGUILayout.BeginHorizontal();
         if(GUILayout.Button("Add")) {
             if(!tile.neighborTiles.ContainsKey(direction)) {
                 tile.neighborTiles[direction] = new Tile.SillyHashSet();
             }
             tile.neighborTiles[direction].Add(addingInt);
         }
         EditorGUILayout.EndHorizontal();
         if(tile.neighborTiles.ContainsKey(direction)) {
             List<int> removes = new List<int>();
             foreach(int index in tile.neighborTiles[direction]) {
                 EditorGUILayout.BeginHorizontal();
               EditorGUILayout.PrefixLabel(index.ToString());
               if(GUILayout.Button("Remove")) {
                   removes.Add(index);
               }
               EditorGUILayout.EndHorizontal();
             }
             foreach(int r in removes) {
                 tile.neighborTiles[direction].Remove(r);
             }
         }
     }
 }

What could be going wrong here?