Instantiated NavMeshAgent prefabs begin behaving oddly to RaycastHit after a few seconds...

I'm just trying to make a simple RTS style game where you select a tile and a unit is instantiated at that location, and then you can select that unit to move them wherever you want them to go. The code works perfectly... for about 5 seconds, then for whatever reason my raycast collision starts to pass through the desired unit so I can no longer click on it. What's stranger is I'm using effectively the same raycast to select the tiles that spawn the units as on the units themselves, but the raycast does not pass through them despite being infinitely thinner. This means it's not simply the raycast somehow going through the unit's collider, but must instead have something to do with the NavMeshAgent I'm using to control them.

Here are some gifs illustrating the problem to make it more clear:

1. https://gyazo.com/bbd3c58a3a7df0051b36a0b236774dab This one shows me clicking on a tile to create a unit. Notice how the tile's material changes when it reads the mouse hovering over it, and is then interrupted once the unit spawns above it.

   ---

2. https://gyazo.com/3a864d520387108105a880e9c959f217 This one is kind of hard to see, but it starts with the unit being spawned in and interrupting the raycast from my mouse to the tile below. Then after a few seconds the tile begins to flash yellow, inferring that somehow the unit's collider is somehow compromised (notice how the tile underneath flickers, meaning some frames recognize the hit and some don't).

   ---

3. https://gyazo.com/2fb9db8a3248b07caff4af98e3b2ca7c And finally here you can see me show how one older unit constantly allows the raycast to pass through it (left) after a certain amount of time, while the newer unit (right) has a few seconds of response before too breaking down.

   ---

using System.Collections; using System.Collections.Generic; using UnityEngine; using UnityEngine.AI;

public class UnitAI: MonoBehaviour { [SerializeField] Vector3 destination;

 NavMeshAgent navAgent;

 public GameObject ground;

 public bool selected;

 void Start()
 {
     navAgent = GetComponent<NavMeshAgent>();

     selected = false;

     //GetComponent<Collider>().enabled = false;
     //GetComponent<Collider>().enabled = true;
     //Note: I added these two lines because it was suggested as a possible solution to what seems like potentially the same problem in this forum post, but it made no difference - https://forum.unity.com/threads/colliders-on-instantiated-objects-stopped-working-after-2018-update.543776/
 }

 void OnMouseOver()
 {
     SelectKohlrabi();
 }

 void FixedUpdate()
 {
     if (selected)
     {
         SetDestination();
     }
 }

 void SetDestination()
 {
     if (Input.GetKeyDown(KeyCode.Mouse1))
     {
         RaycastHit hit;

         if (Physics.Raycast(Camera.main.ScreenPointToRay(Input.mousePosition), out hit, Mathf.Infinity))
         {
             destination = hit.point;

             Vector3 targetVector = destination;

             navAgent.SetDestination(targetVector);

             selected = false;
         }
     }
 }

 void SelectKohlrabi()
 {
     if (Input.GetKeyDown(KeyCode.Mouse0))
     {
         RaycastHit hit;

         if (Physics.SphereCast(Camera.main.ScreenPointToRay(Input.mousePosition), 1.0f, out hit, Mathf.Infinity))
         {
             if (hit.rigidbody.gameObject.tag == "Player") //Note: I'm using hit.rigidbody here instead of hit.transform because it was suggested as a possible solution to what seems like potentially the same problem in this forum post, but it made no difference - https://issuetracker.unity3d.com/issues/moving-a-rigidbody-with-transform-dot-position-causes-its-kinematic-child-object-to-be-misplaced-after-the-movement 
             {
                 if(!selected)
                 {
                     selected = true;
                 }
                 else
                 {
                     selected = false;
                 }
             }
         }
     }
 }

}

I've read about a few other people experiencing what sounds like the same issue, but their solutions are kind of non-solutions and don't even work for me regardless - https://forum.unity.com/threads/colliders-on-instantiated-objects-stopped-working-after-2018-update.543776/. If anybody could help me out with this problem I've been ripping my hair out over this, and am especially annoyed because it's almost certainly a problem on Unity's part which means I can't even chalk it up to my own poor programming skills and hope to stumble upon some simple fix!

Here's the tile script too so you can see the difference:

using System.Collections; using System.Collections.Generic; using UnityEngine;

public class TileScript : MonoBehaviour { Renderer rend;

 public Material dirtMat, hoverMat, growingMat;

 public GameObject Kohlrabi;

 public bool growing;


 void Start () 
 {
     rend = GetComponent<Renderer> ();

     rend.sharedMaterial = dirtMat;

     growing = false;
 }

 void Update()
 {
     if (growing) 
     {
         rend.sharedMaterial = growingMat;

         transform.GetChild (0).gameObject.SetActive (true);

         transform.GetChild (0).localScale += Vector3.one * .1f;

         if(transform.GetChild(0).localScale.x >= 10.0f)
         {
             KohlrabiIsBorn();
         }

     }
     else
     {
         if(rend.sharedMaterial == growingMat)
         {
             rend.sharedMaterial = dirtMat;
         }

         transform.GetChild (0).gameObject.SetActive (false);
         transform.GetChild (0).localScale = Vector3.one;
     }
 }

 void OnMouseOver()
 {
     Debug.Log(transform.name);

     if (rend.sharedMaterial == dirtMat) 
     {
         rend.sharedMaterial = hoverMat;
     }
 }

 void OnMouseExit()
 {
     if (rend.sharedMaterial == hoverMat) 
     {
         rend.sharedMaterial = dirtMat;
     }
 }

 void OnMouseDown()
 {
     if (!growing) 
     {
         growing = true;
     } 
     else 
     {
         growing = false;
     }
 }

 void KohlrabiIsBorn()
 {
     GameObject currentKohlrabi = Instantiate(Kohlrabi, transform.position, Quaternion.identity);
     currentKohlrabi.transform.position = currentKohlrabi.transform.position + Vector3.up;

     growing = false;
 }

}