How to make a NavMeshAgent follow player in FixedUpdate to avoid jitters?

So in my game, the player movement and camera follow scripts are done in FixedUpdate. Whenever the player moves, the enemy follows player but jitters. I set enemy's motion in FixedUpdate but still no change. Here is the chaseplayer method called on FixedUpdate:

 public void ChasePlayer() {
         if ( agent.speed == 0 )
         {
             agent.speed = speed;
         }
         agent.SetDestination(player.position);
         float playerEnemyDist = Vector3.Magnitude(player.position - transform.position);
         if ( playerEnemyDist <= attackRange )
         {
             AttackPlayer();
         }
     }

Is the jittering happening because agent.SetDestination is not being called in fixedupdate ?