Smoothly move continuous moving object

Currently I am working on clone of Ping Pong type game.
I want to move my ball continuously on screen. At present I have code for ball movement but it looks jittery.

public class ContinuousMoveSinglePlayer : MonoBehaviour
{
	private bool isStartMoving;
	private Vector3 direction;
	private float speed = 10f;
	private int opponentPaddleCounter;

	// Use this for initialization
	void Start ()
	{
		InitializeValues ();
	}

	private void InitializeValues ()
	{
		float xDirection = Random.Range (0, 2) * 2 - 1;
		float yDirection = Random.Range (0, 2) * 2 - 1;
		direction = new Vector3 (xDirection, yDirection).normalized;
	}

	// Update is called once per frame
	void FixedUpdate ()
	{
		if (isStartMoving) {
			rigidbody2D.velocity = direction * speed;
		}
	}
	
	void OnCollisionEnter2D (Collision2D otherCollision)
	{
		if (otherCollision.gameObject.CompareTag (Constants.TAG_OPPONENT_PADDLE)) {
			opponentPaddleCounter++;
			if (opponentPaddleCounter > Constants.BALL_SPEED_LIMIT) {
				opponentPaddleCounter = 0;
				speed += 1f;
			}
		}

		Vector3 normal = otherCollision.contacts [0].normal;
		direction = Vector3.Reflect (direction, normal);
	}
}

I want to move ball smoothly on screen. I already tried with following code

transform.position = Vector3.Lerp(transform.position,(transform.position + direction), Time.deltaTime * speed);

But above line creates poor collision response. It detect collision with some time delay so that it looks likes ball go inside the wall.
I need your suggestions to solve this problem.

Multiplying by deltaTime might help.

rigidbody2D.velocity = direction * speed * Time.deltaTime;