Zero friction, zero drag, zero gravity bouncy ball in 2D

I'm working in 2D with essentially just two visible objects: a playfield and a ball.

I've got a top-down rectangular playfield which is surrounded by one EdgeCollider2D forming the walls of the playfield. This EdgeCollider2D uses a Material with Friction = 0, and Bounciness =1. Other than the EdgeCollider2D, the playfield only has the Transform and a Sprite Renderer. No other component.

Then I've got the ball. Aside from the Transform, it has a Rigidbody2D with mass = 1, linear drag = 0, angular drag = 0, gravity scale = 0, Material = Sprites-Default; and a CircleCollider2D using the same Material as the EdgeCollider2D of the playfield, i.e. with Friction = 0, and Bounciness = 1.

Then I've got a script which applies a once-off force to the ball, setting it in motion on the playfield.

Using the once-off force, when I set the ball off such that it hits the walls mostly at 90 degree angles, then it keeps bouncing around without loosing its initial speed. This is what I would expect as I've set all the frictions and drags = 0 and all the bounciness = 1.

However, when I set the ball off such that it hits the walls at small angles (of just a few degrees), then the speed of the ball changes. It sometimes increases, sometimes decreases. At times, it's come to a complete stand still as it kept loosing speed.

Why would it loose speed if I've set all the frictions and drags = 0 and all the bounciness = 1? And what can I do to make the ball and playfield truly zero friction and zero drag, so that no matter what angle it hits the walls it will never gain or loose its initial speed, which it reached by applying the initial once-off force?

I had the hunch there there might be some rotational force or spin at play, however, I cannot find any reference to such in any of the parameters of the Rigidbody2D or Collider2D or Material.

As a workaround for the time being to keep the speed of the bouncy ball constant, I've included a line of code in FixedUpdate which manually sets the rigidbody's speed to a constant value:

void FixedUpdate () { rigidBodyRef.velocity = rigidBodyRef.velocity.normalized * 3; }

However, I wonder whether this manual setting of the rigidbody's speed interferes with Unity's physics engine, and whether there are any other undesirable consequences I'm not aware of. It certainly doesn't seem like a good way of creating a zero friction, zero drag, zero gravity 2D bouncy ball.