Audio playing for a very long time

I am spawning an audio source object for each collision between two objects of a certain type. These collisions happen often.

The problem is that the audio is about 3s long, but for some reason it sometimes takes a long time to play and die (I have a script that destroys the object when it finishes playing). The scene ends up having a ton of them not dying and they are screwing up sound playback. I have done some debugging and in some cases it takes 20s+ to run the 3s of the clip.

Probably worth noting that:

1. A lot of these are spawned.

2. I am tweaking the pitch per collision in relation to the impact of the collision.

3. I've put the priority to 255 so these don't block the "main" sounds of the game.

Does that ring any bells to anyone?