Audio clip problems

Basically I have a game with two scenes in it , the first scene has an empty object with an audio clip and a script attached to it,the script is a Singleton script that does not destroy on load, which allows the empty object with the audio source to carry on playing between scenes. Everything's works perfectly, but on my second scene i would like a UI button to destroy the script the audio and the empty object attach to the first scene. I have attempt this before, the problem with this is the script has don't destroy on load so when I attempt to destroy it with another script the audio still continues to play because of don't destroy on load. Is there a way around this?