Help with Team style voice chat

I am currently trying to make a team style voice chat system where there can be multiple channels a player can be on and have an announcer that can be heard by everyone. I am currently using DaikonForgeVoIP as the underlying structure, and if possible I would like to keep it. Originally my plan was to attach a team sting to the audio data as it was sent out to the server, and then if the receiving player was a part of that team, it would convey the message.

But the way the the client server relationship works in Unity does not make this possible. since the game objects speaking on both clients send out the same audio message, then the server always returns the same player variable and thus the players can always hear each other, no matter the team.

Does anyone have some advice for another way to approach this issue I am running into, thank you for your time