uNET - Send big amount of data over network (How to split send byte[]?)

hi all.

I’m trying to send an object from server to client. the object holds (beside some strings) a byte array which represents an image file. Thus, the objects can be quite huge (depending on the image, e.g. like 300kb). I tried to send it via RPC as custom struct which failed because of the size. Now i’m sending it as NetworkMessage via NetworkServer.SendByChannelToAll on the reliableFragmented channel. I’ve set the connectionConfig.FragmentSize to 1024. This works for small images. Big images are not being transferred completely, so that only some pixels of the image are shown on the client.

Is there a better way to send this huge amount of data?
I thought about serializing the whole object as byte array, split it in chunks and set them one after another, but I’m not sure in how to do that. Is there any example out there?

Cheers,
Phineliner

I figured it out by myself. Roughly it can be achieved as follows:

  • I serialize the object into one big byte array which then is sent in several chunks (of 1024 byte) to all clients.

  • I use ClientRPC to send the chunks.

  • I use a ReliableSequenced channel / qostype, to ensure the chunks arrive in the right order.

  • On client side I collect the chunks, add them up to a big byte array again and then I deserialize it back to the object.

Edit:

As I was asked to provide some sample code, here you go:

This is my NetworkTransmission Component:

public class NetworkTransmitter : NetworkBehaviour {

    private static readonly string LOG_PREFIX = "[" + typeof(NetworkTransmitter).Name + "]: ";
    public const int RELIABLE_SEQUENCED_CHANNEL = MyNetworkManager.CHANNEL_RELIABLE_SEQUENCED;
    private static int defaultBufferSize = 1024; //max ethernet MTU is ~1400

    private class TransmissionData{
        public int curDataIndex; //current position in the array of data already received.
        public byte[] data;

        public TransmissionData(byte[] _data){
            curDataIndex = 0;
            data = _data;
        }
    }

    //list of transmissions currently going on. a transmission id is used to uniquely identify to which transmission a received byte[] belongs to.
    List<int> serverTransmissionIds = new List<int>();

    //maps the transmission id to the data being received.
    Dictionary<int, TransmissionData> clientTransmissionData = new Dictionary<int,TransmissionData>();

    //callbacks which are invoked on the respective events. int = transmissionId. byte[] = data sent or received.
    public event UnityAction<int, byte[]> OnDataComepletelySent;
    public event UnityAction<int, byte[]> OnDataFragmentSent;
    public event UnityAction<int, byte[]> OnDataFragmentReceived;
    public event UnityAction<int, byte[]> OnDataCompletelyReceived;

    [Server]
    public void SendBytesToClients(int transmissionId, byte[] data)
    {
        Debug.Assert(!serverTransmissionIds.Contains(transmissionId));
        StartCoroutine(SendBytesToClientsRoutine(transmissionId, data));
    }

    [Server]
    public IEnumerator SendBytesToClientsRoutine(int transmissionId, byte[] data)
    {
        Debug.Assert(!serverTransmissionIds.Contains(transmissionId));
        Debug.Log(LOG_PREFIX + "SendBytesToClients processId=" + transmissionId + " | datasize=" + data.Length);

        //tell client that he is going to receive some data and tell him how much it will be.
        RpcPrepareToReceiveBytes(transmissionId, data.Length);
        yield return null;

        //begin transmission of data. send chunks of 'bufferSize' until completely transmitted.
        serverTransmissionIds.Add(transmissionId);
        TransmissionData dataToTransmit = new TransmissionData(data);
        int bufferSize = defaultBufferSize; 
        while (dataToTransmit.curDataIndex < dataToTransmit.data.Length-1)
        {
            //determine the remaining amount of bytes, still need to be sent.
            int remaining = dataToTransmit.data.Length - dataToTransmit.curDataIndex;
            if (remaining < bufferSize)
                bufferSize = remaining;

            //prepare the chunk of data which will be sent in this iteration
            byte[] buffer = new byte[bufferSize];
            System.Array.Copy(dataToTransmit.data, dataToTransmit.curDataIndex, buffer, 0, bufferSize);

            //send the chunk
            RpcReceiveBytes(transmissionId, buffer);
            dataToTransmit.curDataIndex += bufferSize;

            yield return null;

            if (null != OnDataFragmentSent)
                OnDataFragmentSent.Invoke(transmissionId, buffer);
        }

        //transmission complete.
        serverTransmissionIds.Remove(transmissionId);

        if (null != OnDataComepletelySent)
            OnDataComepletelySent.Invoke(transmissionId, dataToTransmit.data);
    }

    [ClientRpc]
    private void RpcPrepareToReceiveBytes(int transmissionId, int expectedSize)
    {
        if (clientTransmissionData.ContainsKey(transmissionId))
            return;

        //prepare data array which will be filled chunk by chunk by the received data
        TransmissionData receivingData = new TransmissionData(new byte[expectedSize]);
        clientTransmissionData.Add(transmissionId, receivingData);
    }

    //use reliable sequenced channel to ensure bytes are sent in correct order
    [ClientRpc(channel = RELIABLE_SEQUENCED_CHANNEL)]
    private void RpcReceiveBytes(int transmissionId, byte[] recBuffer)
    {
        //already completely received or not prepared?
        if (!clientTransmissionData.ContainsKey(transmissionId))
            return;

        //copy received data into prepared array and remember current dataposition
        TransmissionData dataToReceive = clientTransmissionData[transmissionId];
        System.Array.Copy(recBuffer, 0, dataToReceive.data, dataToReceive.curDataIndex, recBuffer.Length);
        dataToReceive.curDataIndex += recBuffer.Length;

        if (null != OnDataFragmentReceived)
            OnDataFragmentReceived(transmissionId, recBuffer);

        if (dataToReceive.curDataIndex < dataToReceive.data.Length - 1)
            //current data not completely received
            return;

        //current data completely received
        Debug.Log(LOG_PREFIX + "Completely Received Data at transmissionId=" + transmissionId);
        clientTransmissionData.Remove(transmissionId);

        if (null != OnDataCompletelyReceived)
            OnDataCompletelyReceived.Invoke(transmissionId, dataToReceive.data);
    }
}

And this is how one would use the component (make sure it is attached to the same game-object of the script that wants to send the data / has a network identity):

//inside a class, derived from network behaviour, which has the NetworkTransmitter component attached...

...

//on client and server
NetworkTransmitter networkTransmitter = GetComponent<NetworkTransmitter>();

...

//on client: listen for and handle received data
networkTransmitter.OnDataCompletelyReceived += MyCompletelyReceivedHandler;
networkTransmitter.OnDataFragmentReceived += MyFragmentReceivedHandler;

...

//on server: transmit data. myDataToSend is an object serialized to byte array.
StartCoroutine(networkTransmitter.SendBytesToClientsRoutine(0, myDataToSend))

...

//on client this will be called once the complete data array has been received
[Client]
private void MyCompletelyReceivedHandler(int transmissionId, byte[] data){
      //deserialize data to object and do further things with it...
}

//on clients this will be called every time a chunk (fragment of complete data) has been received
[Client]
private void MyFragmentReceivedHandler(int transmissionId, byte[] data){
        //update a progress bar or do something else with the information
}

On how to Serialize an object to byte array, you will find help in other threads.

Hope it helps,
Cheers phineliner

You can also use a fragmented (ReliableFragmented) channel to send bigger chunks up to a point.

Hello @phineliner. Is it possible to reverse the functionality of this script? Rather than sending a large chunk of data to clients, could I send a large chunk of data to the server? I’ve been trying to do this with this script but I keep getting the error: Trying to send command for object without authority on the CmdPrepareToReceiveBytes() and CmdReceiveBytes() functions. Any ideas how I might get this to work?

Hi and thx for the nice example
Is there a simple way to prevent sending the data to the local client created when starting a host ?,Hy… Thx for the very nice example.
Is there a possibility to prevent sending the data to the local client created starting the host ?

Full example here:

hi If you just want to split a large array, you can use the following package.
network Tool :