Downloading multiple textures with a single http request (UnityWebRequest)

Getting a one single texture is not a problem. i’ve figured out how to do that. But i need a few thousands for which i dynamically generate url…

The problem is that by making an http request for each one, after maybe 50 of them load i start getting http error 429 “too many requests”.

How should i deal with this? I can’t find in unity documentation anything for downloading multiple files with a single request. (maybe i just don’t understand it)

You can use a load balancer like this one which I’ve quickly thrown:together:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Networking;

public class LoadBalancer<T>
{
    public class Request
    {
        public UnityWebRequest request;
        public UnityWebRequestAsyncOperation response = null;
        public T data;
        public System.Action<UnityWebRequestAsyncOperation, T> callback;
        public Request(UnityWebRequest aRequest, T aData,  System.Action<UnityWebRequestAsyncOperation, T> aCallback)
        {
            request = aRequest;
            callback = aCallback;
            data = aData;
        }
    }
    private Queue<Request> m_Queue = new Queue<Request>();
    private List<Request> m_ActiveList = new List<Request>();
    private MonoBehaviour m_Host;
    private Coroutine m_Coroutine = null;
    private int m_MaxTasks;
    public bool IgnoreMissingCallback = false;
    public System.Action<UnityWebRequestAsyncOperation, T> defaultCallback;
    public LoadBalancer(MonoBehaviour aHost, int aMaxTasks, System.Action<UnityWebRequestAsyncOperation, T> aDefaultCallback = null)
    {
        m_Host = aHost;
        m_MaxTasks = aMaxTasks;
        defaultCallback = aDefaultCallback;
    }

    public void AddTask(UnityWebRequest aTask, T aData = default, System.Action<UnityWebRequestAsyncOperation, T> aCallback = null)
    {
        if (aCallback == null)
        {
            aCallback = defaultCallback;
            if (!IgnoreMissingCallback && aCallback == null)
                Debug.LogWarning("Added task without callback");
        }
        m_Queue.Enqueue(new Request(aTask, aData,  aCallback));
        if (m_Coroutine == null)
            m_Coroutine = m_Host.StartCoroutine(RunRequests());
    }

    public void AbortAllRequests()
    {
        m_Host.StopCoroutine(m_Coroutine);
        foreach (var request in m_ActiveList)
            request.request.Dispose();
        m_ActiveList.Clear();
        m_Queue.Clear();
    }

    private IEnumerator RunRequests()
    {
        while (m_Queue.Count > 0 || m_ActiveList.Count > 0)
        {
            while (m_ActiveList.Count < m_MaxTasks && m_Queue.Count > 0)
            {
                var request = m_Queue.Dequeue();
                request.response = request.request.SendWebRequest();
                m_ActiveList.Add(request);
            }
            for(int i = 0; i < m_ActiveList.Count; i++)
            {
                var request = m_ActiveList*;*

if (request.response.isDone)
{
m_ActiveList = m_ActiveList[m_ActiveList.Count - 1];
m_ActiveList.RemoveAt(m_ActiveList.Count - 1);
if (request.callback != null)
request.callback(request.response, request.data);
}
}
yield return null;
}
m_Coroutine = null;
}
}
You can create an instance of this loader, specify the max number of parallel tasks and just add as many tasks you like. Each task can have its own callback method as well as a custom data instance. The callback receives the web response as well as the custom data instance. When no custom callback is provided, it will use the default callback specified at the load balancer itself.
Here’s an example script that uses the load balancer:
public class LoadImages : MonoBehaviour
{
LoadBalancer loader;
int xPos = 0;
void Start()
{
loader = new LoadBalancer(this, 10, HandleResponse);
for (int i = 0; i < 100; i++)
{
loader.AddTask(LoadImage(“https://upload.wikimedia.org/wikipedia/commons/thumb/c/c4/Unity_2021.svg/1200px-Unity_2021.svg.png”), CreateCube(xPos++));
loader.AddTask(LoadImage(“https://upload.wikimedia.org/wikipedia/commons/thumb/1/19/Unity_Technologies_logo.svg/800px-Unity_Technologies_logo.svg.png”), CreateCube(xPos++));
loader.AddTask(LoadImage(“https://i1.wp.com/joyofandroid.com/wp-content/uploads/2019/01/powered-by-unity1-1024x587.jpg?w=696&ssl=1”), CreateCube(xPos++));
}
}
UnityWebRequest LoadImage(string aURL)
{
var request = UnityWebRequest.Get(aURL);
request.downloadHandler = new DownloadHandlerTexture();
return request;
}

GameObject CreateCube(int aXPos)
{
var cube = GameObject.CreatePrimitive(PrimitiveType.Cube);
cube.transform.position = new Vector3(1.1f * aXPos, 0, 0);
return cube;
}

void HandleResponse(UnityWebRequestAsyncOperation aResponse, GameObject aGO)
{
Debug.Log($“Completed: {aResponse.webRequest.url} data: {aResponse.webRequest.downloadedBytes} bytes”);
aGO.GetComponent().material.mainTexture = DownloadHandlerTexture.GetContent(aResponse.webRequest);
}
}
This will load the same 3 images 100 times and assigns them to cube objects. So the custom data object in this examples is simply a cube GameObject. However you could use whatever class you like. As you can see I simply use a single callback that will handle each request. Since the callback gets the request as well as the custom data object you know what request you’re handling.
The LoadBalancer only runs a single coroutine which handles all the requests. The internal queue holds the queued tasks that has not yet been started and the active list holds the requests that are currently processed. The balancer would only schedule “MaxTasks” requests at the same time. Once a request has finished, if will load the next from the queue. The coroutine autostarts and terminates as necessary. The required MonoBehaviour “host” argument is used to actually run the coroutine.
pps: I’m wondering when you load thousands of images, how large are those images? Because loading that many textures you could easily run into memory issues.

Unity Does not provide a method for Requesting and downloading Multiple files with one Request.
You can control how many request you do have running at the same time in order not to overload the server.

If you control the server another option is the Request a list of images via a small PHP script that then can package the requested images into a container like a zip file , then unity gets back the URL for the container , downloads it , extracts the images and the load it via multiple webreqeuest from disk.