Run parallel tasks in batches using .NET Core, C# and Async coding

If you have several tasks that you need to run, but each task takes up resources, it can be a good idea to run the tasks in batches. There are a few tools out there, and one of them is the SemaphoreSlim class. The SemaphoreSlim limits the number of concurrent threads with a few easy lines of code.

But first we need an imaginary method that we need to run in parallel:

private async Task DoStuff(string value)
{
  // pseudo code, you just need to imagine
  // that this method executes a task
  await something(value);
}

And we need a bunch of DoStuff values that need to run:

List<string> values = new List<string>();
values.Add("value1");
values.Add("value2");
values.Add("value3");
...
...
...

Now, to start “DoStuff()” with all the values from the “values” list, but run them in batches, we can do this:

using System.Threading;

public async Task BatchRun(List<string> values)
{
    // Limit the concurrent number of 
    // threads to 5
    var throttler = new SemaphoreSlim(5);

    // Create a list of tasks to run
    var tasks = values.Select(async value =>
    {
        // ... but wait for each 5 tasks
        // before running the next 5 tasks
        await throttler.WaitAsync();
        try
        {
            // Run the task
            return await DoStuff(value);
        }
        catch (Exception exception)
        {
            // handle the exception if any
        }
        finally
        {
            // Always release the semaphore
            // when done
            throttler.Release();
        }
    });

    // Now we actually run the tasks
    await Task.WhenAll(tasks);
}

The code will first setup a SemaphoreSlim and ask it to throttle the number of threads to 5. Next it will generate a list of tasks, and use the WaitAsync() to ask the SemaphoreSlim to wait for the first 5 tasks to finish before starting the next 5 tasks. Finally, the Task.WhenAll() method is running the tasks.

The greatness of the SemaphoreSlim lies within the simplicity of the code. You control the batch count one place, and all you need to do is to call WaitAsync() and Release() as extra lines of code. The rest looks like any other task runner.

MORE TO READ:

About briancaos

Developer at Pentia A/S since 2003. Have developed Web Applications using Sitecore Since Sitecore 4.1.
This entry was posted in .net, .NET Core, c#, General .NET and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.