Run tasks in parallel using .NET Core, C# and async coding

If you have several tasks that can be run in parallel, but still need to wait for all the tasks to end, you can easily achieve this using the Task.WhenAll() method in .NET Core.

Imagine you have this imaginary method that takes a URL and uploads the file to some location:

private async Task UploadFile(string fileName)
{
  // pseudo code, you just need to imagine
  // that this metod executes a task
  if (file exists)
    await _fileRepository.UploadFile(fileName);
}

RUN ONCE:

This method can be called from your main method:

private static async Task Main(string[] args)
{
  await UploadFile("c.\\file.txt");
}

RUN IN SEQUENCE:

If you have 2 files to be uploaded you can call it twice:

private static async Task Main(string[] args)
{
  await UploadFile("c.\\file.txt");
  await UploadFile("c.\\file2.txt");
}

This will upload the first file, then the next file. There is no parallelism here, as the “async Task” does not automatically make something run in in parallel.

RUN IN PARALLEL:

But with Task.WhenAll() you can run both at the same time in parallel:

private static async Task Main(string[] args)
{
  var task1 = UploadFile("c.\\file.txt");
  var task2 = UploadFile("c.\\file2.txt");
  await Task.WhenAll(task1, task2);
}

This will spawn 2 threads, run them simultaneously, and return when both threads are done.

RUN IN PARALLEL THE FLEXIBLE WAY:

If you want even more flexibility, you can call it using an IEnumerable list of objects:

private static async Task Main(string[] args)
{
  List<string> fileNames = new List<string>();
  fileNames.Add("c:\\file.txt");
  fileNames.Add("c:\\file2.txt");
  var tasks = fileNames.Select(f => UploadFile(f));
  await Task.WhenAll(tasks);
}

This will create a list of Tasks to be run at the same time. You can add many filename to the fileNames list and have them run, each of them in their own thread.

RUN IN PARALLEL IN BATCHES:

Beware of the limitations of threading. Spawning a thread have a small but significant overhead, and running too many threads at once could be slower than running them in sequence. If you have 100’s of files to be uploaded, you should run the tasks in batches:

private static async Task Main(string[] args)
{
  List<string> fileNames = new List<string>();
  fileNames.Add("c:\\file.txt");
  fileNames.Add("c:\\file2.txt");
  // ... adding 100's of files

  var batchSize = 10;
  int batchCount = (int)Math.Ceiling((double)userIds.Count() / batchSize);
  for(int i = 0; i < batchCount; i++)
  {
    var filesToUpload = fileNames.Skip(i * batchSize).Take(batchSize);
    var tasks = filesToUpload.Select(f => UploadFile(f));
    await Task.WhenAll(tasks));
  }
}

This will spawn 10 threads and wait for them to finish before taking the next 10.

MORE TO READ:

About briancaos

Developer at Pentia A/S since 2003. Have developed Web Applications using Sitecore Since Sitecore 4.1.
This entry was posted in .NET Core, c# and tagged , , , . Bookmark the permalink.

1 Response to Run tasks in parallel using .NET Core, C# and async coding

  1. Pingback: C# get results from Task.WhenAll | Brian Pedersen's Sitecore and .NET Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.