Write to file from multiple threads async with C# and .NET Core

There are several patterns on how to allow multiple threads to write to the same file. the ReaderWriterLock class is invented for this purpose. Another classic is using semaphors and the lock statement to lock a shared resource.

This article explains how to use a ConcurrentQueue and a always running Task to accomplish the same feat.

The theory behind this is:

  • Threads deliver what to write to the file to the ConcurrentQueue.
  • A task running in the background will read from the ConcurrentQueue and do the actual file writing.

This allows the shared resource to be access from one thread only (the task running in the background) and everyone else to deliver their payload to a thread-safe queue.

But enough talk, lets code.

THE FILE WRITER CLASS

using System.Collections.Concurrent;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace MyCode
{
  public class MultiThreadFileWriter
  {
    private static ConcurrentQueue<string> _textToWrite = new ConcurrentQueue<string>();
    private CancellationTokenSource _source = new CancellationTokenSource();
    private CancellationToken _token;

    public MultiThreadFileWriter()
    {
      _token = _source.Token;
      // This is the task that will run
      // in the background and do the actual file writing
      Task.Run(WriteToFile, _token);
    }

    /// The public method where a thread can ask for a line
    /// to be written.
    public void WriteLine(string line)
    {
      _textToWrite.Enqueue(line);
    }

    /// The actual file writer, running
    /// in the background.
    private async void WriteToFile()
    {
      while (true)
      {
        if (_token.IsCancellationRequested)
        {
          return;
        }
        using (StreamWriter w = File.AppendText("c:\\myfile.txt"))
        {
          while (_textToWrite.TryDequeue(out string textLine))
          {
            await w.WriteLineAsync(textLine);
          }
          w.Flush();
          Thread.Sleep(100);
        }
      }
    }
  }
}

// Somewhere in the startup.cs or the Main.cs file
services.AddSingleton<MultiThreadFileWriter>();
// Now you can add the class using constructor injection
// and call the WriteLine() function from any thread without
// worrying about thread safety

Nothice that my code introduces a Thread.Sleep(100) statement. This is not needed, but it can be a good idea to give your application a little breathing space, especially if there are periods where nothing is written. Remove the line if your code requires an instant file write pattern.

MORE TO READ:

About briancaos

Developer at Pentia A/S since 2003. Have developed Web Applications using Sitecore Since Sitecore 4.1.
This entry was posted in .net, .NET Core, c#, General .NET and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.