Write to file from multiple threads async with C# and .NET Core

There are several patterns on how to allow multiple threads to write to the same file. the ReaderWriterLock class is invented for this purpose. Another classic is using semaphors and the lock statement to lock a shared resource.

This article explains how to use a ConcurrentQueue and a always running Task to accomplish the same feat.

The theory behind this is:

  • Threads deliver what to write to the file to the ConcurrentQueue.
  • A task running in the background will read from the ConcurrentQueue and do the actual file writing.

This allows the shared resource to be access from one thread only (the task running in the background) and everyone else to deliver their payload to a thread-safe queue.

But enough talk, lets code.

THE FILE WRITER CLASS

using System.Collections.Concurrent;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace MyCode
{
  public class MultiThreadFileWriter
  {
    private static ConcurrentQueue<string> _textToWrite = new ConcurrentQueue<string>();
    private CancellationTokenSource _source = new CancellationTokenSource();
    private CancellationToken _token;

    public MultiThreadFileWriter()
    {
      _token = _source.Token;
      // This is the task that will run
      // in the background and do the actual file writing
      Task.Run(WriteToFile, _token);
    }

    /// The public method where a thread can ask for a line
    /// to be written.
    public void WriteLine(string line)
    {
      _textToWrite.Enqueue(line);
    }

    /// The actual file writer, running
    /// in the background.
    private async void WriteToFile()
    {
      while (true)
      {
        if (_token.IsCancellationRequested)
        {
          return;
        }
        using (StreamWriter w = File.AppendText("c:\\myfile.txt"))
        {
          while (_textToWrite.TryDequeue(out string textLine))
          {
            await w.WriteLineAsync(textLine);
          }
          w.Flush();
          Thread.Sleep(100);
        }
      }
    }
  }
}

// Somewhere in the startup.cs or the Main.cs file
services.AddSingleton<MultiThreadFileWriter>();
// Now you can add the class using constructor injection
// and call the WriteLine() function from any thread without
// worrying about thread safety

Nothice that my code introduces a Thread.Sleep(100) statement. This is not needed, but it can be a good idea to give your application a little breathing space, especially if there are periods where nothing is written. Remove the line if your code requires an instant file write pattern.

MORE TO READ:

About briancaos

Developer at Pentia A/S since 2003. Have developed Web Applications using Sitecore Since Sitecore 4.1.
This entry was posted in .net, .NET Core, c#, General .NET and tagged , , , , . Bookmark the permalink.

6 Responses to Write to file from multiple threads async with C# and .NET Core

  1. Zac says:

    I keep getting an error
    System.ArgumentException: Destination array was not long enough. Check…..

    I can’t seem to track down the exact reason for this. I currently have two threads outputting data. However I may have more in the future so a multithread file logger is critical for me.

    Doesn’t ConcurrentQueue specific manage multithread lists for this?

    Like

  2. briancaos says:

    You should check that the MultiThreadFileWriter is implemented as a singleton, by either instantiating it as such in the services collection (services.AddSingleton();), or as a static variable somewhere in your code.

    Thread errors are hard to debug. I’m not sure that the “System.ArgumentException: Destination array was not long enough” error is part of MultiThreadFileWriter as this error usually happens in Array.Copy methods.

    Like

  3. Dv says:

    Thank you for this logic. I used this logic in my wpf application which has two tasks running and writing to the same file when I run this app on window server, it occupies 97% of CPU because of the While(true) loop.can you please tell me if it’s possible to reduce the CPU usage?

    Like

  4. briancaos says:

    The code comes from a .NET Core 3.1 worker process, not a WPF application, so I am curious as whether WPF handles Task.Run diffrently.

    The loop should not use 97% of your CPU, especially if you use the Thread.Sleep(100) method as well. Try commenting out some of the code inside the while(true) loop until the CPU usage drops.

    Also, check to ensure that you instantiate the MultiThreadFileWriter class as a singleton, not a transient.

    Like

  5. Pingback: Thread safe writing to text file – YOGENDRA GAUTAM

  6. Pingback: C# Thread Safe File Writer and Reader | Brian Pedersen's Sitecore and .NET Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.