C# Azure TelemetryClient will leak memory if not implemented as a singleton

I noticed that my classic .net web application would leak memory after I implemented metrics for some background tasks.

Memory usage of web application

Memory usage of web application

Further investigation showed that my MetricAggregationManager would not release its memory.

Object was not garbage collected

Object was not garbage collected

Since one of the major changes was the implementation of a TelemetryClient, and since the memory not being released was from the Microsoft.ApplicationInsights.Metrics namespace, I concluded that the problem lies within the creation of the TelemetryClient:

using System;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Metrics;

namespace MyCode
{
  public class BaseProcessor
  {
    private readonly TelemetryClient _telemetryClient;
    
    private BaseProcessor()
    {
      string instrumentationKey = "somekey"
      var telemetryConfiguration = new TelemetryConfiguration { InstrumentationKey = instrumentationKey };
      // This is a no-go. I should not create a new instance for every BaseProcessor
      _telemetryClient = new TelemetryClient(telemetryConfiguration);
    }
  }
}

The code above will create a new TelemetryClient for each creation of my base class. The TelemetryClient will collect metrics and store those in memory until either a set time or number of metrics are met, and then dump the metrics to Application Insights.

So when the BaseClass is disposed, TelemetryClient is not, leaving memory to hang, and thus a memory leak is in effect.

HOW TO SOLVE IT?

The solution is simple. All you need to do is to create a singleton pattern for your TelemetryClient. Having only one instance will allow the client to collect and send metrics in peace. Your code will be much faster (it takes a millisecond or so to create a TelemetryClient) and you will not have any memory leaks.

USE DEPENDENCY INJECTION:

In .NET Core you can add the TelemetryClient to the service collection:

private static void ConfigureServices(IServiceCollection services)
{
  // Add Application Insights
  var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
  telemetryConfiguration.InstrumentationKey = "somekey"
  var telemetryClient = new TelemetryClient(telemetryConfiguration);
  services.AddSingleton(telemetryClient);
}

And then reference it using constructor injection:

using System;
using System.Runtime.Serialization;
using Microsoft.ApplicationInsights;
using Microsoft.AspNetCore.Mvc;

namespace MyCode
{
  [ApiController]
  [Route("/api/[controller]")]
  [Produces("application/json")]
  public class MyController : ControllerBase
  {
    private readonly TelemetryClient _telemetryClient;

    public MyController(TelemetryClient telemetryClient)
    {
      _telemetryClient = telemetryClient;
    }
  }
}

USE A STATIC VARIABLE:

If you do not have access to a DI framework, you could also just create a static variable:

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using System.Collections.Generic;

namespace MyCode
{
  public static class TelemetryFactory
  {
    private static TelemetryClient _telemetryClient;

    public static TelemetryClient GetTelemetryClient()
    {
      if (_telemetryClients == null)
      {
        string instrumentationKey = "somekey";
        var telemetryConfiguration = new TelemetryConfiguration { InstrumentationKey = instrumentationKey };
        _telemetryClient = new TelemetryClient(telemetryConfiguration);
      }

      return _telemetryClient;
    }
  }
}

And then reference the static variable instead:

using System;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Metrics;

namespace MyCode
{
  public class BaseProcessor
  {
    private readonly TelemetryClient _telemetryClient;
    
    private BaseProcessor()
    {
      _telemetryClient = TelemetryFactory.GetTelemetryClient();
    }
  }
}

MORE TO READ:

 

Posted in .net, .NET Core, c#, General .NET, Microsoft Azure | Tagged , , , | Leave a comment

Sitecore Memory Issues – Every memory optimization trick in the book

My recent post, Sitecore high memory usage – not always a problem, claims that high memory usage is not the same as having a memory problem in Sitecore. But the reason you are reading this blog is because the memory have become a problem – right? So let’s see what you can do about it.

I will not go into the real issue, which probably is that your own application code have a memory leak. This not Sitecore’s fault, although we would like it to be. Instead I will show you the Sitecore settings that will limit the memory consumption of the Sitecore platform itself.

TRICK 1: YOUR EVENT QUEUE IS RUNNING HIGH

This is the #1 issue: Too many events in the event queue. First you need to clean the event queue. Read how to clean the event queue here.

Next you need to manage the event queue size. Use the CleanupEventQueue agent and tune the cleanup so you have no more than 7000 items in the queue at any time:

<agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00">
  <IntervalToKeep>04:00:00</IntervalToKeep>
  <DaysToKeep>1</DaysToKeep>
</agent>

TRICK 2: DON’T LET THE SITECORE CACHE RUN WILD

Use the Caching.DisableCacheSizeLimits to enforce the cache limits you have painfully configured through the application:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

Also use one of the many Sitecore Cache Tuning Guides online. Even old blog posts will still be useful, as the cache is part of Sitecore basics and have not changed much.

TRICK 3: YOU HAVE TOO MANY UNUSED VERSIONS OF YOUR ITEMS

Items with many versions will eat your memory every time you open the item in the Sitecore Shell. If possible, delete the old versions. Click here to see how to delete old Sitecore item versions.

TRICK 4: DISABLE EXM

Are you using EXM? No? Then disable EXM:

<setting name="EXM.Enabled" value="false" />

TRICK 6: DO NOT CACHE ITEMS WHEN PUBLISHING

You can disable the Publishing database cache. Then the items being published will not be added to the Sitecore database cache.

<setting name="Publishing.DisableDatabaseCaches" value="true"/>

TRICK 7: LOWER THE BATCH SIZE WHEN INDEXING

The Sitecore SOLR indexing can eat your memory. Lowering the batch size have a positive effect on the amount of memory being used when an item is indexed:

<setting name="ContentSearch.IndexUpdate.BatchSize" value="150" />

TRICK 8: DO NOT CACHE WHEN INDEXING

You can disable the index caching. Then the items being retrieved during an index is not cached:

<setting name="ContentSearch.Indexing.DisableDatabaseCaches" value="true" />

TRICK 9: UTILIZE INTERNING

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. Sitecore have implemented interning on fields, but you have to define which fields should be interned. Legend says that there is a performance penalty when interning, but I have not been able to measure any performance decrease.

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

TRICK 10: FORCE GARBAGE COLLECTION WHEN MEMORY IS LOW

The old peeing in your pants trick is also implemented by Sitecore. You need to enable counters first:

<setting name="Counters.Enabled" value="true"/>

Then you can allow the MemoryMonitorHook to force a garbage collection when memory is running low:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>

I guess that’s it. If you know of a memory trick for Sitecore, please let me know.

MORE TO READ:

 

Posted in Sitecore 6, Sitecore 7, Sitecore 8, Sitecore 9 | Tagged , , , , , | Leave a comment

Sitecore high memory usage – not always a problem

Sitecore is known to be a real memory hog. Especially the CM server is known to love memory. One of the memory issues that Sitecore faces is the high usage of the Large Object Heap.

Heap Fragmentation

Heap Fragmentation

WHY IS THIS A PROBLEM?

The .net garbage collector will start collecting memory from Generation 0. Objects that cannot be collected is upgraded to Generation 1, which is then collected later. Objects can be further upgraded to Generation 2, assuming that these objects are long-lived and does not need to be collected as often.

The Large Object Heap is where objects larger than 85.000 bytes live. This is mostly strings and arrays, and these objects are rarely garbage collected, and the heap is never compacted (not entirely true as newer .net versions can compact this heap) with can lead to high memory usage.

WHY THIS IS PROBABLY NOT A PROBLEM ANYWAY?

Sitecore is well-known to be a large object heap eater, because Sitecore stores fields and field values as arrays of strings, making every Item a large object. And this is a problem – right?

It definitely can be a problem – but not necessarily:

  • The Large Object heap IS garbage collected, but not very often. It will be garbage collected when the dynamic threshold is met, AND when you run low on memory.
  • .NET will consume the memory available. There is no need for a garbage collection if you are not running low on memory. It can be nerve wrecking to see the memory being low on your server, but this is often not a problem, it is by design.
  • The high fragmentation of your large object heap is a sign that objects are being freed, leaving holes in the heap (so at least your application is not leaking memory). These holes will be filled with new objects if they fit into the hole.
  • Memory is becoming increasingly cheaper, making the business case of memory optimization smaller and smaller. Memory leaks is still a problem, but memory usage is not.

HEY, IT’S A PROBLEM FOR ME. WHAT CAN I DO ABOUT IT?

But let’s say that memory IS a problem – what can you do about it?

TUNE YOUR CACHES:

If memory is a problem, you should not let the cache sizes run free. Set Caching.DisableCacheSizeLimits to false and tune your Sitecore caches:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

I have provided a list of links to cache tuning guides at the bottom of this post.

REUSE DUPLICATED FIELD VALUES:

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. But only if you code it that way.

Sitecore have implemented an interning pattern in the \App_Config\Sitecore\CMS.Core\Sitecore.Interning.config file where fields that often have duplicated values can be stored:

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

You can add your own duplicated fields to the list. Nikolay Mitikov created this SQL to find duplicated values:

WITH DuplicatedFieldValues AS  (
SELECT 
	v.FieldId,	
	FieldDefinitionRow.[Name],
	CONVERT(NVARCHAR(250),v.[Value]) AS [Field Value],
	COUNT(1) AS [Hits]
FROM 
	[VersionedFields] v 
JOIN 
	[Items] FieldDefinitionRow ON FieldDefinitionRow.ID = v.fieldID
WHERE 
	v.FieldId NOT IN 
	(
		/* Fields already interned OOB by Sitecore.Interning.config */			
		'BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A' /* updated by */,
		'5DD74568-4D4B-44C1-B513-0AF5F4CDA34F' /* created by */,
		'52807595-0F8F-4B20-8D2A-CB71D28C6103' /* owner */,
		'3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98' /* workflow state */	
	)
GROUP BY 
	FieldId, [Name], CONVERT(NVARCHAR(250),[Value])
HAVING 
	COUNT(*) > 500 /* How many same field values must be met to be shown */)

SELECT * FROM DuplicatedFieldValues
ORDER BY [Hits] DESC

This can release a lot of memory, especially if you have large text fields.

USE Sitecore.Diagnostics.MemoryMonitorHook TO CLEAR CACHE AND FORCE GARBAGE COLLECTION

The Sitecore.Diagnostics.MemoryMonitorHook monitors your memory usage and can be set to force a garbage collection and a cache clear when the threshold is met:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>
  • Threshold: How much memory should be used before doing anything. Set this to 60-80% of your total memory.
  • Check interval: How often should the hook run.
  • ClearCaches: When threshold is met, should the cache be cleared?
  • GarbageCollect: When threshold is met, should Sitecore do a forced garbage collect?
  • AdjustLoadFactor: When threshold is met, should Sitecore attempt to adjust the garbage collection load factor?

MORE TO READ:

 

Posted in Sitecore 6, Sitecore 7, Sitecore 8, Sitecore 9 | Tagged , , , , , | 1 Comment

Remove duplicates from XML feed

Apparently XML isn’t dead yet, and today I received a Google Product Feed in the RSS 2.0 XML format. The feed was full of duplicates and my job is to remove them:

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
    <channel>
        <item>
            <g:id>100</g:id>
            <title>Product 100</title>
            ...
            ...
        </item>
        <item>
            <g:id>100</g:id>
            <title>Product 100</title>
            ...
            ...
        </item>
        <item>
            <g:id>200</g:id>
            <title>Product 200</title>
            ...
            ...
        </item>
        <item>
            <g:id>300</g:id>
            <title>Product 300</title>
            ...
            ...
        </item>
    </channel>
</rss>

As you can see, “Product 100” appears twice.

THE SOLUTION:

A little LINQ can get you far:

using System.Xml;
using System.Xml.Linq;
using System.Linq;

var document = XDocument.Parse(theXMLString);

XNamespace g = "http://base.google.com/ns/1.0";
document.Descendants().Where(node => node.Name == "item");
    .GroupBy(node => node.Element(g+"id").Value)
    .SelectMany(node => node.Skip(1))
    .Remove();

HOW IT WORKS:

  • document.Descendants().Where(node => node.Name == “item”): Get all elements called “item
  • GroupBy(node => node.Element(g+”id”).Value): Group them by the “g:id” element.
  • SelectMany(node => node.Skip(1)): Select every one of them apart from the first one
  • Remove(): Delete all that were selected

MORE TO READ:

Posted in .net, .NET Core, c#, General .NET | Tagged , , , , | Leave a comment

HttpClient retry mechanism with .NET Core, Polly and IHttpClientFactory

A lot of HttpClient errors are temporary and is caused by server overload, temporary nerwork timeouts and generic gliches in the Matrix. These scenarios can be dealt with using a retry pattern. In .NET Core, the most common retry library is the Polly library:

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.
http://www.thepollyproject.org/

Polly makes it relatively easy to implement a retry pattern, as long as you use the IHttpClient and IHttpClientFactory.

But enugh talk, lets code.

STEP 1: THE NUGET PACKAGES

You need (at least) the following NuGet Packages:

  • Polly
  • Microsoft.Extensions.Http.Polly

STEP 2: CONFIGURE SERVICES IN STARTUP.CS

In the services configuration, you need to add a IHttpClientFactory and attach a PolicyHandler to the factory:

//ConfigureServices()  - Startup.cs
services.AddHttpClient("HttpClient").AddPolicyHandler(GetRetryPolicy());

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
  return HttpPolicyExtensions
    // Handle HttpRequestExceptions, 408 and 5xx status codes
    .HandleTransientHttpError()
    // Handle 404 not found
	.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
    // Handle 401 Unauthorized
	.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.Unauthorized)
    // What to do if any of the above erros occur:
	// Retry 3 times, each time wait 1,2 and 4 seconds before retrying.
	.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
}

STEP 3: USE THE IHttpClientFactory IN THE CALLING CLASS

The IHttpClientFactory can be injected using constructor injection. The cool part of the Polly implementation is that your HttpClient code does not contain any special retry-code, just the usual Get or Post calls:

namespace MyCode
{
  public class MyClass
  {
    private readonly IHttpClientFactory _clientFactory;

    public MyClass(IHttpClientFactory clientFactory)
    {
      _clientFactory = clientFactory;
    }

    public async Task<int> PostMessage(string postData)
    {
      var httpClient = _clientFactory.CreateClient("HttpClient");

      using (var content = new StringContent(postData, Encoding.UTF8, "application/json"))
      {
        var result = await httpClient.PostAsync($"{url}", content);
        // The call was a success
        if (result.StatusCode == HttpStatusCode.Accepted)
        {
          return result.StatusCode;
        }
        // The call was not a success, do something
        else
        {
          // Do something
          return result.StatusCode;
        }
      }
    }
  }
}

The httpClient.PostAsync() will retry the post call automatically if any of the conditions described in the GetRetryPolicy() occurs. It will only return after the call is either successful or the retry count is met.

MORE TO READ:

Posted in .NET Core, General .NET | Tagged , , | Leave a comment

.NET Core Worker Services with Application Insights and Serilog

The .NET Core Worker service is yet another tool in the .NET toolbox. They are perfect for background processing like reading from a queue or making health checks. They are cross-platform (of course) and they can run on docker containers, and even run on a locally hosted machine as a Windows service.

It took me quite some time to implement Application Insights and file logging (Serilog) in my worker service, so I thought I might write it down, in case I need it another time.

THE NUGET PACKAGES

First caveat was when I realized that you need a special Microsoft.ApplicationInsights.WorkerService NuGet package before Application Insights will work. Also, if you wish to run the worker service as a Windows service you need the Microsoft.Extensions.Hosting.WindowsServices package.

The packages I needed was:

Microsoft.ApplicationInsights.AspNetCore
Microsoft.ApplicationInsights.WorkerService
Microsoft.Extensions.Hosting
Microsoft.Extensions.Hosting.WindowsServices
Serilog
Serilog.AspNetCore
Serilog.Extensions.Hosting
Serilog.Extensions.Logging
Serilog.Settings.Configuration
Serilog.Sinks.Console
System.Configuration.ConfigurationManager

THE APPSETTINGS.JSON CONFIGURATION

The configuration is pretty straight forward. Only caveat is that Serilog is not configured under “Logging” but under it’s own namespace:

{
  "ApplicationInsights": {
    "InstrumentationKey": "the-appinsights-guid"
  },
  "Serilog": {
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "log_.txt",
          "rollingInterval": "Day"
        }
      }
    ]
  },
  "Logging": {
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information",
        "Microsoft": "Warning"
      }
    },
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Information",
      "Microsoft.Hosting.Lifetime": "Warning"
    }
  }
}

THE PROGRAM.CS

The examples online was too complex for me, so I simplified it and came up with this:

namespace MyWorkerService
{
  public class Program
  {
    public static void Main(string[] args)
    {
      // This will run the worker service
      CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args)
    {
      var host = Host.CreateDefaultBuilder(args);
      // Add this line to be able to run as a windows service
      // https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-3.1&tabs=visual-studio
      host.UseWindowsService();
      // This will add the configuration to the application.
      // It will allow you to inject the configuration as 
      // IConfiguration configuration in your constructors 
      host.ConfigureAppConfiguration(
            (hostContext, config) =>
            {
              config.SetBasePath(Directory.GetCurrentDirectory());
              config.AddJsonFile("appsettings.json", false, true);
              config.AddCommandLine(args);
            }
      );
      // This will configure logging. It reads the log settings from 
      // the appsettings.json configuration file, and adds serilog,
      // allowing the application to write logs to file
      host.ConfigureLogging(
            loggingBuilder =>
            {
              var configuration = new ConfigurationBuilder()
                 .AddJsonFile("appsettings.json")
                 .Build();
              var logger = new LoggerConfiguration()
                  .ReadFrom.Configuration(configuration)
                  .CreateLogger();
              loggingBuilder.AddSerilog(logger, dispose: true);
            }
      );
      // The AddHostedServer adds the worker service to the application.
      // The AddApplicationInsightsTelemetryWorkerService is very important. Without this,
      // the Application Insights logging will not work.
      host.ConfigureServices((hostContext, services) =>
      {
        services.AddHostedService<Worker>();
        services.AddApplicationInsightsTelemetryWorkerService();
      });

      return host;
    }
  }
}

The worker itself is the “Worker” class and it looks like this:

using System.Threading;
using System.Threading.Tasks;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Configuration;

namespace MyWorkerService
{
  public class Worker : BackgroundService
  {
    private readonly ILogger<Worker> _logger;
    private readonly TelemetryClient _telemetryClient;

    public Worker(ILogger<Worker> logger,
      TelemetryClient telemetryClient,
      IConfiguration configuration)
    {
      _logger = logger;
      _telemetryClient = telemetryClient;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
      while (!stoppingToken.IsCancellationRequested)
      {
        using (_telemetryClient.StartOperation<RequestTelemetry>("Execute Async"))
        {
          _logger.LogInformation("ExecuteAsync Started");
          DoWork();
        }
        await Task.Delay(1000, stoppingToken);
      }
    }
  }
}

The DoWork() is where you would do actual work.

MORE TO READ:

Posted in .net, .NET Core, c# | Tagged , , , | 4 Comments

System.DllNotFoundException: Unable to load DLL ‘sni.dll’ or one of its dependencies: The specified module could n ot be found. (0x8007007E)

This message happens when deploying my .NET Core 3.1 application to production (when compiling the code to an .exe file), but not when running the application locally.

It turns out, that Dapper is missing a reference to System.Data.SqlClient. Adding the System.Data.SqlClient NuGet package to the project solves the issue:

System.Data.SqlClient

System.Data.SqlClient

The full error message is:

Unhandled exception. Unhandled exception. System.TypeInitializationException: The type initializer for ‘System.Data.SqlClient.TdsParser’ threw an exception.
—> System.TypeInitializationException: The type initializer for ‘System.Data.SqlClient.SNILoadHandle’ threw an exception.
—> System.DllNotFoundException: Unable to load DLL ‘sni.dll’ or one of its dependencies: The specified module could not be found. (0x8007007E)
at System.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize(IntPtr pmo)
at System.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize()
at System.Data.SqlClient.SNILoadHandle..ctor()
at System.Data.SqlClient.SNILoadHandle..cctor()
— End of inner exception stack trace —
at System.Data.SqlClient.TdsParser..cctor()
— End of inner exception stack trace —
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObject
sTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& co
nnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry
, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`
1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Dapper.SqlMapper.QueryImpl[T](IDbConnection cnn, CommandDefinition command, Type effectiveType)+MoveNext() in C:\projects\dapper\Dapper\SqlMapper.cs:line 1079
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in C:\projects\dapper\Dapper\SqlMapper.cs:line 721

MORE TO READ:

Posted in .net, .NET Core, c# | Tagged , , | 1 Comment

Sync Sitecore content to external database

The legal department of the client requested that we sync parts of the Sitecore tree to an external, system-versioned temporal table, so that we have a full audit trail of changes to these items including before-and-after values.

System-Versioned Table

Example of a System-Versioned Table. When looking into the History you can see a full audit trail of the Sitecore items

Fear not, this is what Sitecore does best. All we need to do is to create a SQL server database with temporal tables, create upsert (a combined insert and update statement) and delete Stored Procedures, and then hook into the Sitecore item:created, item:renamed, item:saved, item:moved and item:deleted events.

STEP 1: THE BASIC DATABASE PATTERN

First we need to create a table for the data. Maybe you like to create your tables in a different matter, and that is OK, but this is the minimum required fields needed to perform a data audit:

CREATE TABLE [dbo].[MyItem]
(
    [SitecoreItemID] UNIQUEIDENTIFIER NOT NULL PRIMARY KEY,
	...
	... here you define the fields from the Sitecore template
	...
    [Created] DATETIME NOT NULL DEFAULT GetDate(), 
    [Updated] DATETIME NULL, 
    [Deleted] DATETIME NULL, 
    [IsDeleted] BIT NOT NULL, 
    [SitecoreUserName] NVARCHAR(255) NOT NULL,
    [SysStartTime] DATETIME2 GENERATED ALWAYS AS ROW START NOT NULL,
    [SysEndTime] DATETIME2 GENERATED ALWAYS AS ROW END NOT NULL,
    PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)
    )
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.MyItem));
  • [SitecoreItemID]
    Is the key value is the Sitecore item id. Some like to have an auto-incrementing integer as key, and that’s also fine, as long as you at least have the Sitecore item ID in the database.
  • [Created], [Updated], [Deleted]
    Timestamps are used to determine when database operations are carried out
  • [IsDeleted]
    Instead of deleting items, we mark them as deleted. The [Deleted] timestamp will tell us when the item was deleted.
  • [SitecoreUsername]
    Is the name of the Sitecore editor creating/updating/deleting the item
  • [SysStartTime],[SysEndTime]
    Is used by SQL to handle the system-versioning.
  • Remember to add all the fields from your Sitecore template that needs to be synchronized.

Also note that in my example, I do not store the item hierarchy, item languages or item versions.

STEP 2: UPSERT AND DELETE STORED PROCEDURES:

The basic UPSERT Stored procedure is as follows:

  • Select from database with [SitecoreItemID]
  • If found: Update
    Update with values from the Sitecore template
    Set [Updated] to current datetime
    Set [SitecoreUserName] with the Sitecore username
  • If not found: Insert
    Insert values from the Sitecore template
    Set [SitecoreUserName] with the Sitecore username

The basic DELETE Stored procedure is as follows:

  • Set [IsDeleted] to TRUE
    Set [SitecoreUserName] with the Sitecore username
    Set [Deleted] with the current datetime.

STEP 3: HOOK INTO SITECORE 

Now for the fun part, the Sitecore code.

We need to hook into the following Sitecore events:

  • item:created: Call the UPSERT stored procedure
  • item:renamed:Call the UPSERT stored procedure
  • item:saved: Call the UPSERT stored procedure
  • item:moved: Call the UPSERT stored procedure
  • item:deleted: Call the DELETE stored procedure

My code will only sync one language and always the latest version. If you need to sync more languages or versions, remember to add those as key values to your database.

First the code. This is a pseudo-code example, outlining what you need to implement.

using System;
using Sitecore.Data.Items;
using Sitecore.Events;

namespace MyCode
{
  public class SyncItems
  {
    public void OnItemChanged(object sender, EventArgs args)
    {
      Item changedItem = Event.ExtractParameter(args, 0) as Item;
      if (changedItem == null)
        return;
      // We only synchronize one language and only the latest version:
      if (changedItem.Language != Language.Parse("en"))
        return false;
      if (!changedItem.Versions.IsLatestVersion())
        return false;
        
      // If changed item is of the template we wish to synchronize:
      // Call the upsert stored procedure
    }

    public void OnItemMoved(object sender, EventArgs args)
    {
      Item movedItem = Event.ExtractParameter(args, 0) as Item;
      if (movedItem == null)
        return;
      // We only synchronize one language and only the latest version:
      if (movedItem.Language != Language.Parse("en"))
        return false;
      if (!movedItem.Versions.IsLatestVersion())
        return false;

      // If moved item is of the template we wish to synchronize:
      // Call the upsert stored procedure
    }
    
    public void OnItemDeleted(object sender, EventArgs args)
    {
      Item deletedItem = Event.ExtractParameter(args, 0) as Item;
      if (deletedItem == null)
        return;
      // We only synchronize one language and only the latest version:
      if (deletedItem.Language != Language.Parse("en"))
        return false;
      if (!deletedItem.Versions.IsLatestVersion())
        return false;

      // If deleted item is of the template we wish to synchronize:
      // Call the delete stored procedure
    }
}   

Then the Sitecore configuration:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:env="http://www.sitecore.net/xmlconfig/env/">
  <sitecore>
    <events>
      <event name="item:created">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemChanged"/>
      </event>
      <event name="item:renamed">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemChanged"/>
      </event>
      <event name="item:saved">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemChanged"/>
      </event>
      <event name="item:moved">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemMoved"/>
      </event>
      <event name="item:deleted">
        <handler type="MyCode.SyncItems, MyCode" method="OnItemDeleted"/>
      </event>
    <events>
  </sitecore>
</configuration>

THINGS TO CONSIDER:

Should you use a synchronous or asynchronous update method?

If you call the stored procedures synchronously, all code will wait for and answer from the  SQL server. If you a asynchronous approach, you don’t catch exceptions from the database. It is up to you how critical errors are. There are coding patterns that will give you a more foul proof approach such as database retry mechanisms, or implementing a queue in between Sitecore and the database. You decide what is best.

What happens if you move an item with children?

If you are synchronizing items with children, and you stored the hierarchy of items in the database, you will need to make a cascading update on those children. Sitecore will not throw individual events for the children, only the root item that is moved.

What happens if users restore items from the recycle bin?

This is a situation that is currently undetermined, as there is no event attached to the recycle bin restore.

Are there any benefits? Why don’t just use Sitecore versioning and workflows?

Workflows are designed to enforce business rules upon editors, and implies a very stringent way of working. Workflows discourages casual editing and is often abandoned for that very reason. You should decide if a simple workflow would work for you.

MORE TO READ:

Posted in .net, c#, Sitecore 5, Sitecore 6, Sitecore 7, Sitecore 8, Sitecore 9 | Tagged , , , , , , | Leave a comment

C# Get expiry timestamp from JWT token

JWT tokens (or Json Web Tokens) are an open-standard the defines a way to transmit information between 2 parties in a secure manner. Identity Server 4 uses JWT as a security token.

These tokens have an expiry timestamp, and if you handle the tokens yourself, you need to read the token expiry and refresh the token if the token is expired.

Microsoft have made a brilliant library, System.IdentityModel.Tokens.Jwt to handle JWT tokens, but the package does also have a lot of dependencies that were incompatible with my application, so I chose to use JWT.Net instead, as this package does not have any dependencies at all.

THE ANATOMY OF A JWT TOKEN:

Json Web Token Anatomy

Json Web Token Anatomy

A JWT token consists of a header, a payload and a signature. It is in the payload that you find the expiry timestamp in the “exp” field. The timestamp is the stupid UNIX timestamp format, but fear not, .NET knows how to convert the timestamp to a real DateTime.

STEP 1: CREATE A PAYLOAD MODEL CLASS

JWT.Net is not as powerful as System.IdentityModel.Tokens.Jwt, so you need to create a model class of the payload section. The class, however, is very simple:

namespace MyCode
{
  public class JwtToken
  {
    public long exp { get; set; }
  }
}

STEP2: USE JWT.Net TO GET THE EXPIRY FROM THE TOKEN PAYLOAD

Final step is to take the JWT Token string and decode it to the JwtToken class, then convert the UNIX timestamp to a local time:

using System;
using JWT;
using JWT.Algorithms;
using JWT.Serializers;

namespace MyCode
{
  public class JWTService
  {
    private IJsonSerializer _serializer = new JsonNetSerializer();
    private IDateTimeProvider _provider = new UtcDateTimeProvider();
    private IBase64UrlEncoder _urlEncoder = new JwtBase64UrlEncoder();
    private IJwtAlgorithm _algorithm = new HMACSHA256Algorithm();

    public DateTime GetExpiryTimestamp(string accessToken)
    {
      try
      {
        IJwtValidator _validator = new JwtValidator(_serializer, _provider);
        IJwtDecoder decoder = new JwtDecoder(_serializer, _validator, _urlEncoder, _algorithm);
        var token = decoder.DecodeToObject<JwtToken>(accessToken);
        DateTimeOffset dateTimeOffset = DateTimeOffset.FromUnixTimeSeconds(token.exp);
        return dateTimeOffset.LocalDateTime;
      }
      catch (TokenExpiredException)
      {
        return DateTime.MinValue;
      }
      catch (SignatureVerificationException)
      {
        return DateTime.MinValue;
      }
      catch (Exception ex)
      {
        // ... remember to handle the generic exception ...
        return DateTime.MinValue;
      }
    }
  }
}

That’s it. You are now a security expert. Happy coding.

FUNNY FINAL NOTE:

The term “JWT Token” is a redundant acronym syndrome, or RAS-syndrome. It is the use of the last word of the acronym in conjunction with the abbreviated form. It’s like saying “PIN number” or “PDF format”. In reality, when saying “JWT Token”, you are really saying “json web token token” :).

MORE TO READ:

Posted in .net, .NET Core, c#, General .NET, Microsoft Azure | Tagged , , , , | Leave a comment

Application Insights not working in .NET Core Console Application – Remember to flush the TelemetryClient

My .NET Core console application did not write to my Application Insights instance. Well, it did, but only sometimes.

Writing to Application Insights is done asynchronously, usually every 30 seconds or 500 items. This means that you need to give your application some time to flush the cache before closing down. This can be done by configuring a ProcessExit method. This method is called when your Main() function exists.

This is a template where I initialize my TelemetryClient, and then flushes and waits 5 seconds for the cache to flush before the application is exited:

using System;
using System.Threading;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.DependencyInjection;

namespace MyCode
{
  internal class Program
  {
    public static void Main(string[] args)
    {
      // Initialize Service Collection
      var serviceCollection = new ServiceCollection();
      ConfigureServices(serviceCollection);

      // Initialize Service Provider
      var serviceProvider = serviceCollection.BuildServiceProvider();

      // Handling finalizing when process is ended
      AppDomain.CurrentDomain.ProcessExit += (s, e) => FinalizeApplication(serviceProvider);

      // Run Application
      // ... do run ...
    }

    private static void FinalizeApplication(ServiceProvider serviceProvider)
    {
      // Give TelemetryClient 5 seconds to flush it's content to Application Insights
      serviceProvider.GetService<TelemetryClient>().Flush();
      Thread.Sleep(5000);
    }

    private static void ConfigureServices(IServiceCollection serviceCollection)
    {
      // Add Application Insights
      var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
      telemetryConfiguration.InstrumentationKey = "add-your-own-key";
      var telemetryClient = new TelemetryClient(telemetryConfiguration);
      serviceCollection.AddSingleton(telemetryClient);
    }

  }
}

HOW IT WORKS:

My .NET Core 3.1 console application is started by the Main() function. It starts by initializing the service collection. The only thing I do is adding a TelemetryClient to the collection, but this is where you would all all of your services.

The AppDomain.CurrentDomain.ProcessExit is configured to call the “FinalizeApplication” method, and this method will now be called whenever the Main() function exits.

The FinalizeApplication will then flush the TelemetryClient, and waits 5 seconds for the flush to do it’s magic.

Without the 5 second wait, I risk that the Flush is not done flushing before the thread is discontinued, hence the wait. For some reason, 5 seconds seems to be the magic threshold for a flush to do it’s thing.

MORE TO READ:

Posted in .NET Core, Microsoft Azure | Tagged , , | Leave a comment