Method not found: ‘Void Sitecore.ContentSearch.Diagnostics.AbstractLog.SingleWarn(System.String, System.Exception)’.

I struggled with this error in my development environment:

Method not found: ‘Void Sitecore.ContentSearch.Diagnostics.AbstractLog.SingleWarn(System.String, System.Exception)’.

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Sitecore.ContentSearch.SolrProvider.LinqToSolrIndex`1.Execute[TResult](SolrCompositeQuery compositeQuery)
at Sitecore.ContentSearch.Linq.QueryableExtensions.GetResults[TSource](IQueryable`1 source)

After an hour of so of debugging and not understanding that the error is only in my development environment, and not in production, I did a full rebuild, and lo and behold – the error dissapeared?

Well, it turns out that I had the wrong version of the Sitecore.ContentSearch.dll in my development environment. Some NuGet reference had overwritten the correct hotfix dll 3.1.1-r00161 Hotfix 206976-1 with an older version 3.1.1-r00161, and this version does apparently not contain the AbstractLog.SingleWarn(System.String, System.Exception) method.

Morale: If the error message state that your method is missing, it could be true. Check your dependencies before panicking.

Sitecore.ContentSearch.dll

Sitecore.ContentSearch.dll

MORE TO READ:

 

Posted in .net, c#, Sitecore 8, Sitecore 9 | Tagged , , | Leave a comment

C# Set local folder for .net Core Windows Services

When developing .NET Core Worker Services, you can allow the service to run as Windows Service:

public static IHostBuilder CreateHostBuilder(string[] args)
{
  var host = Host.CreateDefaultBuilder(args);
  host.UseWindowsService();
  ...
  ...

The side effect is that the root folder changes from the local folder to the System32 folder, which means that any log files that you would expect to find in your local folder suddenly ends up in another folder.

The fix is easy, simple add the following to the main function of your application:

public static void Main(string[] args)
{
  Directory.SetCurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
  CreateHostBuilder(args).Build().Run();
}

SetCurrentDirectory will then rebase the local folder to the base directory of your application, and your log files will be written to the local folder.

MORE TO READ:

 

Posted in .NET Core, c# | Tagged , , | Leave a comment

Manipulating XML Google Merchant Data using C# and LINQ

Receiving a Google Merchant Data feed (also known as a Google Product Feed) can be fairly easily manipulated on import time using a little C# and LINQ.

The feed is basically a XML RSS 2.0 feed with some added properties using the namespace xmlns:g=”http://base.google.com/ns/1.0.

These feeds often comes from older systems and data is created by busy merchants, so data can be relatively dirty, and a cleanup is required before you add them to your product database.

The feed could look like this:

<?xml version="1.0" encoding="utf-8" ?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
    <channel>
        <title>Google product feed</title>
        <link href="https://pentia.dk" rel="alternate" type="text/html"/>
        <description>Google product feed</description>
        <item>
            <g:id><![CDATA[1123432]]></g:id>
            <title><![CDATA[Some product]]></g:title>
            <link><![CDATA[https://pentia.dk]]></g:link>
            <g:description><![CDATA[description]]></g:description>
            <g:gtin><![CDATA[5712750043243446]]></g:gtin>
            <g:mpn><![CDATA[34432-00]]></g:mpn>
            <g:image_link><![CDATA[https://pentia.dk/someimage.jpg]]></g:image_link>
            <g:product_type><![CDATA[Home &gt; Dresses &gt; Maxi Dresses]]></g:product_type>
            <g:condition><![CDATA[new]]></g:condition>
            <g:availability><![CDATA[in stock]]></g:availability>
            <g:price><![CDATA[15.00 USD]]></g:price>
            <g:sale_price><![CDATA[10.00 USD]]></g:sale_price>
        </item>
        ...
        ...
    </channel>
</rss>

See the full specification in the Google Merchant Center help.

Sometimes the feed would contain content that you does not need, and a little XML manipulation is required.

But first thing first:

STEP 1: GET THE XML FEED AND CONVERT IT INTO AN XML DOCUMENT

using System;
using System.Net;
using System.Net.Http;
using System.Xml;
using System.Xml.Linq;
using System.Linq;
using System.Dynamic;

private static HttpClient _httpClient = new HttpClient();

public static async Task<string> GetFeed(string url)
{
  using (var result = await _httpClient.GetAsync($"{url}"))
  {
    string content = await result.Content.ReadAsStringAsync();
    return content;
  }
}

public static void Run()
{
  // Get the RSS 2.0 XML data
  string feedData = GetData("https://url/thefeed.xml").Result;

  // Convert the data into an XDocument
  var document = XDocument.Parse(feedData);
  // Speficy the Google namespace
  XNamespace g = "http://base.google.com/ns/1.0";
  // Get a list of all "item" nodes
  var items = document.Descendants().Where(node =&amp;gt; node.Name == "item");
    
  // Now we are ready to manipulate
  // ...
  // ...
}

NOW TO THE MANIPULATIONS:

EXAMPLE 1: Remove duplicates – all products with the same ID is removed:

items.GroupBy(node => node.Element(g+"id").Value)
  .SelectMany(node => node.Skip(1))
  .Remove();

EXAMPLE 2: Remove all products out of stock:

items = document.Descendants()
  .Where(node => node.Name == "item" 
         && node.Descendants()
         .Any(desc => desc.Name == g + "availability" 
              && desc.Value == "out of stock"));
items.Remove();

EXAMPLE 3: Remove adverts not on sale (all adverts that do not have a g:sale_price node)

items = document.Descendants()
  .Where(node => node.Name == "item" 
         && node.Descendants()
         .Any(desc => desc.Name == g + "sale_price" 
         && desc.Value.Trim() == string.Empty));
items.Remove();

EXAMPLE 4: ADD TRACKING PARAMETERS TO URL’S (adding query string parameters to the URL)

var items = document.Descendants().Where(node => node.Name == "item");
foreach (var item in items)
{
  string url = item.Element("link").Value;
  if (url.Contains("?"))
    item.Element("link").ReplaceNodes(new XCData(url + "&" + "utm_source=s&utm_medium=m&utm_campaign=c"));
  else  
    item.Element("link").ReplaceNodes(new XCData(url + "?" + "utm_source=s&utm_medium=m&utm_campaign=c"));
}

EXAMPLE 5: CHANGE THE TITLE (for example, if the feed contains used products, you might want to add the word “used” to the title

var items = document.Descendants().Where(node => node.Name == "item");
foreach (var item in items)
{
  var title = "USED " + item.Element("title").Value;
  item.Element("title").ReplaceNodes(title);
}

…AND THE EXOTIC EXAMPLE: COMBINE ALL PRODUCTS IF THEY BELONG TO A PRODUCT_TYPE THAT CONTAIN LESS THAN 2 PRODUCTS

foreach(var group in items.GroupBy(node => node.Element(g+"product_type").Value))
{
  if (group.Count() <= 2)
  {
    foreach (var advert in group)
    {
      advert.Element(g+"product_type").ReplaceNodes(new XCData("Other"));
    }
  }
}

Finally you can grab the manipulated document and do what you need to do:

// Grab converted content
string convertedFeedData = document.ToString();

I hope this gives some examples on how to do much with less code.

MORE TO READ:

Posted in .net, .NET Core, c#, General .NET | Tagged , , , , , | Leave a comment

C# Azure TelemetryClient will leak memory if not implemented as a singleton

I noticed that my classic .net web application would leak memory after I implemented metrics for some background tasks.

Memory usage of web application

Memory usage of web application

Further investigation showed that my MetricAggregationManager would not release its memory.

Object was not garbage collected

Object was not garbage collected

Since one of the major changes was the implementation of a TelemetryClient, and since the memory not being released was from the Microsoft.ApplicationInsights.Metrics namespace, I concluded that the problem lies within the creation of the TelemetryClient:

using System;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Metrics;

namespace MyCode
{
  public class BaseProcessor
  {
    private readonly TelemetryClient _telemetryClient;
    
    private BaseProcessor()
    {
      string instrumentationKey = "somekey"
      var telemetryConfiguration = new TelemetryConfiguration { InstrumentationKey = instrumentationKey };
      // This is a no-go. I should not create a new instance for every BaseProcessor
      _telemetryClient = new TelemetryClient(telemetryConfiguration);
    }
  }
}

The code above will create a new TelemetryClient for each creation of my base class. The TelemetryClient will collect metrics and store those in memory until either a set time or number of metrics are met, and then dump the metrics to Application Insights.

So when the BaseClass is disposed, TelemetryClient is not, leaving memory to hang, and thus a memory leak is in effect.

HOW TO SOLVE IT?

The solution is simple. All you need to do is to create a singleton pattern for your TelemetryClient. Having only one instance will allow the client to collect and send metrics in peace. Your code will be much faster (it takes a millisecond or so to create a TelemetryClient) and you will not have any memory leaks.

USE DEPENDENCY INJECTION:

In .NET Core you can add the TelemetryClient to the service collection:

private static void ConfigureServices(IServiceCollection services)
{
  // Add Application Insights
  var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
  telemetryConfiguration.InstrumentationKey = "somekey"
  var telemetryClient = new TelemetryClient(telemetryConfiguration);
  services.AddSingleton(telemetryClient);
}

And then reference it using constructor injection:

using System;
using System.Runtime.Serialization;
using Microsoft.ApplicationInsights;
using Microsoft.AspNetCore.Mvc;

namespace MyCode
{
  [ApiController]
  [Route("/api/[controller]")]
  [Produces("application/json")]
  public class MyController : ControllerBase
  {
    private readonly TelemetryClient _telemetryClient;

    public MyController(TelemetryClient telemetryClient)
    {
      _telemetryClient = telemetryClient;
    }
  }
}

USE A STATIC VARIABLE:

If you do not have access to a DI framework, you could also just create a static variable:

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using System.Collections.Generic;

namespace MyCode
{
  public static class TelemetryFactory
  {
    private static TelemetryClient _telemetryClient;

    public static TelemetryClient GetTelemetryClient()
    {
      if (_telemetryClients == null)
      {
        string instrumentationKey = "somekey";
        var telemetryConfiguration = new TelemetryConfiguration { InstrumentationKey = instrumentationKey };
        _telemetryClient = new TelemetryClient(telemetryConfiguration);
      }

      return _telemetryClient;
    }
  }
}

And then reference the static variable instead:

using System;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Metrics;

namespace MyCode
{
  public class BaseProcessor
  {
    private readonly TelemetryClient _telemetryClient;
    
    private BaseProcessor()
    {
      _telemetryClient = TelemetryFactory.GetTelemetryClient();
    }
  }
}

MORE TO READ:

 

Posted in .net, .NET Core, c#, General .NET, Microsoft Azure | Tagged , , , | Leave a comment

Sitecore Memory Issues – Every memory optimization trick in the book

My recent post, Sitecore high memory usage – not always a problem, claims that high memory usage is not the same as having a memory problem in Sitecore. But the reason you are reading this blog is because the memory have become a problem – right? So let’s see what you can do about it.

I will not go into the real issue, which probably is that your own application code have a memory leak. This not Sitecore’s fault, although we would like it to be. Instead I will show you the Sitecore settings that will limit the memory consumption of the Sitecore platform itself.

TRICK 1: YOUR EVENT QUEUE IS RUNNING HIGH

This is the #1 issue: Too many events in the event queue. First you need to clean the event queue. Read how to clean the event queue here.

Next you need to manage the event queue size. Use the CleanupEventQueue agent and tune the cleanup so you have no more than 7000 items in the queue at any time:

<agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00">
  <IntervalToKeep>04:00:00</IntervalToKeep>
  <DaysToKeep>1</DaysToKeep>
</agent>

TRICK 2: DON’T LET THE SITECORE CACHE RUN WILD

Use the Caching.DisableCacheSizeLimits to enforce the cache limits you have painfully configured through the application:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

Also use one of the many Sitecore Cache Tuning Guides online. Even old blog posts will still be useful, as the cache is part of Sitecore basics and have not changed much.

TRICK 3: YOU HAVE TOO MANY UNUSED VERSIONS OF YOUR ITEMS

Items with many versions will eat your memory every time you open the item in the Sitecore Shell. If possible, delete the old versions. Click here to see how to delete old Sitecore item versions.

TRICK 4: DISABLE EXM

Are you using EXM? No? Then disable EXM:

<setting name="EXM.Enabled" value="false" />

TRICK 6: DO NOT CACHE ITEMS WHEN PUBLISHING

You can disable the Publishing database cache. Then the items being published will not be added to the Sitecore database cache.

<setting name="Publishing.DisableDatabaseCaches" value="true"/>

TRICK 7: LOWER THE BATCH SIZE WHEN INDEXING

The Sitecore SOLR indexing can eat your memory. Lowering the batch size have a positive effect on the amount of memory being used when an item is indexed:

<setting name="ContentSearch.IndexUpdate.BatchSize" value="150" />

TRICK 8: DO NOT CACHE WHEN INDEXING

You can disable the index caching. Then the items being retrieved during an index is not cached:

<setting name="ContentSearch.Indexing.DisableDatabaseCaches" value="true" />

TRICK 9: UTILIZE INTERNING

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. Sitecore have implemented interning on fields, but you have to define which fields should be interned. Legend says that there is a performance penalty when interning, but I have not been able to measure any performance decrease.

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

TRICK 10: FORCE GARBAGE COLLECTION WHEN MEMORY IS LOW

The old peeing in your pants trick is also implemented by Sitecore. You need to enable counters first:

<setting name="Counters.Enabled" value="true"/>

Then you can allow the MemoryMonitorHook to force a garbage collection when memory is running low:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>

I guess that’s it. If you know of a memory trick for Sitecore, please let me know.

MORE TO READ:

 

Posted in Sitecore 6, Sitecore 7, Sitecore 8, Sitecore 9 | Tagged , , , , , | Leave a comment

Sitecore high memory usage – not always a problem

Sitecore is known to be a real memory hog. Especially the CM server is known to love memory. One of the memory issues that Sitecore faces is the high usage of the Large Object Heap.

Heap Fragmentation

Heap Fragmentation

WHY IS THIS A PROBLEM?

The .net garbage collector will start collecting memory from Generation 0. Objects that cannot be collected is upgraded to Generation 1, which is then collected later. Objects can be further upgraded to Generation 2, assuming that these objects are long-lived and does not need to be collected as often.

The Large Object Heap is where objects larger than 85.000 bytes live. This is mostly strings and arrays, and these objects are rarely garbage collected, and the heap is never compacted (not entirely true as newer .net versions can compact this heap) with can lead to high memory usage.

WHY THIS IS PROBABLY NOT A PROBLEM ANYWAY?

Sitecore is well-known to be a large object heap eater, because Sitecore stores fields and field values as arrays of strings, making every Item a large object. And this is a problem – right?

It definitely can be a problem – but not necessarily:

  • The Large Object heap IS garbage collected, but not very often. It will be garbage collected when the dynamic threshold is met, AND when you run low on memory.
  • .NET will consume the memory available. There is no need for a garbage collection if you are not running low on memory. It can be nerve wrecking to see the memory being low on your server, but this is often not a problem, it is by design.
  • The high fragmentation of your large object heap is a sign that objects are being freed, leaving holes in the heap (so at least your application is not leaking memory). These holes will be filled with new objects if they fit into the hole.
  • Memory is becoming increasingly cheaper, making the business case of memory optimization smaller and smaller. Memory leaks is still a problem, but memory usage is not.

HEY, IT’S A PROBLEM FOR ME. WHAT CAN I DO ABOUT IT?

But let’s say that memory IS a problem – what can you do about it?

TUNE YOUR CACHES:

If memory is a problem, you should not let the cache sizes run free. Set Caching.DisableCacheSizeLimits to false and tune your Sitecore caches:

<setting name="Caching.DisableCacheSizeLimits" value="false" />

I have provided a list of links to cache tuning guides at the bottom of this post.

REUSE DUPLICATED FIELD VALUES:

.NET introduced a string object pool pattern that allows you to reference strings that are duplicated. Strings cannot be changed once created (also called “immutable objects”), and can therefore be referenced using interning. But only if you code it that way.

Sitecore have implemented an interning pattern in the \App_Config\Sitecore\CMS.Core\Sitecore.Interning.config file where fields that often have duplicated values can be stored:

<interning>
  <fieldIdsToIntern>
    <workflowState>{3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98}</workflowState>
    <workflow>{A4F985D9-98B3-4B52-AAAF-4344F6E747C6}</workflow>
    <updatedBy>{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}</updatedBy>
    <createdBy>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</createdBy>
    ...
    ...
  </fieldIdsToIntern>
</interning>

You can add your own duplicated fields to the list. Nikolay Mitikov created this SQL to find duplicated values:

WITH DuplicatedFieldValues AS  (
SELECT 
	v.FieldId,	
	FieldDefinitionRow.[Name],
	CONVERT(NVARCHAR(250),v.[Value]) AS [Field Value],
	COUNT(1) AS [Hits]
FROM 
	[VersionedFields] v 
JOIN 
	[Items] FieldDefinitionRow ON FieldDefinitionRow.ID = v.fieldID
WHERE 
	v.FieldId NOT IN 
	(
		/* Fields already interned OOB by Sitecore.Interning.config */			
		'BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A' /* updated by */,
		'5DD74568-4D4B-44C1-B513-0AF5F4CDA34F' /* created by */,
		'52807595-0F8F-4B20-8D2A-CB71D28C6103' /* owner */,
		'3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98' /* workflow state */	
	)
GROUP BY 
	FieldId, [Name], CONVERT(NVARCHAR(250),[Value])
HAVING 
	COUNT(*) > 500 /* How many same field values must be met to be shown */)

SELECT * FROM DuplicatedFieldValues
ORDER BY [Hits] DESC

This can release a lot of memory, especially if you have large text fields.

USE Sitecore.Diagnostics.MemoryMonitorHook TO CLEAR CACHE AND FORCE GARBAGE COLLECTION

The Sitecore.Diagnostics.MemoryMonitorHook monitors your memory usage and can be set to force a garbage collection and a cache clear when the threshold is met:

<hooks>
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
        <param desc="Threshold">8192MB</param>
        <param desc="Check interval">00:15:00</param>
        <ClearCaches>true</ClearCaches>
        <GarbageCollect>true</GarbageCollect>
        <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
</hooks>
  • Threshold: How much memory should be used before doing anything. Set this to 60-80% of your total memory.
  • Check interval: How often should the hook run.
  • ClearCaches: When threshold is met, should the cache be cleared?
  • GarbageCollect: When threshold is met, should Sitecore do a forced garbage collect?
  • AdjustLoadFactor: When threshold is met, should Sitecore attempt to adjust the garbage collection load factor?

MORE TO READ:

 

Posted in Sitecore 6, Sitecore 7, Sitecore 8, Sitecore 9 | Tagged , , , , , | 1 Comment

Remove duplicates from XML feed

Apparently XML isn’t dead yet, and today I received a Google Product Feed in the RSS 2.0 XML format. The feed was full of duplicates and my job is to remove them:

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
    <channel>
        <item>
            <g:id>100</g:id>
            <title>Product 100</title>
            ...
            ...
        </item>
        <item>
            <g:id>100</g:id>
            <title>Product 100</title>
            ...
            ...
        </item>
        <item>
            <g:id>200</g:id>
            <title>Product 200</title>
            ...
            ...
        </item>
        <item>
            <g:id>300</g:id>
            <title>Product 300</title>
            ...
            ...
        </item>
    </channel>
</rss>

As you can see, “Product 100” appears twice.

THE SOLUTION:

A little LINQ can get you far:

using System.Xml;
using System.Xml.Linq;
using System.Linq;

var document = XDocument.Parse(theXMLString);

XNamespace g = "http://base.google.com/ns/1.0";
document.Descendants().Where(node => node.Name == "item");
    .GroupBy(node => node.Element(g+"id").Value)
    .SelectMany(node => node.Skip(1))
    .Remove();

HOW IT WORKS:

  • document.Descendants().Where(node => node.Name == “item”): Get all elements called “item
  • GroupBy(node => node.Element(g+”id”).Value): Group them by the “g:id” element.
  • SelectMany(node => node.Skip(1)): Select every one of them apart from the first one
  • Remove(): Delete all that were selected

MORE TO READ:

Posted in .net, .NET Core, c#, General .NET | Tagged , , , , | Leave a comment

HttpClient retry mechanism with .NET Core, Polly and IHttpClientFactory

A lot of HttpClient errors are temporary and is caused by server overload, temporary nerwork timeouts and generic gliches in the Matrix. These scenarios can be dealt with using a retry pattern. In .NET Core, the most common retry library is the Polly library:

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.
http://www.thepollyproject.org/

Polly makes it relatively easy to implement a retry pattern, as long as you use the IHttpClient and IHttpClientFactory.

But enugh talk, lets code.

STEP 1: THE NUGET PACKAGES

You need (at least) the following NuGet Packages:

  • Polly
  • Microsoft.Extensions.Http.Polly

STEP 2: CONFIGURE SERVICES IN STARTUP.CS

In the services configuration, you need to add a IHttpClientFactory and attach a PolicyHandler to the factory:

//ConfigureServices()  - Startup.cs
services.AddHttpClient("HttpClient").AddPolicyHandler(GetRetryPolicy());

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
  return HttpPolicyExtensions
    // Handle HttpRequestExceptions, 408 and 5xx status codes
    .HandleTransientHttpError()
    // Handle 404 not found
	.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
    // Handle 401 Unauthorized
	.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.Unauthorized)
    // What to do if any of the above erros occur:
	// Retry 3 times, each time wait 1,2 and 4 seconds before retrying.
	.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
}

STEP 3: USE THE IHttpClientFactory IN THE CALLING CLASS

The IHttpClientFactory can be injected using constructor injection. The cool part of the Polly implementation is that your HttpClient code does not contain any special retry-code, just the usual Get or Post calls:

namespace MyCode
{
  public class MyClass
  {
    private readonly IHttpClientFactory _clientFactory;

    public MyClass(IHttpClientFactory clientFactory)
    {
      _clientFactory = clientFactory;
    }

    public async Task<int> PostMessage(string postData)
    {
      var httpClient = _clientFactory.CreateClient("HttpClient");

      using (var content = new StringContent(postData, Encoding.UTF8, "application/json"))
      {
        var result = await httpClient.PostAsync($"{url}", content);
        // The call was a success
        if (result.StatusCode == HttpStatusCode.Accepted)
        {
          return result.StatusCode;
        }
        // The call was not a success, do something
        else
        {
          // Do something
          return result.StatusCode;
        }
      }
    }
  }
}

The httpClient.PostAsync() will retry the post call automatically if any of the conditions described in the GetRetryPolicy() occurs. It will only return after the call is either successful or the retry count is met.

MORE TO READ:

Posted in .NET Core, General .NET | Tagged , , | 1 Comment

.NET Core Worker Services with Application Insights and Serilog

The .NET Core Worker service is yet another tool in the .NET toolbox. They are perfect for background processing like reading from a queue or making health checks. They are cross-platform (of course) and they can run on docker containers, and even run on a locally hosted machine as a Windows service.

It took me quite some time to implement Application Insights and file logging (Serilog) in my worker service, so I thought I might write it down, in case I need it another time.

THE NUGET PACKAGES

First caveat was when I realized that you need a special Microsoft.ApplicationInsights.WorkerService NuGet package before Application Insights will work. Also, if you wish to run the worker service as a Windows service you need the Microsoft.Extensions.Hosting.WindowsServices package.

The packages I needed was:

Microsoft.ApplicationInsights.AspNetCore
Microsoft.ApplicationInsights.WorkerService
Microsoft.Extensions.Hosting
Microsoft.Extensions.Hosting.WindowsServices
Serilog
Serilog.AspNetCore
Serilog.Extensions.Hosting
Serilog.Extensions.Logging
Serilog.Settings.Configuration
Serilog.Sinks.Console
System.Configuration.ConfigurationManager

THE APPSETTINGS.JSON CONFIGURATION

The configuration is pretty straight forward. Only caveat is that Serilog is not configured under “Logging” but under it’s own namespace:

{
  "ApplicationInsights": {
    "InstrumentationKey": "the-appinsights-guid"
  },
  "Serilog": {
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "log_.txt",
          "rollingInterval": "Day"
        }
      }
    ]
  },
  "Logging": {
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Information",
        "Microsoft": "Warning"
      }
    },
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Information",
      "Microsoft.Hosting.Lifetime": "Warning"
    }
  }
}

THE PROGRAM.CS

The examples online was too complex for me, so I simplified it and came up with this:

namespace MyWorkerService
{
  public class Program
  {
    public static void Main(string[] args)
    {
      // This will run the worker service
      CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args)
    {
      var host = Host.CreateDefaultBuilder(args);
      // Add this line to be able to run as a windows service
      // https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-3.1&tabs=visual-studio
      host.UseWindowsService();
      // This will add the configuration to the application.
      // It will allow you to inject the configuration as 
      // IConfiguration configuration in your constructors 
      host.ConfigureAppConfiguration(
            (hostContext, config) =>
            {
              config.SetBasePath(Directory.GetCurrentDirectory());
              config.AddJsonFile("appsettings.json", false, true);
              config.AddCommandLine(args);
            }
      );
      // This will configure logging. It reads the log settings from 
      // the appsettings.json configuration file, and adds serilog,
      // allowing the application to write logs to file
      host.ConfigureLogging(
            loggingBuilder =>
            {
              var configuration = new ConfigurationBuilder()
                 .AddJsonFile("appsettings.json")
                 .Build();
              var logger = new LoggerConfiguration()
                  .ReadFrom.Configuration(configuration)
                  .CreateLogger();
              loggingBuilder.AddSerilog(logger, dispose: true);
            }
      );
      // The AddHostedServer adds the worker service to the application.
      // The AddApplicationInsightsTelemetryWorkerService is very important. Without this,
      // the Application Insights logging will not work.
      host.ConfigureServices((hostContext, services) =>
      {
        services.AddHostedService<Worker>();
        services.AddApplicationInsightsTelemetryWorkerService();
      });

      return host;
    }
  }
}

The worker itself is the “Worker” class and it looks like this:

using System.Threading;
using System.Threading.Tasks;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Configuration;

namespace MyWorkerService
{
  public class Worker : BackgroundService
  {
    private readonly ILogger<Worker> _logger;
    private readonly TelemetryClient _telemetryClient;

    public Worker(ILogger<Worker> logger,
      TelemetryClient telemetryClient,
      IConfiguration configuration)
    {
      _logger = logger;
      _telemetryClient = telemetryClient;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
      while (!stoppingToken.IsCancellationRequested)
      {
        using (_telemetryClient.StartOperation<RequestTelemetry>("Execute Async"))
        {
          _logger.LogInformation("ExecuteAsync Started");
          DoWork();
        }
        await Task.Delay(1000, stoppingToken);
      }
    }
  }
}

The DoWork() is where you would do actual work.

MORE TO READ:

Posted in .net, .NET Core, c# | Tagged , , , | 6 Comments

System.DllNotFoundException: Unable to load DLL ‘sni.dll’ or one of its dependencies: The specified module could n ot be found. (0x8007007E)

This message happens when deploying my .NET Core 3.1 application to production (when compiling the code to an .exe file), but not when running the application locally.

It turns out, that Dapper is missing a reference to System.Data.SqlClient. Adding the System.Data.SqlClient NuGet package to the project solves the issue:

System.Data.SqlClient

System.Data.SqlClient

The full error message is:

Unhandled exception. Unhandled exception. System.TypeInitializationException: The type initializer for ‘System.Data.SqlClient.TdsParser’ threw an exception.
—> System.TypeInitializationException: The type initializer for ‘System.Data.SqlClient.SNILoadHandle’ threw an exception.
—> System.DllNotFoundException: Unable to load DLL ‘sni.dll’ or one of its dependencies: The specified module could not be found. (0x8007007E)
at System.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize(IntPtr pmo)
at System.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize()
at System.Data.SqlClient.SNILoadHandle..ctor()
at System.Data.SqlClient.SNILoadHandle..cctor()
— End of inner exception stack trace —
at System.Data.SqlClient.TdsParser..cctor()
— End of inner exception stack trace —
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObject
sTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& co
nnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry
, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`
1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Dapper.SqlMapper.QueryImpl[T](IDbConnection cnn, CommandDefinition command, Type effectiveType)+MoveNext() in C:\projects\dapper\Dapper\SqlMapper.cs:line 1079
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in C:\projects\dapper\Dapper\SqlMapper.cs:line 721

MORE TO READ:

Posted in .net, .NET Core, c# | Tagged , , | 1 Comment