Sitecore and Application Insights – How to remove 90% of all log lines without sacrificing traceability

In this article I will explain how you can remove up to 90% of all log lines from the Application Insights log, but still keep every log line in the file log. All of this without loosing any important information.

Sitecore uses a Log4Net Appender to inject log lines into Application Insights. If you take a look at the App_Config/Sitecore/Azure/Sitecore.Cloud.ApplicationInsights.config file that is deployed to your CM/CD servers, you will find the appender at the top, and it could look like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:env="http://www.sitecore.net/xmlconfig/env/" xmlns:role="http://www.sitecore.net/xmlconfig/role/"> 
<sitecore> 
  <log4net> 
    <root> 
      <appender-ref ref="ApplicationInsightsAppender" patch:after="appender-ref[@ref='LogFileAppender']" />
    </root> 
    <appender name="ApplicationInsightsAppender" type="Sitecore.Cloud.ApplicationInsights.Logging.LevelTraceAppender, Sitecore.Cloud.ApplicationInsights" patch:after="appender[@name='LogFileAppender']"> 
      <threshold value="INFO" /> 
        <layout type="log4net.Layout.PatternLayout"> 
          <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
        </layout> 
    </appender> 
  </log4net> 
</sitecore> 
</configuration>

Sitecore does log a lot of information that probably should have been marked as debug information. This is fine when logging to a log file. But with Application Insights there are issues with logging too much information:

  • You have to pay for what you log. Although Azure is priced reasonable, it is not free. Any excessive logging can make your budget explode.
  • You might run into  sampling issues (“Data received from your application is being sampled to reduce the volume of telemetry data retained; only sampled documents will be returned. The sampling may be applied by the Application Insights SDK or on ingestion by Application Insights.“). This can be disabled, but sampling usually are there for a reason, as Application Insights cost money the more lines you log.

When I analyze my log files, there are several lines that have never helped me:

ManagedPoolThread #6 00:02:46 INFO Job started: Job started: Index_Update_IndexName=sitecore_Master_index
ManagedPoolThread #9 00:04:48 INFO Scheduling.DatabaseAgent started. Database: master
ManagedPoolThread #13 00:04:48 INFO Cache created: 'ExperienceAnalytics.DimensionItems' (max size: 2MB, running total: 9090MB)

The “Job started” alone can take up to 90% of the log lines in a solution!

To remove these log lines from the Application Insights log, we can use the log4net.Filter.StringMatchFilter. This filter can add or remove lines containing a string.

Filters are added to the appender attribute in the config file like this:

<log4net>
  <root>
    <appender-ref ref="ApplicationInsightsAppender" patch:after="appender-ref[@ref='LogFileAppender']" />
  </root>
  <appender name="ApplicationInsightsAppender" type="Sitecore.Cloud.ApplicationInsights.Logging.LevelTraceAppender, Sitecore.Cloud.ApplicationInsights" patch:after="appender[@name='LogFileAppender']">
    <threshold value="INFO" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
    </layout>
    <filter type="log4net.Filter.StringMatchFilter">
      <stringToMatch value="Job started: Index_Update_IndexName=" />
      <acceptOnMatch value="false" />
    </filter>
    ...
    ...
  </appender>
</log4net>

The “stringToMatch” is the string that Log4Net looks for, and the acceptOnMatch=false ensure that the log line is not appended to the log.

I have found the following candidates for removal from the Application Insights log:

  • Job started: Index_Update_IndexName=
  • Job ended: Index_Update_IndexName=
  • Job started: UpdateIndexDocuments
  • Job ended: UpdateIndexDocuments
  • Job started: Sitecore.
  • Job ended: Sitecore.
  • Request is redirected to no layout page
  • HttpModule is being initialized
  • File is being deleted by cleanup task
  • Cache created:
  • Health.

There are probably more depending on your Sitecore installation. You should use the old but excellent Sitecore Log Analyzer to analyse the file logs and find the repetitive log lines that you never use, and filter them from the Application Insights log. Remember that what does not help me, might help you. But removing noise really does give a better overview, and your client’s wallet will be happy too :).

MORE TO READ:

 

About briancaos

Developer at Pentia A/S since 2003. Have developed Web Applications using Sitecore Since Sitecore 4.1.
This entry was posted in Microsoft Azure, Sitecore 8, Sitecore 9 and tagged , , , , , . Bookmark the permalink.

1 Response to Sitecore and Application Insights – How to remove 90% of all log lines without sacrificing traceability

  1. This is a great write up on how to save some money in Azure. I found out the hard way how much some of these default logging methods actually log out of the box when I setup my dev environment in Azure. It is wild as you stated how “Job Started” takes up almost the entire log LOL. Great tips for reducing the log size while keeping a useful log!

    Cheers

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.