Sitecore General error when submitting contact – Another contact with the same identifier already exists

In Sitecore when creating your own contacts you can get the following exception:

10604 10:16:27 ERROR General error when submitting contact.
Exception: System.InvalidOperationException
Message: Another contact with the same identifier already exists.
Source: Sitecore.Analytics.MongoDB
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDataAdapterProvider.SaveContactWithIdentifier(IContact contact, ContactSaveOptions saveOptions)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDataAdapterProvider.<>c__DisplayClass9.<SaveContact>b__7()
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDataAdapterProvider.Try(Action action)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDataAdapterProvider.SaveContact(IContact contact, ContactSaveOptions saveOptions)
   at Sitecore.Analytics.Data.ContactRepository.SaveContact(Contact contact, ContactSaveOptions options)
   at Sitecore.Analytics.Tracking.ContactManager.SubmitContact(Contact contact, ContactSaveOptions options, Contact& realContact)

The error can occur if you try to flush a contact with a new contact ID but with an existing identifier. It usually is a result of Sitecore having a mismatch between what the ContactRepository and the MongoDB contains.

This pseudocode explains the situation:

ContactRepository contactRepository = Factory.CreateObject("tracking/contactRepository", true) as ContactRepository;
ContactManager contactManager = Factory.CreateObject("tracking/contactManager", true) as ContactManager;
// Contact is loaded from repository but repository cannot find it
Contact contact = contactRepository.LoadContactReadOnly(username);
// ... so we try to create it
if (contact == null)
  // Contact is created in memory
  contact = contactRepository.CreateContact(ID.NewID);
 contact.Identifiers.Identifier = username;
  // And we try to write it to xDB (MongoDB)
  // but the contact is already there. 

The problem arises as the FlushContactToXdb does not throw an exception, so you are left with the impression that the Contact is created and everything is fine.

If you are using the ExtendedContactRepository as I described in the blog post Sitecore Contacts – Create and save contacts to and from xDB (MongoDB), you risk that the GetOrCreateContact method goes in an infinite loop because:

  1. We try to load the contact, but the contact is not found
  2. We Create a new contact
  3. We flush the contact, but the flush method fails silently
  4. We call step 1

To avoid the infinite loop, please see Dan’s suggestion to how to solve it:

Or you can add a retry counter in the class.

Add the following 2 lines to the class:

private const int _MAX_RETRIES = 10;
private int _retries;

Then modify the private method GetOrCreateContact:

private Contact GetOrCreateContact(string username, LockAttemptResult<Contact> lockAttempt, ContactRepository contactRepository, ContactManager contactManager)
  switch (lockAttempt.Status)
	case LockAttemptStatus.Success:
	  Contact lockedContact = lockAttempt.Object;
	  lockedContact.ContactSaveMode = ContactSaveMode.AlwaysSave;
	  return lockedContact;

	case LockAttemptStatus.NotFound:
	  Contact createdContact = CreateContact(username, contactRepository);
	  Log.Info(string.Format("{0}: Flushing contact '{1}' (contact ID '{2}') to xDB.", GetType().Name, createdContact.Identifiers.Identifier, createdContact.ContactId), this);

	  // NEW CODE: Check for retries, and throw an exception if retry count is reached
	  if (_retries >= _MAX_RETRIES)
		throw new Exception(string.Format("ExtendedContactRepository: Contact {0} could not be created. ", username));

	  return GetOrCreateContact(username);

	  throw new Exception("ExtendedContactRepository: Contact could not be locked - " + username);

This is still experimental code, but as described before, so is the ExtendedContactRepository.



Posted in .net, c#, General .NET, Sitecore 8 | Tagged , , , , , | 1 Comment

Sitecore Scheduled Tasks – Run on certain server instance

If you would like certain Sitecore Scheduled Tasks run on a certain server instance (your CM server for example) the approach is pretty simple.

First, create a new folder under /sitecore/system/Tasks. I have a new folder called “CMSchedules“. Place your Scheduled Tasks in this folder.

Scheduled Tasks in separate folder

Scheduled Tasks in separate folder

Then tell Sitecore to execute these tasks. Place the following configuration on the instance you wish should run these tasks. Do not place the configuration on servers that should not run the tasks.

You need to patch the Sitecore.config (for Sitecore 8.1 and up) or web.config (for versions prior to Sitecore 8.1). Add the following to the /sitecore/scheduling section:

<agent type="Sitecore.Tasks.DatabaseAgent" method="Run" interval="00:05:00">
  <param desc="database">master</param>
  <param desc="schedule root">/sitecore/system/Tasks/CMSchedules</param>

The configuration tells Sitecore to execute all tasks in the /sitecore/system/Tasks/CMSchedules every 5 minutes.



Posted in Sitecore 5, Sitecore 6, Sitecore 7, Sitecore 8 | Tagged , | 3 Comments

Sitecore Publishing – Programmatically determine if item should be published

Sitecore uses it’s publish settings to determine if an item should be published. But you can only control a publish from versions and date times.

Sitecore Publishing Settings

Sitecore Publishing Settings

So what if you have other values that determine if an item must be published or not? Say, a status field or a checkbox, or a combination of both? And what do you do if these fields are updated from external systems?

One way is to extend the publishItem pipeline. Every item that must be published goes through this pipeline one by one and you can therefore add a new processor that determines if an item is published or not.


These are our publish options: A status field and a checkbox is updated by users and external systems to determine if this item is eligible for publishing.

Publish Options

Publish Options

The publishItem pipeline works “the other way round”, meaning that if the item can be published we leave it alone, and if not we change the publish status to “DeleteTargetItem“.

Here is my new processor (some pseudocode, apply your own checks):

using Sitecore.Data.Items;
using Sitecore.Diagnostics;
using Sitecore.Globalization;
using Sitecore.Publishing;
using Sitecore.Publishing.Pipelines.PublishItem;

namespace MyCode.Infrastructure.Pipelines.PublishItem
  public class DetermineActionForItem : PublishItemProcessor
    public override void Process(PublishItemContext context)
      // First a list of checks to ensure that
      // it is our item and we can make the decision
      // about it's publish status
      if (context == null)
      if (context.Aborted)
      Item sourceItem = context.PublishHelper.GetSourceItem(context.ItemId);
      if (sourceItem == null)
      if (!sourceItem.Paths.IsContentItem)
      // I will only check our specific item
      if (!sourceItem.TemplateName == "mytemplate")

      // OK, now we know that this is our item and
      // we can determine it's faith
      // Check every language to see if it is eligible for publishing
      foreach (Language language in sourceItem.Languages)
        Item languageVersion = sourceItem.Versions.GetLatestVersion(language);
        // A little pseudocode, here is the check to see if the item can be published
        if (StatusIsOK(languageVersion["StatusField"]) && CheckBoxIsChecked(languageVersion["CheckBoxField"])
          // Yes, the item can be published

      // No, the item cannot be published, set the publishaction == DeleteTargetItem
      Log.Info(string.Format("{0}: Unpublishing Item '{1}' from database '{2}' because it is not in the correct state.",
      context.Action = PublishAction.DeleteTargetItem;

As you can see, the function returns if the item being processed is not our item, or if the item can be published. If the item is not eligible for publishing, we change the Action to DeleteTargetItem.

The processor is added to the publish pipeline just before the Sitecore “DetermineAction” processor:

<publishItem help="Processors should derive from Sitecore.Publishing.Pipelines.PublishItem.PublishItemProcessor">
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.RaiseProcessingEvent, Sitecore.Kernel"/>
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.CheckVirtualItem, Sitecore.Kernel"/>
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.CheckSecurity, Sitecore.Kernel"/>
  <!-- Our Processor inserted before Sitecore's "DetermineAction" proecssor -->
  <processor type="MyCode.Infrastructure.Pipelines.PublishItem.DetermineActionForItem, MyDll" />
  <!-- ... -->
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.DetermineAction, Sitecore.Kernel"/>



Posted in General .NET, Sitecore 6, Sitecore 7, Sitecore 8 | Tagged , , , | Leave a comment

Sitecore Event Queue – How to clean it – and why

Yesterday the dreaded Sitecore Event Queue almost killed me again – well it certainly almost killed my CM server. The server went from being busy but stable to being unresponsive. CPU and memory load skyrocketed:

A fully loaded CM server

A fully loaded CM server

Obviously it happened after a system update, so after a panic debug and rollback the system owner pointed out: “Hey, the event queue table is quite big?“.
Of course, the system updated flooded the event with 1.5 million events, and the problem did not go away because I keep 1 day of events in the queue.


First we need to stabilize the system, then we need to update the configuration.


The following SQL statement will clean out the history table, publish queue and event queue, leaving only 12 hours of history and publish data and 4 hours of events. Replace YOURDATABASE with the name of your database:

/****** History ******/
delete FROM [YOURDATABASE_Core].[dbo].[History] where Created < DATEADD(HOUR, -12, GETDATE())
delete FROM [YOURDATABASE_Master].[dbo].[History] where Created < DATEADD(HOUR, -12, GETDATE())
delete FROM [YOURDATABASE_Web].[dbo].[History] where Created < DATEADD(HOUR, -12, GETDATE())
/****** Publishqueue ******/
delete FROM [YOURDATABASE_Core].[dbo].[PublishQueue] where Date < DATEADD(HOUR, -12, GETDATE());    
delete FROM [YOURDATABASE_Master].[dbo].[PublishQueue] where Date < DATEADD(HOUR, -12, GETDATE());
delete FROM [YOURDATABASE_Web].[dbo].[PublishQueue] where Date < DATEADD(HOUR, -12, GETDATE());
/****** EventQueue ******/
delete FROM [YOURDATABASE_Master].[dbo].[EventQueue] where [Created] < DATEADD(HOUR, -4, GETDATE())
delete FROM [YOURDATABASE_Core].[dbo].[EventQueue] where [Created] < DATEADD(HOUR, -4, GETDATE())
delete FROM [YOURDATABASE_Web].[dbo].[EventQueue] where [Created] < DATEADD(HOUR, -4, GETDATE())


With the system stabilized, we need to take more care of the table sizes.


Sitecore is already configured to clean the tables so they only contain 12 hours of data. 12 hours of data is usually what any SQL server will handle, and you will have up to 10.000 rows in the table.

  <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.Kernel">
    <param connectionStringName="$(id)" />


Sitecore keeps 30 days of publish queue. If you insert and update items often, you should lower this number. For each item change (including any change that the system does) is stored here.

<agent type="Sitecore.Tasks.CleanupPublishQueue, Sitecore.Kernel" method="Run" interval="04:00:00">


The event queue is the most important table to keep small. In a distributed environment, each server will read the contents of the table every 5 seconds, using a time stamp stored in the Properties table as key. Any row before the time stamp will not be read.

You therefore need enough history to cater that a server will be offline for a while, but at the same time so little contents that any read and write will be amazingly fast.

If you can keep the number of rows below 7.000, most SQL server should be able to handle that amount of data. Even smaller numbers are preferred as well.

Before Sitecore 8.1, Sitecore would only allow you to clean events older that 1 day. This is way too much, especially if you publish often. The new IntervalToKeep will allow you to determine the hours to keep as well:

<agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00">



After these changes, my server is back to normal, busy but responsive:

Normal Load

Normal Load



Posted in Sitecore 8 | Tagged , , , , | 2 Comments

Sitecore Measure Pipeline performance and profile processors

In Sitecore, Pipelines are by far my favorite dependency injection pattern, and I have used them since Sitecore 5. One of the secret features of pipelines are the built in profiling. Oh, yes, by flipping 2 switches you can measure the performance of your pipeline processors.


Enable the \App_Config\Include\Sitecore.PipelineProfiling.config.disabled by removing the .disabled extension.

Or add the following 2 settings to your /App_Config/sitecore.config:

<setting name="Pipelines.Profiling.Enabled" value="true" />
<setting name="Pipelines.Profiling.MeasureCpuTime" value="true" />

The Pipelines.Profiling.MeasureCpuTime is not enabled by default in the Sitecore.PipelineProfiling.config file as it introduces a performance overhead, so only use this setting on your test environment.


Sitecore comes with a profiling page (it has been there since Sitecore 7 i guess):


Pipeline Profiler

Pipeline Profiler

The page displays all pipelines executed since you pressed the refresh button, how many times it has been executed, and the average wall time (wall time = real-world time from processor started to it finished, as opposed to CPU time which is the time spend by the CPU executing the processor) per execution.



Posted in Sitecore 7, Sitecore 8 | Tagged , , , | Leave a comment

Sitecore Advanced Configuration in .config files

Sitecore allows you to create configuration sections in /configuration/sitecore/ that consists of hierarchies of configurations including lists, and it will even help you serialize the configuration to object instances.

Observe the following real life example of a configuration I created to configure lists of IP adresses and IP ranges to ignore in my web application:

  <restriction type="Restrictions.Model.Configuration, Restrictions">
    <ipraddresses hint="raw:AddIpAddresses" >
<address ip="" />
<address ip="" />
    <ipranges hint="raw:AddIpRanges">
      <range from="" to=""/>
      <range from="" to=""/>

The configuration section is added to a Sitecore .config file under /configuration/sitecore/.

Please notice the “type” property “Restrictions.Model.Configuration, Restrictions” and the 2 “hint” properties. These allows Sitecore to perform some reflection magic that will map the configuration section to classes.

So let’s code. First I will create an interface that outlines my configuration, a list of Ip Addresses and a list of Ip Ranges

namespace Restrictions.Model
  public interface IConfiguration
    IEnumerable<IpAddress> IpAddresses { get; }
    IEnumerable<IpRange> IpRanges { get; }

I need implementations of IpAddress and IpRange. The implementations contains the XML attributes as properties:

namespace Restrictions.Model
  public class IpAddress
    public string Ip { get; set; }

  public class IpRange
    public string From { get; set; }
    public string To { get; set; }

For the final touch,  I will assemble it all in a Configuration class. And here is where the magic lies. Remember the “hint=”raw:AddIpAddresses” and “hint=”raw.AddIpRanges”” attributes? These are method names in my class. Sitecore calls these for each Xml node:

using System.Collections.Generic;
using System.Xml;

namespace Restrictions.Model
  public class Configuration : IConfiguration
    private readonly List<IpAddress> _ipAddresses = new List<IpAddress>();
    public IEnumerable<IpAddress> IpAddresses { get { return _ipAddresses; } }

    private readonly List<IpRange> _ipRanges = new List<IpRange>();
    public IEnumerable<IpRange> IpRanges  { get { return _ipRanges; } }

    protected void AddIpAddresses(XmlNode node)
      if (node == null)
      if (node.Attributes == null)
      if (node.Attributes["ip"] == null)
      if (node.Attributes["ip"].Value == null)

      _ipAddresses.Add(new IpAddress() { Ip = node.Attributes["ip"].Value });

    protected void AddIpRanges(XmlNode node)
      if (node == null)
      if (node.Attributes == null)
      if (node.Attributes["from"] == null)
      if (node.Attributes["from"].Value == null)
      if (node.Attributes["to"] == null)
      if (node.Attributes["to"].Value == null)

      _ipRanges.Add(new IpRange() { From = node.Attributes["from"].Value, To = node.Attributes["to"].Value });

The final class is the repository that will use reflection magic to convert an XmlNode to a class implementation:

using System.Xml;

namespace Restrictions.Model.Repositories
  public class ConfigurationRepository
    public IConfiguration GetConfiguration(string restrictionName)
      XmlNode xmlNode = Sitecore.Configuration.Factory.GetConfigNode("pt_restrictions/restriction");
      return Sitecore.Configuration.Factory.CreateObject<IConfiguration>(xmlNode);

I use the class like this:

ConfigurationRepository rep = new ConfigurationRepository();
IConfiguration config = rep.GetConfiguration();

foreach (var ipaddress in config.IpAddresses)
  // do stuff;
foreach (var iprange in config.IpRanges)
  // do stuff

It looks like a big job (and it probably is) but once you get it running, it is easy to extend and use elsewhere.



Posted in .net, c#, General .NET, Sitecore, Sitecore 5, Sitecore 6, Sitecore 7, Sitecore 8 | Tagged , , , , | Leave a comment

Sitecore PostSessionEndPipeline failed – MongoDB.Driver.MongoQueryException

In my (huge) Sitecore solution we experienced the following error:

ERROR PostSessionEndPipeline failed.
Exception: MongoDB.Driver.MongoQueryException
Message: QueryFailure flag was Runner error: Overflow sort stage buffered data usage of 33570904 bytes exceeds internal limit of 33554432 bytes (response was { “$err” : “Runner error: Overflow sort stage buffered data usage of 33570904 bytes exceeds internal limit of 33554432 bytes”, “code” : 17144 }).
Source: MongoDB.Driver
at Sitecore.Analytics.Pipelines.CommitSession.SubmitSession.Process(CommitSessionPipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Analytics.Pipelines.PostSessionEnd.CommitSession.Process(PostSessionEndArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)

Followed by the following message:

QueryFailure flag was Runner error: Overflow sort stage buffered data usage of xxxxxxxx bytes exceeds internal limit of 33554432 bytes (response was { “$err” : “Runner error: Overflow sort stage buffered data usage of xxxxxxxx bytes exceeds internal limit of 33554432 bytes”, “code” : 17144 }).

The error is caused by an internal 32MB memory size restriction in MongoDB.

The problem can be solved by adding an index to the Interactions collection. This will allow Sitecore to stream interactions sorted rather than attempting to load them all into memory and do the sorting in memory.

Connect to the analytics index using RoboMongo or MongoVUE and run the following query:

    "StartDateTime" : 1,
    "_id" : 1


According to MongoDB, adding indexes to a live environment does not have any side effects. But running the above query on my 7.5 million entries in the Interactions collection caused my entire Sitecore to fail (total catastrophic failure) for 5-10 minutes until the index was built.

Thank you to Sitecore Support and Joshua Wheelock for helping with the issue.


Posted in Sitecore 8 | Tagged , , | Leave a comment

Sitecore get host name from a different context

Ohno, you are running in one context, say the “shell” context, and all of your URL’s have to point to the “website” context. The hustler will obviously hard-code the domain name of the “website” into the code. But the Sitecore aficionado will check the SiteContext and retrieve the “TargetHostName” property.

This tiny class will qualify your relative URL’s to absolute URL’s, matching the domain name as specified in the <sites> section of your configuration (code simplified for readability, you should apply your own exception handling):

using System;
using System.Web;
using Sitecore.Sites;

namespace MyNameSpace
  internal static class FullyQualifiedUrlService
    public static string Qualify(string relativeUrl, string sitename)
      SiteContext sitecontext = SiteContext.GetSite(sitename);
      return Qualify(relativeUrl, sitecontext);

    public static string Qualify(string relativeUrl, SiteContext sitecontext)
      if (!relativeUrl.StartsWith("/"))
        relativeUrl = relativeUrl + "/";
      return string.Format("{0}://{1}{2}", sitecontext.SiteInfo.Scheme, sitecontext.TargetHostName, relativeUrl);


Imagine this is your sites definition for the “website” context:

  <site name="website" targetHostName="" hostName="|" scheme="http" ... .../>

The class is called using the following code:

string relativeUrl = "/this/is/my/page";
string absoluteUrl = FullyQualifiedUrlService.Qualify(relativeUrl, "website");

And will return the following result:

Both the scheme and the targetHostName is resolved from the context using the Scheme and TargetHostName properties of the SiteContext.

Sitecore uses 2 properties for host resolving:

  • The “hostName” can be a pipe-separated list of domains and is used to target the number of possible URL’s that points to this context.
  • The “targetHostName” is one URL which is used internally to resolve and fully qualify your URL.



Posted in c#, General .NET, Sitecore 6, Sitecore 7, Sitecore 8 | Tagged , , , | Leave a comment

Sitecore EXM keeps reloading

Have you experienced that in Sitecore, the EXM Email Experience Manager window keeps reloading?

Email Experience Manager

Email Experience Manager

In my solution, the problem was that I have a cookie with a “:” (colon) in the cookie name. According to the old RFC 2616 specification, colons are not allowed in cookie names.

Although this is not a problem in the website or in the Sitecore Shell, it causes problems when this particular SPEAK app calls the server API.

The solution is to avoid using colons in cookie names.


Posted in Sitecore 8 | Tagged , , , | Leave a comment

Sitecore SVG files

In Sitecore 8.1, there is a tiny but annoying glitch in the matrix, where SVG files are not allowed in the Media Library.

But do not worry, the fix is easy. Go to your /App_Config/Sitecore.config, find the /sitecore/mediaLibrary/mediaTypes/mediaType section, and add the SVG section yourself:

<mediaType name="SVG" extensions="svg">
  <mediaValidator type="Sitecore.Resources.Media.ImageValidator"/>
    <generator type="Sitecore.Resources.Media.ImageThumbnailGenerator, Sitecore.Kernel">


Posted in Sitecore, Sitecore 8 | Tagged , | 1 Comment