Sitecore media library missing PDF icons

When uploading PDF (or Word/Excel/…) documents to the Sitecore media library, the PDF icon is not shown:

PDF icon Missing

PDF icon Missing

In this case it is only my test server that is not showing the icons. My local development machine (my own laptop) displays the icons perfectly:

PDF icon Shown

PDF icon Shown on my laptop

The reason is that Siteore use the associated icon from the OS. Acrobat Reader is installed on my laptop, not on my test server. So the test server displays a default icon instead.

WHAT’S THE SOLUTION THEN?

You can specify which static file to use for which extension in the web.config. To add the PDF icon, do the following:

Download a .pdf icon from the Adobe Website (due to licensing restrictions, Sitecore is not shipped with an Acrobat Reader icon). Place the icon wherever you wish.

Go to the web.config and find the <mediaType name=”PDF file” extensions=”pdf”> section.

Add the following:

<mediaType name="PDF file" extensions="pdf">
  <mimeType>application/pdf</mimeType>
  <forceDownload>false</forceDownload>
  <sharedTemplate>system/media/unversioned/pdf</sharedTemplate>
  <versionedTemplate>system/media/versioned/pdf</versionedTemplate>
  <!-- New section -->
  <thumbnails>
    <generator type="Sitecore.Resources.Media.MediaThumbnailGenerator, Sitecore.Kernel">
      <extension>png</extension>
    </generator>
    <staticFile>File location. For example /sitecore/shell/Themes/pdficon_large.png</staticFile>
  </thumbnails>
  <!-- end:New section -->
</mediaType>

Clear the following folders:

  • /temp/ folder (at least all files named “fileIcon”)
  • The Sitecore media cache (default located at /app_data/mediacache/)

Clear your browser cache.

Restart the website.

That’s it. You now have PDF icons in Sitecore.

MORE READING:

Posted in Sitecore, Sitecore 6 | Tagged , , , , | 2 Comments

Sitecore Save Dialog appears for unchanged content

Have you ever experienced that Sitecore keeps popping the “Do you want to save the changes to the item?” dialog box, even when you have made no changes to the item? And the box keeps appearing, even when you have chosen “yes”?

Save Dialog

Save Dialog

This scenario can occur if the contents of any multiline field or textbox field contains new lines \n or carriage returns \r.

Memo Box with \r or \n

Memo Box with \r or \n

In rare situations, the browser will interprete \n and \r differently from Sitecore, thus telling Sitecore that the contents in the box differs from the contents in the Sitecore database.

The situation rarely occurs with manually entered text, but if you have programatically added the text to the box, the \n and \r might misalign.

So if you add text to the multiline field (or textbox field) you should ensure that any \n and \r are removed before adding the text.

I made this string extension method to remove all new lines, carriage returns, tabs and strange \xA0’s (which looks like a space but isn’t):

public static string CleanNewLinesTabsAndOtherStuff(this string s)
{
  return s.Replace("\r\n", " ").Replace("\r", " ").Replace("\n", " ").Replace("\t", " ").Replace('\xA0', '\x20');
}

I replace all of them with spaces, but in other situations you could replace them with string.Empty.

Posted in c#, Sitecore 6 | Tagged , , , , , | 4 Comments

Sitecore Users and C#

The Sitecore security framework is based on the .NET security. Managing Authentication, Authorization, User Accounts and Roles can be done using the standard System.Web.Security namespace.

But Sitecore also provides its own Security framework that you can use for easy manipulation of users and roles seen from a Sitecore perspective.

BASICS OF SITECORE SECURITY

There is 2 things you need to know about Security in Sitecore:

  • Sitecore prefixes user names with a domain name. This is used to differentiate users between those with access to the Sitecore editor (domain: sitecore) and those with access to the Sitecore extranet (domain: extranet).
    So when accessing Sitecore users from System.Web.Security, make sure you remember to ask for sitecore\admin, and not admin.
    (Advanced Sitecore users know that you can create as many domains as you like).
  • In Sitecore there is no such thing as “not been logged in”. If you are not logged in, you will have a user called “extranet\Anonymous“.
    This means that you will always have a user, no matter the context you are running in.

ENOUGH TALK, LETS CODE

Get a user from the domain name, user name and password:

using System.Linq;
using Sitecore.Common;
using Sitecore.Security;
using Sitecore.Security.Accounts;

namespace PT.Framework.NemLogin
{
  public class UserRepository
  {
    /// <summary>
    /// Gets the <see cref="Sitecore.Security.Accounts.User"/>.
    /// </summary>
    /// <param name="domainName">Name of the domain.</param>
    /// <param name="userName">Name of the user.</param>
    /// <param name="password">The password.</param>
    /// <returns><see cref="Sitecore.Security.Accounts.User"/> if found or null if not found</returns>
    public static User GetUser(string domainName, string userName, string password)
    {
      if (!System.Web.Security.Membership.ValidateUser(domainName + @"\" + userName, password))
        return null;
      if (User.Exists(domainName + @"\" + userName))
        return User.FromName(domainName + @"\" + userName, true);
      return null;
    }
  }
}

The above function demonstrates how you can use the System.Web.Security and the Sitecore.Security namespace simultaneously. The function first validates the user using standard .NET security, then uses the Sitecore namespace to get the user.

Login:

The following function will do a login of a specified user:

using Sitecore.Security.Accounts;

using Sitecore.Security.Authentication;
using Sitecore.Web.Authentication;

public static bool Login(string domainName, string userName, string password)
{
 return AuthenticationManager.Login(domainName + @"\" + userName, password, false);
}

And this function will also do a login, but it utilizes the Sitecore TicketManager. The TicketManager manages persistent logins and is used to remember you when you log into the Sitecore backend:

public static bool Login(User user)
{
  string ticketID = TicketManager.GetCurrentTicketId();
  if (!string.IsNullOrEmpty(ticketID))
    TicketManager.RemoveTicket(ticketID);
  return AuthenticationManager.Login(user);
}

Managing Custom Properties on User Profiles:

This is an example on how to store custom data on a user profile, and later search for the user based on the value in the custom field:

#region

using System.Linq;
using Sitecore.Common;
using Sitecore.Security;
using Sitecore.Security.Accounts;

#endregion

namespace MyCode
{
  public class UserRepository
  {
    public static User GetUserFromCustomField(string fieldName, string fieldValue)
    {
      IFilterable<User> allUsers = UserManager.GetUsers();
      return allUsers.Where(user => user.Profile.GetCustomProperty(fieldName) == fieldValue).FirstOrDefault();
    }

    public static void SetCustomField(User user, string fieldName, string fieldValue)
    {
      UserProfile profile = user.Profile;
      profile.SetCustomProperty(fieldName, fieldValue);
      profile.Save();
    }
  }
}

Read more here:

 

Posted in c#, Sitecore 6 | Tagged , , , , , , , | 5 Comments

Migrating huge amounts of Sitecore content – Use the Sitecore Serializer

Migrating contents from one Sitecore to another is a common task. We move templates, layouts, content, media library items etc. from development to test and from test to production all the time.

Usually we use the Sitecore Pagkage Designer to pack contents into a package that can be installed on another Sitecore. But sometimes that’s not the best solution.

MIGRATING HUGE AMOUNTS OF DATA

Yesterday I had to move 28.000 items + 16 GB of Media Library contents from development to test.

The packager will not support this amount of contents. A Sitecore package is basically a .zip file with serialized Sitecore items. Each item is a file inside the .zip file.

I tried to create a package containing my 28.000 items, but the size of the zip file grew bigger than 2 GB which caused the package to fail.

HOW TO EASILY MIGRATE 16+GB OF DATA – THE INITIAL SETUP

Instead of creating one (or several) packages of data I create one package containing the “basics” of the project to migrate, i.e.:

  • New Templates (adding dynamically from /sitecore/templates)
  • New Layouts (adding dynamically from/sitecore/layouts)
  • New System settings (adding dynamically from /sitecore/system)
  • New …

Furthermore i add the ROOT node of the contents to migrate:

  • The root node of the Media Library folder I used to create my media library items
  • The root node of the contents to move
Adding root nodes only

Adding root nodes only

The package is now substantially smaller, about 1 Mb  which is a great size for a package. It installs fast and is easy to move around.

This package is installed on the destination Sitecore.

MOVING THE 16+GB DATA

Now I am ready to migrate the actual contents. As said before, 16 GB of data is not easily movable. Unless you use … the Sitecore tree serializer!

The Sitecore serializer can be found on the “Developer” tab. Right click the tabs and select the Developer tab:

Developer Tab

Right click to select the Developer Tab

Select the item to serialize and click the “Serialize Tree” button:

Serialize Tree

Now Sitecore is serializing the item + all sub items into separate files on your hard drive  usually in the /app_data/serialization folder:

Serialized Data

Serialized Data

It will take some time. (It took me 3 hours to generate 16 GB media library items in 12.000 individual files.)

When it’s finished, copy the files to your destination Sitecore machine, and reverse the process by selecting the root node you included in your package and clicking “Revert Tree”:

Revert Tree

Revert Tree

It will take a LOT of time if you have 16+gb of data. But don’t worry, Sitecore is creating a Sitecore Job to de-serialize the data. So you can close the browser while the process is running.

When the job is done you can remove the serialized files from the source and destination machines.

Posted in Sitecore 6 | Tagged , , , , | 6 Comments

Sitecore Links with LinkManager and MediaManager

This article describes how Sitecore handles internal and external links, how you can resolve the correct URL to an item, and what you should know about links when building multisite and multilanguage websites.

To render a corrent URL you use the LinkManager:

public string GetUrl(Sitecore.Data.Items.Item item)
{
  return Sitecore.Links.LinkManager.GetItemUrl(item);
}

If the Sitecore Item you are linking to is a Media Item, you cannot use the LinkManager as this will return the Url to the Sitecore item, not the actual media. Instead use the MediaManager:

public string GetUrl(Sitecore.Data.Items.Item item)
{
  return Sitecore.Resources.Media.MediaManager.GetMediaUrl(item);
}

If you have a Linkfield (which can link to Internal items, Media Items, external pages, anchors, email and javascript) you can use a function like this to automatically get the correct URL:

Sitecore.Data.Fields.LinkField lf = Sitecore.Context.Item.Fields["Link"];
switch (lf.LinkType.ToLower())
{
  case "internal":
    // Use LinkMananger for internal links, if link is not empty
    return lf.TargetItem != null ? Sitecore.Links.LinkManager.GetItemUrl(lf.TargetItem) : string.Empty;
  case "media":
    // Use MediaManager for media links, if link is not empty
    return lf.TargetItem != null ? Sitecore.Resources.Media.MediaManager.GetMediaUrl(lf.TargetItem) : string.Empty;
  case "external":
    // Just return external links
    return lf.Url;
  case "anchor":
    // Prefix anchor link with # if link if not empty
    return !string.IsNullOrEmpty(lf.Anchor) ? "#" + lf.Anchor : string.Empty;
  case "mailto":
    // Just return mailto link
    return lf.Url;
  case "javascript":
    // Just return javascript
    return lf.Url;
  default:
    // Just please the compiler, this
    // condition will never be met
    return lf.Url;
}

WHAT ELSE IS THERE TO KNOW?

Well, all internal links in Sitecore starts as a GUID. The GUID represents the internal item you link to. By using GUIDs internally, Sitecore allows you to move pages without worrying about links being invalid.

It is first when you render the page that the GUID is converted into a valid URL. Links in Rich Text Fields does it automatically (actually they use the ExpandLinks processor in the renderField pipeline), Link fields and lookup fields need help from you.

CAN I CONTROL HOW THE URL LOOKS?

Yes you can. You control the URLs with the linkManager settings in web.config:

<linkManager defaultProvider="sitecore">
  <providers>
    <clear />
    <add name="sitecore" type="Sitecore.Links.LinkProvider, Sitecore.Kernel" addAspxExtension="true" alwaysIncludeServerUrl="false" encodeNames="true" languageEmbedding="asNeeded" languageLocation="filePath" shortenUrls="true" useDisplayName="false" />
  </providers>
</linkManager>

Read the comments in the web.config on how to use the attributes. You can also replace the LinkProvider completely.

HOW ABOUT LINKS IN MULTISITE SOLUTIONS?

Multisite solutions are solutions where the same Sitecore installation contains more than one website, each site with its own unique domain name.

When creating multisite websites, you must remember to:

  • Set the Rendering.SiteResolving setting in web.config to true.
  • Set the targetHostName property to a unique domain for each website in the sites section in web.config.

This allows Sitecore to identify each site and add the correct domain name to links between sites.

BUT FOR NO APPARENT REASON Sitecore has chosen to render links made in the Rich Text Editor and links rendered by the LinkManager differently. Links in the Rich Text Editor use the Rendering.SiteResolving setting, while the LinkManager does not.

So you have to set it when using the LinkManager, using UrlOptions():

Sitecore.Links.UrlOptions urlOptions = new Sitecore.Links.UrlOptions();
urlOptions.SiteResolving = true;
Sitecore.Links.LinkManager.GetItemUrl(someSitecoreItem, urlOptions)

A clever developer will might do as Paul is doing in this blog post and modify the LinkProvider so the option is always set.

Sitecore have also made a free LinkProvider that does the same and more. You can get it here:

http://trac.sitecore.net/LinkProvider#

HOW ABOUT NESTED MULTISITES?

When nesting sites (a new site begins below the URL of an existing site) it becomes more tricky. Forunately Sitecore allows us to replace their LinkProvider, as my colleague Uli has done here: http://reasoncodeexample.com/2012/08/09/sitecore-cross-site-links/

HOW ABOUT MULTILANGUAGE SOLUTIONS?

Multilanguage solutions are solutions where the same site exist in several language versions, each language having a unique URL.

From Sitecore 6.4, Update 6, Sitecore introcuced to more settings called Rendering.SiteResolvingMatchCurrentSite and Rendering.SiteResolvingMatchCurrentLanguage. These settings allow URL’s to guestimate the correct domain and langauge and create the correct URL accordingly.

Read more about it here: https://briancaos.wordpress.com/2012/03/29/sitecore-cross-site-and-cross-language-links-resolved-in-6-4-1-release-120113/

Unfortunately, these settings are also only supported by the Rich Text Editor, not by the LinkManager.

MORE TO READ:

 

Posted in .net, c#, Sitecore 6 | Tagged , , , , | 4 Comments

showModalDialog returnValue is undefined in Google Chrome

For some reason, when using a Javascript showModalDialog from an ASP.NET project, Google Chrome will return an empty (“undefined”) returnValue. This error has been known by the Google Chromium team since 2010, and they have yet to fix it.

Fortunately there is a workaround.

First the modal window:

When closing the modal window it is not enough to set the window.returnValue to the specified return value. Instead, check for the window.opener and set the returnValue there as well:

<script language="javascript" type="text/javascript">
  if (window.opener) {
    window.opener.returnValue = "your return value";
  }
  window.returnValue = "your return value";
  self.close();
</script>

Then the window calling the modal window:

To receive the return value from the modal window you need to do this:

window.returnValue = undefined;
var result = window.showModalDialog("modalwindow.aspx", window, "dialogHeight:650px; dialogWidth:900px;");
  if (result == undefined)
    result = window.returnValue;
  if (result != null && result != "undefined")
    // Do something with the return value
    // defined in "result"

This has beed tested in IE9 and Google Chrome 21.0.1180.83 m. According to other sources it will work in later Firefox versions as well.

Further reading:

Posted in Uncategorized | Tagged , , , | 4 Comments

Validate CheckBoxList using a CustomValidator

In this article I explain how you can validate that at least one checkbox is checked in an asp:CheckBoxList using an asp:CustomValidator, and how to make it work server side and client side.

None of the built in .net validators will validate the asp:CheckBoxList, so you will have to create your own. If you are too lazy or in too big a hurry to create your own validation control, you can quickly whiff up a validation scenario using an asp:CustomValidator.

SETUP THE LIST AND THE VALIDATOR:

First the CheckBoxList:

<asp:CheckBoxList
  ID="cblInquiry"
  RepeatLayout="Flow"
  DataSource="<%# SomeDataSource %>"
  DataTextField="SomeField"
  DataValueField="SomeField">
  runat="server"
</asp:CheckBoxList>

Then add the CustomValidator:

<asp:CustomValidator
  OnServerValidate="valInquiry_ServerValidation"
  ID="valInquiry"
  EnableClientScript="true"
  ClientValidationFunction="verifyCheckboxList"
  ErrorMessage="Some error message"
  runat="server"
</asp:CustomValidator>

Do NOT set the ControlToValidate property on the asp:CustomValidator. If you do you will get the following error:

Validation Error

APPLY SERVER SIDE VALIDATION:

The CustomValidator has a code-behind function called valInquiry_ServerValidation. this function is quite simple:

protected void valInquiry_ServerValidation(object source, ServerValidateEventArgs args)
{
  args.IsValid = cblInquiry.SelectedItem != null;
}

The custom validator will now execute the validation on server side. Use the Page.IsValid to ensure that all validations are successfull.

APPLY CLIENT-SIDE VALIDATION:

The CustomValidator has also a client-side JavaScript referenced in the ClientValidationFunction property. This JavaScript looks like this:

<script type="text/javascript" language="javascript">
  function verifyCheckboxList(source, arguments) {
    var val = document.getElementById("<%# cblInquiry.ClientID %>");
    var col = val.getElementsByTagName("*");
    if (col != null) {
      for (i = 0; i < col.length; i++) {
        if (col.item(i).tagName == "INPUT") {
          if (col.item(i).checked) {
            arguments.IsValid = true;
            return;
          }
        }
      }
    }
    arguments.IsValid = false;
  }
</script>

Please also note that since the ControlToValidate property cannot be used on a CheckBoxList you need to hard-select the clientID of the CheckBoxList in the JavaScript.

You will need to do a DataBind() on Page_Load() in order for the <%# cblInquiry.ClientID %> to be databound.

That’s it. Happy coding.

Posted in .net, c#, General .NET | Tagged , , , , , , , , | 6 Comments

Google Maps for Sitecore

Finally the Advanced Sitecore Google Maps (WCAG Edition) for Sitecore has been upgraded to use the V3 API of Google Maps. Advanced Sitecore Google Maps is a commercial module for Sitecore that allows you to add Google Maps to your Sitecore solution and still comply to the accessibility requirements from WCAG.

Advanced Sitecore Google Maps

Advanced Sitecore Google Maps

The module works with Sitecore 6 and with the Sitecore Page Editor from Sitecore 6.4. A number of improvements have been made since the previous version. Setting up the map center and initial zoom level is still done by either entering an adress or simple by dragging and zooming on a map. But other settings such as seleting zoom control, map types, enabling street view and control positions have been moved. All the common user needs to select is the overlays, the presentation (the “view”) and the options:

Google Map Editor

Google Map Editor

One of the key features of the module is the ability to add extra visual elements to the map simply by adding user controls to “views”. The module comes with a set of views including an accessibility list. Adding this list allows you to add maps to your solution and still comply with WCAG rules by offering navigation possibilities for visually impaired.

Google Map with Accessibility list

From the developer standpoint the module has been greatly improved. The rendering of maps is now done using Sitecore pipelines. This allows you as a developer to modify the Google Map JavaScript created by the module, or to add your own Google maps functionality.

This is an example on how to add weather and cloud overlays. First I create a pipeline processor:

using PT.GoogleMaps.Core.Pipeline.RenderMap;

namespace PT.Map
{
  public class RenderWeatherLayer
  {
    public void Process(RenderMapArgs args)
    {
      args.JavaScript.AppendLine("    var weatherLayer = new google.maps.weather.WeatherLayer({");
      args.JavaScript.AppendLine("      temperatureUnits: google.maps.weather.TemperatureUnit.FAHRENHEIT");
      args.JavaScript.AppendLine("    });");
      args.JavaScript.AppendFormat("    weatherLayer.setMap({0});\n", args.RenderSettings.MapClassName);
      args.JavaScript.AppendLine("    var cloudLayer = new google.maps.weather.CloudLayer();");
      args.JavaScript.AppendFormat("    cloudLayer.setMap({0});\n", args.RenderSettings.MapClassName);
    }

  }
}

Then I add the processor to the pt_googlemaps_v3_rendermap pipeline by adding an include file to /App_Config/Include:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:x="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <setting name="PT.GoogleMaps.Libraries" value="weather" />
    </settings>
    <pipelines>
      <pt_googlemaps_v3_rendermap>
        <processor patch:after="*[@type='PT.GoogleMaps.Core.Pipeline.RenderMap.RenderEvents, PT.GoogleMaps.Core']" type="PT.Map.RenderWeatherLayer, PT.Map"/>
      </pt_googlemaps_v3_rendermap>
    </pipelines>
  </sitecore>
</configuration>

And the weather and cloud overlay is now on my map:

Google Map Weather and Cloud Map

Google Map Weather and Cloud Map

Here is a summary of the most important features:

  • Add Google Maps to any page.
  • Support for Sitecore Page Editor and Sitecore Shell.
  • The following overlays are supported out of the box: Markers, lines, polygons, rectangles, circles and images.
  • Group overlays for easy deploy to a map.
  • Add WCAG compliant maps by adding accessibility lists to the page.
  • Select between many views: Map, map with accessibility list, map with group selector, map with 3-level group selector, static image, “Find nearest” and combination of all.
  • Create your own views simply by adding user controls to the page.
  • Control map settings: Map types (roadmap, hybrid, satellite and terrain), zoom-in/zoom-out, disable double click, disable scroll, enable street view
  • Control map control positions. Specify position and appearance of overview, pan, rotate, scale and zoom controls
  • Create static maps using the Google Static Maps API.
  • Pipeline-based map rendering. Create your own layers or add functionality by adding pipeline processors.

Contact Pentia A/S for licensing information.

And check out the demo here: googlemaps.pentia.dk.

 

Posted in c#, General .NET, Sitecore 6 | Tagged , , , | 3 Comments

Unable to read data from the transport connection: The connection was closed

This is my second network error in 2 weeks. When trying to retrieve an image using a WebRequest I get the following error:

Exception: System.IO.IOException Message: Unable to read data from the transport connection: The connection was closed.
Source: System
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)

The code has been working fine for years, but suddently it fails. After some Googling I found some hints to why this might happen:

The trick that helped me was to quit using a WebRequest and use a HttpWebRequest instead. The HttpWebRequest allows me to play with 3 important settings:

The code that worked for me is:

private Byte[] RetrieveAsset(Uri uri, out string contentType)
{
  try
  {
    Byte[] bytes;
    HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create(uri);
    webRequest.KeepAlive = true;
    webRequest.ProtocolVersion = HttpVersion.Version10;
    webRequest.ServicePoint.ConnectionLimit = 24;
    webRequest.Headers.Add("UserAgent", "Pentia; MSI");
    using (WebResponse webResponse = webRequest.GetResponse())
    {
      contentType = webResponse.ContentType;
      using (Stream stream = webResponse.GetResponseStream())
      {
        using (MemoryStream memoryStream = new MemoryStream())
        {
          // Stream the response to a MemoryStream via the byte array buffer
          Byte[] buffer = new Byte[0x1000];
          Int32 bytesRead;
          while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
          {
            memoryStream.Write(buffer, 0, bytesRead);
          }
          bytes = memoryStream.ToArray();
        }
      }
    }
    return bytes;
  }
  catch (Exception ex)
  {
    throw new Exception("Failed to retrieve asset from '" + uri + "': " + ex.Message, ex);
  }
}

In this case all I needed was to set the ProtocolVersion to 1.0, but other scenarios migth require you to disable KeepAlive or limiting the number of connections.

Therefore you should set the KeepAlive, ProtocolVersion and ConnectionLimit using variable settings as different scenarios require different values (see An existing connection was forcibly closed by the remote host).

Posted in c# | Tagged , , , , , , | 3 Comments

An existing connection was forcibly closed by the remote host

This error occured randomly in my project when I tried to read images from a HttpWebRequest. The exception is:

Exception: System.IO.IOException
Message: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

Nested Exception

Exception: System.Net.Sockets.SocketException Message: An existing connection was forcibly closed by the remote host
Source: System
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

Errors like these are very hard to find, and several forums (like this and this) have solutions, but none of them worked for me.

The error occurs in the following code, where I use a HttpWebRequest to retrieve an image from another server:

private Byte[] RetrieveAsset(Uri uri, out string contentType)
{
  try
  {
    Byte[] bytes;
    HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(uri);
    webRequest.Headers.Add("UserAgent", "Pentia; MSI");
    using (WebResponse webResponse = webRequest.GetResponse())
    {
      contentType = webResponse.ContentType;
      using (Stream stream = webResponse.GetResponseStream())
      {
        using (MemoryStream memoryStream = new MemoryStream())
        {
          // Stream the response to a MemoryStream via the byte array buffer
          Byte[] buffer = new Byte[0x1000];

          Int32 bytesRead;
          while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
          {
            memoryStream.Write(buffer, 0, bytesRead);
          }
          bytes = memoryStream.ToArray();
        }
      }
    }
    return bytes;
  }
  catch (Exception ex)
  {
    throw new Exception("Failed to retrieve asset from '" + uri + "': " + ex.Message, ex);
  }
}

To solve my problem I needed 3 things, all of them in connection with the HttpWebRequest:

STEP 1: Disable KeepAlive

As Denis Pitcher describes on his blog, there is a bug in .NET that closes the connection prior to it being finished. Disabling KeepAlive closes the connection for every request:

webRequest.KeepAlive = false;

STEP 2: Set ProtocolVersion to Version10

Setting the ProtocolVersion to Version10 forces the HTTP requets to use HTTP 1.0. I can’t explain why it’s needed, but it is:

webRequest.ProtocolVersion = HttpVersion.Version10;

STEP3: Limiting the number of service points

As this MSDN article describes, The ConnectionLimit property sets the maximum number of connections that the ServicePoint object can make to an Internet resource. Apparently the server I am requesting data from is underpowered, why I have to limit my number of connections to it.

webRequest.ServicePoint.ConnectionLimit = 1;

Mircosoft states that the ConnectionLimit should be set at 12 connections per CPU (see this MSDN article, under “Performance Issues”) which in my case is 24 connections, but anything above 2 connections will force the connection to close.

SUMMARY:

The final function is this:

private Byte[] RetrieveAsset(Uri uri, out string contentType)
{
  try
  {
    Byte[] bytes;
    HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(uri);
    webRequest.KeepAlive = false;
    webRequest.ProtocolVersion = HttpVersion.Version10;
    webRequest.ServicePoint.ConnectionLimit = 1;
    webRequest.Headers.Add("UserAgent", "Pentia; MSI");
    using (WebResponse webResponse = webRequest.GetResponse())
    {
      contentType = webResponse.ContentType;
      using (Stream stream = webResponse.GetResponseStream())
      {
        using (MemoryStream memoryStream = new MemoryStream())
        {
          // Stream the response to a MemoryStream via the byte array buffer
          Byte[] buffer = new Byte[0x1000];

          Int32 bytesRead;
          while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
          {
            memoryStream.Write(buffer, 0, bytesRead);
          }
          bytes = memoryStream.ToArray();
        }
      }
    }
    return bytes;
  }
  catch (Exception ex)
  {
    throw new Exception("Failed to retrieve asset from '" + uri + "': " + ex.Message, ex);
  }
}

This might not solve your problem, but it solved mine.

ADDITIONAL RESOURCES:

Posted in .net, c#, General .NET | Tagged , , , , , | 4 Comments