Application Insights & Semantic Logging for Service Fabric Microservices

2016-08-12

Borrowing heavily from MSDN documentation, the term semantic logging refers specifically to the use of strongly typed events and consistent structure of log messages. In Service Fabric sematic logging is baked right into the platform and tooling. For example if we look at any auto generated .cs file for an actor, stateful or stateless service we see examples of logging via the ServiceEventSource or ActorEventSource classes:

ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(AcmeService).Name);

When an event such as the one above is logged, it includes a payload containing individual variables as typed values that match a pre-defined schema. Moreover as we’ll see later on in this article, when the event is routed to a suitable destination such as Application Insights, the event’s payload is written as discrete elements making it much easier to analyse, correlate and query. For those new to Application Insights the following offical introduction provides a good starting point.

Having briefly defined semantic logging and mentioning that it’s baked into Service Fabric we should clarify that ServiceEventSource and ActorEventSource inherit from EventSource, which in turn writes events to ETW. Event Tracing for Windows or more commonly ETW is an efficient kernel-level tracing facility built into Windows that logs kernel or application-defined events.

Given the above we now turn our attention to exporting these ETW events to Application Insights or for that matter to any other supported target via two libraries, the Microsoft library aptly named Semantic Logging (formerly known as the Semantic Logging Application Block or SLAB) and the SemanticLogging.ApplicationInsights library (also known as SLAB_AppInsights).

As all my Service Fabric projects are in .Net Core xproj structure (see previous articles) I ended up contributing to Fidel’s excellent library by converting the SemanticLogging.ApplicationInsights project to .Net Core xproj. My humble contribution has been merged into the master SemanticLogging.ApplicationInsights branch by Fidel and is used in the rest of the article below. As the NuGet package is somewhat behind, we’ll first start by downloading the master branch directly from GitHub and by adding it to our Visual Studio 2015 solution. Your solution will end-up looking something like this:

Semantic Logging & Application Insights

In your Service Fabric service (in my example AcmeService) edit the project.json:

{
  "title": "AcmeService",
  "description": "AcmeService",
  "version": "1.0.0-*",

  "buildOptions": {
    "emitEntryPoint": true,
    "preserveCompilationContext": true,
    "compile": {
      "exclude": [
        "PackageRoot"
      ]
    }
  },

  "dependencies": {
    "Microsoft.ServiceFabric": "5.1.150",
    "Microsoft.ServiceFabric.Services": "2.1.150",
    "EnterpriseLibrary.SemanticLogging": "2.0.1406.1",
    "SemanticLogging.ApplicationInsights": "1.0.0-*",
    "Microsoft.Extensions.PlatformAbstractions": "1.0.0",
    "Microsoft.Extensions.Configuration": "1.0.0",
    "Microsoft.Extensions.Configuration.FileExtensions": "1.0.0",
    "Microsoft.Extensions.Configuration.Json": "1.0.0",
    "Microsoft.Extensions.Configuration.Binder": "1.0.0"
  },

  "frameworks": {
    "net46": {
    }
  },

  "runtimes": {
    "win7-x64": {}
  }

}

Add an appsettings.Development.json file and make sure to set your ASPNETCORE_ENVIRONMENT variable accordingly. Moreover you will need to set the Application Insights InstrumentationKey.

{
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Information"
    }
  },
  "ApplicationInsights": {
    "InstrumentationKey": "YOUR KEY GOES HERE"
  }
}

We’ll add an AppSettings class so that we can bind our settings file to a strongly typed object:

namespace AcmeService
{
    public class AppSettings
    {
        public AppSettings()
        {
            ApplicationInsights = new ApplicationInsightsOptions();
        }

        public ApplicationInsightsOptions ApplicationInsights { get; set; }
    }

    public class ApplicationInsightsOptions
    {
        public string InstrumentationKey { get; set; }
    }
}

In a previous article we looked out how to share Asp.Net Core appsettings.json with Service Fabric Microservices so we’ll re-use the same logic and create a ConfigurationHelper:

using Microsoft.Extensions.PlatformAbstractions;
using Microsoft.Extensions.Configuration;
using System;

namespace AcmeService
{
    public static class ConfigurationHelper
    {
        public static AppSettings GetAppSettings()
        {
            var appSettings = new AppSettings();
            var configRoot = GetConfigurationRoot();
            configRoot.Bind(appSettings);

            return appSettings;
        }

        public static IConfigurationRoot GetConfigurationRoot()
        {
            IConfigurationRoot configuration = null;

            var basePath = PlatformServices.Default.Application.ApplicationBasePath;
            var environmentName = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

            if (!string.IsNullOrEmpty(environmentName))
            {
                var configurationBuilder = new ConfigurationBuilder()
                    .SetBasePath(basePath)
                    .AddJsonFile($"appsettings.{environmentName}.json");

                configuration = configurationBuilder.Build();
            }

            return configuration;
        }
    }
}

Now for the secret sauce, we create a LoggingHelper class which returns an ObservableEventListener. The class configures the Application Insights sink from the SemanticLogging.ApplicationInsights library:

listener.LogToApplicationInsights(...)

and subscribes to Service Fabric ServiceEventSource events using the Semantic Logging library:

listener.EnableEvents(ServiceEventSource.Current.Name, EventLevel.Verbose);

using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Practices.EnterpriseLibrary.SemanticLogging;
using System;
using System.Collections.Generic;
using System.Diagnostics.Tracing;

namespace AcmeService
{
    public static class LoggingHelper
    {
        public static ObservableEventListener GetEventListener()
        {
            ObservableEventListener listener = new ObservableEventListener();

            try
            {
                var appSettings = ConfigurationHelper.GetAppSettings();

                if (appSettings != null)
                {
                    TelemetryConfiguration.CreateDefault();
                    TelemetryConfiguration.Active.InstrumentationKey = appSettings.ApplicationInsights.InstrumentationKey;

                    listener.LogToApplicationInsights(TelemetryConfiguration.Active.InstrumentationKey, new List<ITelemetryInitializer>(TelemetryConfiguration.Active.TelemetryInitializers).ToArray());
                }

                listener.EnableEvents(ServiceEventSource.Current.Name, EventLevel.Verbose);
            }
            catch (Exception ex)
            {
                ServiceEventSource.Current.Message(ex.ToString());
            }

            return listener;
        }
    }
}

All that is now left is the addition of a “one liner” to your Service Fabric Microservice (Program.cs) to enable Semantic Logging:

private static readonly ObservableEventListener _listener = LoggingHelper.GetEventListener();

using Microsoft.Practices.EnterpriseLibrary.SemanticLogging;
using Microsoft.ServiceFabric.Services.Runtime;
using System;
using System.Diagnostics;
using System.Threading;

namespace AcmeService
{
    internal static class Program
    {
        private static readonly ObservableEventListener _listener = LoggingHelper.GetEventListener();

        /// <summary>
        /// This is the entry point of the service host process.
        /// </summary>
        private static void Main()
        {
            try
            {
                ServiceRuntime.RegisterServiceAsync("LoggingServiceType",
                    context => new LoggingService(context)).GetAwaiter().GetResult();

                ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(LoggingService).Name);

                // Prevents this host process from terminating so services keep running.
                Thread.Sleep(Timeout.Infinite);
            }
            catch (Exception e)
            {
                ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
                throw;
            }
        }
    }
}

And that’s about it… how simple is it to get your Service Fabric application events sent to Application Insights! Given the event producer (your Service Fabric application) is decoupled from the target through the magic of ETW and Semantic Logging libraries, the exact same approach and with minimal code changes successfully allows me to target Elastic Search as the event target. In fact for your systems you might also prefer to send some events to Application Insights and others to an Elastic Search cluster. Lastly I would like to conclude by saying if you find any of the above useful in your projects do consider contributing to Fidel’s excellent library or by creating completely new sinks for Semantic Logging!

References