Using Windows Server AppFabric Caching for Storing SSO & Configuration data

Storing SSO data in cache can be very useful in a low latency scenario. Performance of services can also be improved by caching configuration data such as a Status Codes table (which has business level exception status codes and descriptions).

Using cache in SOA and BPM solutions is not new neither storing SSO data in the cache. You can read Using SSO Efficiently in the Service Oriented Solution and Business Process Management Solution.

Problems with Enterprise Library Caching:

Before AppFabric we could use Enterprise Library caching which had scalability and synchronization problems which means that it was a single server in memory cache which would not be an option with BizTalk running in a Farm and it would cause inconsistency. The other problem is in-process residing of cache in BizTalk Host instances. If the host instances are restarted it means that the cache would load again and causing the cache to populate. The other problem would be if there was a change in the source data the cache would have to be refreshed either by the time interval specified or by enforcement by restarting the host instances. Restarting the host instance can be a heavy operation for the services running and is not an option in production environment. If the cache would be refreshed after a specific time interval it is fine for a single server but in a multi-server environment it can create inconsistency of data for a specific period of time. Consider the refresh interval is 5 minutes on both servers. On Server A the cache will be refreshed after 3 minutes but on Server B it will be after 15 Seconds. In this scenario Server A will be running with the old data for 3 minutes. Windows Server AppFabric has solved this issue and now we can leverage with the features of the technology and incorporate it with BizTalk.

Windows AppFabric Cache features and advantages:

The best would be to go through this article to have an understanding of the architecture and benefits of the caching features. Here is an excerpt of features from this article.

• Caches any serializable CLR object and provides access through simple cache APIs.
• Supports enterprise scale: tens to hundreds of computers.
• Configurable to run as a service accessed over the network
• Supports dynamic scaling-out by adding new nodes.
• Backup copy provides high availability.
• Automatic load balancing.
• Integration with administration and monitoring tools such as PowerShell, Event Tracing for Windows, System Center, etc.
• Provides seamless integration with ASP.NET to be able to cache session data in without having to write it to source databases. It can also be used as a cache for application data to be able to cache application data across the entire Web farm.
• Follows the cache-aside architecture (also known as Explicit Caching) for V1. That is, you must decide explicitly which objects to put/remove in your applications and the cache does not synchronize with any source database automatically.

Let’s see now how the problems of previous caching techniques can be solved by Windows AppFabric. For this first we need to understand our requirements because there are a lot of variations of AppFabric caching host and client. You need to analyze which caching hosting options would fit into your scenario and what client you would be writing for your caching. The overview of them is given below.

Windows AppFabric Host Configurations:

On one or more servers the AppFabric Cache service will be running as a Windows service. The servers should be clustered and when using with BizTalk I would install and configure AppFabric on all of my BizTalk Server machines and configure the Cache Cluster. Of course if you have a shortage of servers like us   then you have to use the existing BizTalk servers but if you can make dedicated cache servers for large cache data you can.

In BizTalk as we will not be storing services data but only SSO data and some configuration data which we get from a Sharepoint List then configuring the existing BizTalk Server clusters for AppFabric caching is a good idea.

You have to follow the installation and configuration guide on how to make a cluster and where to store the configuration data of the cluster. The configuration data can be stored in XML file stored on a shared folder or in SQL Server database I have chosen the latter one. Here is the physical architecture diagram of a cache cluster.

1- Partitioned Cache:

I will be assuming that you are familiar with the logical hierarchy of the AppFabric cache. If the caching is configured on a cluster of servers and named cache is defined on each of the servers then the regions can be distributed among the servers and therefore providing availability or scalability.

a) Scalability:

The cache item would reside in one of the regions of the cache and that region can reside on any one of the cache cluster node. The region is guaranteed to reside on one server and the partition cannot be further distributed among the cluster. Therefore all the items residing in one region will reside in one cluster node. Defining region is optional when you add an item to the cache therefore the cache service itself will load balance the region and assign keys to the regions it has created internally on any server. There is a routing layer on the cache level which routes the put and get operations to the relevant cluster node having the key.

b) Availability:

In the availability scenario the cluster nodes can be defined as secondary and one node can be the primary node thus all the nodes having a copy of the cache items. If the primary node fails when a put or get operation is called for the cache one of the secondary nodes becomes the primary node and the applications continue accessing the cache.

It doesn’t matter on which node the get and put operations are called the routing layer of the cache determines the primary node and routes the request to it. The primary node is responsible for the synchronization of data as the data is updated in this node. When the item is accessed or updated it updates itself and then sends the operation to all the secondary nodes to update themselves. It then waits for an acknowledgment from the secondary nodes. When the acknowledgments are received from each node it then sends the acknowledgment of success of operation back to the client.

2- Local Cache:

If there is no need for availability and scalability the local cache host can be configured just as on my development machine I will have a local cache. This will have the cache in one server therefore will be fast as there will be no network hops and deserialization of data. For this when configuring the AppFabric you will have the new AppFabric cluster without having any other nodes joined with them. For making a multi-server cluster you can install AppFabric on another machine and cluster by selecting the Join Existing Cluster option in the configuration wizard and apply the appropriate settings. It still depends upon the client on which server it is accessing cache.

AppFabric Cache Clients:

There are two types of clients that can be configured in AppFabric.

1- Routing Client:

The routing client will have its own mechanism to keep track and management of the cached objects. It should know on which server the region resides and which key is placed in which region. We will not be using this in the middleware since we are just storing the SSO and Configuration data in the cache but it can be used by services or mainly by web applications depending upon the requirements.

2- Simple Client:

The Simple client is not aware about the locations of regions in the cluster. Therefore they just try to access the object in the cache in their respective region (if they are using regions). The cache routing mechanism itself takes care of routing but it depends if the cache cluster is configured for Scalability or Availability. Their routing mechanism is defined above in the host configurations section.

Implementation:

After having some basic concepts about architecture, usage and advantages I am using the AppFabric cache with the Local Cache host configuration and a base client.

For the host you have to install AppFabric and configure the AppFabric caching services. The first configuration would be to store the configuration (which is SQL Database in my case can be a shared XML File). Second configuration would be to add to an existing cluster or to create a new one. I will have another blog post for this but it’s pretty simple and one can follow the AF installation and configuration.

Now it’s time to write the client which would be BizTalk. We have a Common project which is referenced by each service for using common functionality like to read the configuration data/getting status code etc. Therefore in the same common project I will be writing the client code. I have provided the sample for download in the widget which is free from our organization helper functions so it can be used on a BizTalk machine having SSO and Caching configured.

The client can have the configurations in a configuration file or have the configurations programmatically. I will be using a configuration file and will not require recompiling when changing hosting environments. These settings can be stored into machine.config or BTSNTSvc.exe.config. I will not be explaining the configuration file as it is self explanatory with comments. Feel free to copy and modify it according to your needs as it has all the configuration sections with all the parameters.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <!--configSections must be the FIRST element -->
  <configSections>
    <!-- required to read the <dataCacheClient> element -->
    <!--
    Cache Client Setting

    1- Client time-out (milliseconds)
    The requestTimeout attribute in the dataCacheClient element. We do not recommend specifying a value less than 10000 (10 seconds). Default value is 15000.

    2- Channel open time-out (milliseconds)
    The channelOpenTimeout attribute in the dataCacheClient element. This value can be set to 0 in order to immediately handle any network problems. For more information, see Configuring Cache Client Timeouts (Windows Server AppFabric Caching). The default value is 3000.

    3- Maximum number of connections to the server
    The maxConnectionsToServer attribute in the dataCacheClient element. The default value is 1.
    -->
    <section name="dataCacheClient"
       type="Microsoft.ApplicationServer.Caching.DataCacheClientSection,
            Microsoft.ApplicationServer.Caching.Core, Version=1.0.0.0,
            Culture=neutral, PublicKeyToken=31bf3856ad364e35"
       allowLocation="true"
       allowDefinition="Everywhere"/>
  </configSections>
  <dataCacheClient>
    <!-- (optional) specify local cache Remove in a multi-server farm -->
    <!-- Local Cache Settings

    1- Local cache enabled
        The isEnabled attribute in the localCache element. Values may be true or false. The localCache element may also be missing to indicate that it is disabled.

    2- Local cache invalidation method
        The sync attribute in the localCache element. Use the TimeoutBased value to indicate a time-out value should be used. Use NotificationBased to indicate cache notifications should also be used.

    3- Local cache time-out (seconds)
        The ttlValue attribute in the localCache element.

    4- Specific cache notifications poll interval (seconds)
        (optional) Specified by the pollInterval attribute of the clientNotification element. The clientNotification element is a child of the dataCacheClient element, and not a child of the localCache element. If not specified, a value of 300 seconds will be used.

    5- Maximum locally-cached object count
        (optional) Specified by the objectCount attribute in the localCache element. Triggers when eviction on the local cache should start; it will then attempt to remove 20 percent of the least recently used locally cached objects. If not specified, the default value of 10,000 objects is used.
        The ObjectCount
    -->
    <localCache
       isEnabled="true"
       sync="TimeoutBased"
       objectCount="100000"
       ttlValue="3000" />
    <!--(optional) specify cache notifications poll interval -->
    <!-- Client Notification Settings

    1- Specific cache notifications poll interval (seconds)
        Specified by the pollInterval attribute of the clientNotification element. If not specified, a value of 300 seconds will be used.

    2- Maximum queue length
        The maxQueueLength attribute of the clientNotification element. If not specified, the default value is 10000.

    -->
    <!-- <clientNotification pollInterval="300" /> -->
    <hosts>
      <!-- Cache Host Settings

         1- Cache server name
         The name attribute of the host element.

         2- Cache port number
         The cachePort attribute of the host element.
     -->
      <host
       name="D001MWWS3"
       cachePort="22233"/>
      <!-- In a Mult-Server Environment add the second Server OR More for caching
      <host
            name="CacheServer2"
            cachePort="22233"/>
      -->
      <!-- Security Settings
      1- Mode
        The mode attribute of the securityProperties element. Possible values include Transport and None. The default value is Transport.

      2- Protection level
        The protectionLevel attribute of the securityProperties element. Possible values include None, Sign, and EncryptAndSign. The default value is EncryptAndSign.

      -->

      <!--  <securityProperties mode="Transport" protectionLevel="EncryptAndSign" /> -->

      <!-- Transport Settings
      Connection buffer size (bytes)
 The connectionBufferSize attribute of the transportProperties element.
 The ConnectionBufferSize property of the DataCacheTransportProperties class. This is then assigned to the TransportProperties property of the DataCacheFactoryConfiguration class.

Maximum buffer pool size (bytes)
 The maxBufferPoolSize attribute of the transportProperties element.
 The MaxBufferPoolSize property of the DataCacheTransportProperties class.

Maximum buffer size (bytes)
 The maxBufferSize attribute of the transportProperties element.
 The MaxBufferSize property of the DataCacheTransportProperties class.

Maximum output delay (milliseconds)
 The maxOutputDelay attribute of the transportProperties element.
 The MaxOutputDelay property of the DataCacheTransportProperties class.

Channel initialization timeout (milliseconds)
 The channelInitializationTimeout attribute of the transportProperties element.
 The ChannelInitializationTimeout property of the DataCacheTransportProperties class.

Receive timeout (milliseconds)
 The receiveTimeout attribute of the transportProperties element.
 The ReceiveTimeout property of the DataCacheTransportProperties class.
      -->
      <!-- <transportProperties connectionBufferSize="131072" maxBufferPoolSize="268435456"
                           maxBufferSize="8388608" maxOutputDelay="2" channelInitializationTimeout="60000"
                           receiveTimeout="600000"/>      -->
    </hosts>
  </dataCacheClient>
</configuration>

Get all the data from SSO:

The below code snippet of the function retrieves all the keys from all applications from SSO. I have used it in the CacheManager project where you can find it in the SSOConfigManager class.

/// <summary>

/// Returns list of applications in SSO database.

/// </summary>

/// <returns>Dictionary of application name as key and description as value.</returns>

public static IDictionary<string, string> GetApplications()

{

ISSOMapper ssoMapper = new ISSOMapper();

AffiliateApplicationType appTypes = AffiliateApplicationType.ConfigStore;

IPropertyBag propBag = (IPropertyBag)ssoMapper;

uint appFilterFlagMask = SSOFlag.SSO_FLAG_APP_FILTER_BY_TYPE;

uint appFilterFlags = (uint)appTypes;

object appFilterFlagsObj = (object)appFilterFlags;

object appFilterFlagMaskObj = (object)appFilterFlagMask;

propBag.Write("AppFilterFlags", ref appFilterFlagsObj);

propBag.Write("AppFilterFlagMask", ref appFilterFlagMaskObj);

string[] apps = null;

string[] descs = null;

string[] contacts = null;

ssoMapper.GetApplications(out apps, out descs, out contacts);

Dictionary<string, string> dict1 = new Dictionary<string, string>(apps.Length);

for (int i = 0; i < apps.Length; ++i)

{

if (!apps[i].StartsWith("{"))

dict1.Add(apps[i], descs[i]);

}

return dict1;

}

Creating and managing cache:

Before we perform operations on the cache we have to make sure the cache we are going to use is created. There are some power shell command lines for the administration of AppFabric Cache. There is also a useful GUI based tool for cache management. I would recommend downloading it in case the UAT and Production server administrator is not you. I would continue with both power shell command lines and the tools.

1- Create the cache:

There is always a Default cache which you don’t need to create. I am creating a cache named MWConfigurationCache for storing my middleware configuration data by running the New-Cache command. You can then run the Get-CacheClusterHealth command to see its health.

2- Management:

Some commands will be handy during development. For a full list refer to AppFabric Caching Deployment and Management Guide.

1-  First is the Get-CacheStatistics from which you can see how many items, regions and request are being made to the cache. You can also see the cache size in bytes.

2- The Get-CacheConfig command which gives the following output.

 

Setting	Description
CacheName	The name of the cache.
TimeToLive	The default time that items reside in the cache before expiring.
CacheType	The type of cache. This is always Partitioned.
Secondaries	A value of 1 indicates that the cache uses the high availability feature.
IsExpirable	Indicates whether objects in the cache can expire.
EvictionType	Specifies an eviction type of Least-Recently-Used (LRU) or None.
NotificationsEnabled	Indicates whether notifications are enabled for this cache.

3- You can see all the cache that exists on the cluster by Get-Cache command.

4- Stop and Start the cluster from Stop-CacheCluster and Start-CacheCluster commands respectively.

Note: Starting and Stopping the cluster clears the cache here is the sequence of commands first we can see from the stats that the cache has 3 items after stopping and starting the cache has no items. This can be useful when your source item has been updated and you want to reflect this in your cache. This would require some down time.

Inserting and retrieving Items:

There are a lot of variations in the API of the AppFabric caching. I would recommend to go through them here. In your middleware if you wish to read/write shared data between services then do consider the concurrency models. You can also have tags with keys and tags can be used to group items with your cache.

I am using the basic cache methods of Put and Get.

You can see the code below where I am getting all the key/value from all applications from SSO and adding them to cache. The Put method updates items if they already exists in cache or adds them. There is also and Add method which gives an exception if the item already exists.

public void PopulateCacheFromSSO()
        {
            IDictionary<string, string> apps = SSOConfigManager.GetApplications();

            foreach (string appName in apps.Keys)
            {
                string appUserAcct, appAdminAcct, description, contactInfo;

                HybridDictionary properties = SSOConfigManager.GetConfigProperties(appName, out description, out contactInfo, out appUserAcct, out appAdminAcct);

                System.Diagnostics.EventLog.WriteEntry("SSO Application Name", "Name = " + appName);

                foreach (DictionaryEntry appProperties in properties)
                {
                    System.Diagnostics.EventLog.WriteEntry("SSO Application enteries", "Key = " + appProperties.Key.ToString() + " , " + "Value = " + appProperties.Value.ToString());
                    PutInCache(appName, appProperties.Key.ToString(), appProperties.Value.ToString());
                }
            }
        }

public void PutInCache(string category, string key, string value)
        {
            DataCacheItemVersion itemVersion;

            if ((itemVersion = configCache.Put(category + "_" + key, value)) != null)
                System.Diagnostics.EventLog.WriteEntry("Cache Item Added", "Key = " + key);
            else
                throw new Exception("Cache Item not added");
        }

After running the code you can run the Get-CacheStatistics command to see if the items are added to the cache. Now it’s time to retrieve and item from the cache which you added. The code below gets the items from the cache.

public string GetFromCache(string category, string key)
        {
            string item;

            if ((item = (string)configCache.Get(category + "_" + key)) != null)
                System.Diagnostics.EventLog.WriteEntry("Cache Item Retrieved", "Key = " + key);
            else
                throw new Exception("Cache item could not be found");

            return item;
        }

Try to retrieve the values from the cache after the TTL time configured in the configuration file. You will find that the cache has expired. Also run the Get-CacheStatistics from power shell, see what you find.

Some troubles which I had and would be common to any developer are below.

ErrorCode<ERRCA0017>:SubStatus<ES0007>:There is a temporary failure. Please retry later. (The request failed because the server is in throttled state.)

If you get the error above it means nothing I couldn’t figure it out neither the guys on MSDN I just reset the IIS and this will go away. You will notice in the Task Manager that the w3wp process is taking too much memory.

The type or namespace name ‘ApplicationServer’ does not exist in the namespace ‘Microsoft’ (are you missing an assembly reference?)

If you are getting the error above may be you have not set the target framework 3.5/4. The second thing which I had is that I was adding references to Microsoft.ApplicationServer.Caching.Client and Microsoft.ApplicationServer.Caching.Core assemblies from C:\Windows\SysNative\AppFabric path. It simply didn’t work and the error persisted. I then added the reference from the GAC (I have no explanation for this). You can find the references in the sample.

Did the above solve my middleware problems?

I had to find a solution to the problems which I had from the enterprise library.

1- Availability and Scalability is solved by architecture of the AppFabric cache.

2- If you want to reflect changes immediately in the cache as a result of the update in the source restart the cluster services without restarting the BizTalk host instances. I mentioned that there would be a downtime it means that you don’t need to stop the host instances just stop the receive locations so that no request is entertained by BizTalk.

3- If you don’t need the downtime there can be another trick. Create a new cache of same configuration but with a different name. In the BTSNTSvc.exe.config or machine.config file I assume that you have kept the name of the cache in the appSettings section. This means that you will be retrieving it at runtime. Change it to the new cache which you created.

4- If you can wait till the cache gets expired it’s the best thing the cache will get the fresh data from the source and in the multi-server environment of BizTalk each BizTalk node will have a consistent identical copy of the cache. Great!

Security Considerations:

Without security considerations this article is incomplete and a BizTalk guy reading this cannot compromise the security over the SSO data. Of course if security is not considered with caching the SSO data can be overridden by any client who has access to the cache. As the cache will be clustered, clear text data in the network can also be sniffed.

In AppFabric cache cluster the communication between the client and the server supports Encryption and Signing.

A windows account must be added which has access to the cache cluster. This account must be used by the client application to access the cache cluster. In the BizTalk scenario we would add users such as BizTalk application Users (under which host instances run) and SSO Administrator/Affiliated administrator. This is done by Grant-CacheAllowedClientAccount command from power shell.

Cluster Security options :

After allowing access to the users you have to configure the server and client for security.

For enabling security option to the server you have to use Set-CacheClusterSecurity command from power shell.

Client Security options:

For client you can do it programmatically and in the configuration file locate the security properties tag.

<securityProperties mode=”Transport” protectionLevel=”EncryptAndSign” />

There is a table from Security Model (Windows Server AppFabric Caching) where the matrix of the combination of the cluster and client security options is given. The combination of client and cluster security options will work or not is explained in the following table. 

Client Settings	Mode=None, ProtectionLevel=Any	Mode=Transport, ProtectionLevel=None	Mode=Transport, ProtectionLevel=Sign	Mode=Transport, ProtectionLevel=EncryptAndSign
None, Any	Pass	Fail	Fail	Fail
Transport, None	Fail	Pass	Fail	Fail
Transport, Sign	Fail	Pass	Pass	Fail
Transport, EncryptAndSign	Fail	Pass	Pass	Pass