Enterprise Integration Pattern Part -6- Envelope Wrapper

Sometimes batches are sent with common header information related to messages in the batch. This information is useful for routing purposes whereas the information is also useful for processing the messages. This information may or may not be useful to other client application or processes other than the process which is processing the message. Sometimes the header is preserved if the header is needed sometimes it is stripped off from the batch.

You can read more about Envelope Wrapper Integration pattern here.

Thinking and implementing in terms of BizTalk we have to consider the disassembler component properties specially the preserve header property. I have seen a lot of BizTalkers getting in trouble with the header information and they have no idea where the header goes once they specify the header schema and the document schema property in the disassembler component.

Following is the input flat file which I have used as an example. It contains two items for body schema and two items in the header schema. We will mainly focus on the header items and how to extract them separately and access them in a meaningful way.

 

First thing you notice and should implement is to set the preserve header property to true. With this the header schema value is promoted to the message context and it can be extracted in the orchestration. In the figure below you can also see two other properties HeaderValue1 and HeaderValue2 in the expression editor. These values are much more useful than just extracting the raw header schema value. The raw header value can be extracted by the XNORM.FlatFileHeaderDocument context property of the message.

 

 

 
Now to extract the fields separately from the header you have to create a property schema. In my case I have defined two header fields (HeaderField1 and HeaderField2).

 

 

 

After creating the property fields make sure to change the Property Schema Base property to MessageContextPropertyBase for each of the header fields.

 

 

Next step is to promote these fields from the header schema as shown below.

 

 

 

Now when the project is built and deployed we test the solution by placing the input file in the input folder. What we expect is the two header values in the variables which are written in the event log along with the raw header XML which you can see below.

 

 

 

 

In this way we can use the header values in a more meaningful way and even if we need we can route the message based on the header values. If you could see the output file the header items would be stripped off from the batch.

You can download this sample from the box.net widget available on the right of the page.

Enterprise integration pattern part -5- Aggregator (Sequential Convoy)

In integration it is a very common task to receive different messages from different locations process them individually and produce a result in the form of a single consolidated message. This task is performed by an aggregator. In the aggregator a Singleton orchestration with sequential or parallel convoys is implemented. There are variations in the collection of input messages i.e. it can be a parallel convoy or a sequential convoy with different or same message types.

You can read more about Aggregator Enterprise integration pattern here.

While implementing the aggregator orchestration you need to ensure that the single instance of the orchestration receives the convoy of messages. There can be two types of convoys.

Sequential Convoy

In sequential convoy all the types of messages are received in an ordered sequence. The first receive shape initializes the correlation set while the other receive shapes follow the correlation set. You can have the first receive shape’s activate property to true and set the initialize correlation property. While the other receive shapes will have the follow correlation set property set and activate property false.

Parallel Convoy

In the parallel convoy the messages arriving are not in a sequence and their sequence does not matter but the processing should be started when all the three messages are received. Configuring the parallel convoy is straight forward. Just drag the parallel shape and add the receive shapes in the parallel shape branches. If the parallel shape is the first shape in the orchestration then each parallel receive shape must have activate property to true and initialize the correlation set otherwise follow the correlation set initialized by any first receive shape or send shape.

In my example I have used a sequential convoy and used a transform shape to aggregate all the three messages into a single message and then send this message through the file port.

You can download the example from here.

Enterprise Integration Pattern Part 4 – Splitter (Debatching multiple records)

In many business processes we receive a consolidated group of messages containing multiple messages inside it and we call it a batch. It is a common task to debatch each single message from the batch and process it separately. We can then transform or route these messages for further processing.

You can read more about Splitter pattern here.

This would be really common for people working on BizTalk for some time however on the MSDN forums I still find beginners running into problems with debatching and using xpaths. I had this in mind to have a blog post explaining debatching, however I have explained debatching quite a time on the forums. Guess from now on I will be redirecting them to this post.

This is an overview of the debatching orchestration.

The orchestration works in the following steps.

1-      First we count the number of items to debatch in the original message. We assign the count into an integer variable and use a count function in the xpath. Remember to copy the xpath of the repeating record node from the message schema.

TotalOrders = System.Convert.ToInt32(xpath(OrdersRqMsg,”count(/*[local-name()='Orders' and namespace-uri()='http://Splitter.OrderRq']/*[local-name()='Order' and namespace-uri()=''])”)); 

2-      Then we extract each message and assign it in the message of type single message. You have to create a new schema that would define the single message in the batch and in the construct shape assign the Nth message from the batch to the single message. I have used the position function in the xpath and specified the index throught the loop count variable which is incrementing by 1 in the loop.

xpath_str = System.String.Format(“/*[local-name()='Orders' and namespace-uri()='http://Splitter.OrderRq']/*[local-name()='Order' and namespace-uri()='' and position()={0}]“,LoopCnt);
SingleOrderMsg = xpath(OrdersRqMsg,xpath_str);

3-      Then we can process, transform or route these single messages. I have just placed them inside a folder.

Creating site collections

Site collections are a logical container for a hierarchy of sites which gives the scope for administrative previlages, authentication and authorization, design and restore and backup operations to the sites in the collection.

The motivation for creating site collection with respect to administrative previlages is that the sites hierarchy within the site collection will have the same administrative rights. For e.g. the IT department guy gives administrative previlages to the Business person for Business Department site collection. The business person as an administrator can create and assign roles, users and groups to access sites. Also authorize the users against the content in the sites such as documents, lists and libraries. Hence for the sales department the administrative previlages will be to the person in sales department for the sales department site collection. Therefore isolating administrative previlages among site collections.

The second motivation for creating site collections is authentication and authorization for users and groups. The users as a member of one site collection are independent of the other site collection. A user in one site collection has to do nothing with authorization on content of other site collection. Even if that user is an administrator of one site collection and has authorization over all site hierarchy content withiin its own site collection by default it will not have any previlage on sites in the other site collection.

The third motivation for creating site lists would be the restore and backup operations that can be scheduled for site collections.

The fourth motivation for creating site collections are site elements and custom queries. The WSS object model provides the means to create custom queries that span all the lists within the site collection. Users can also create various custom site elements that can be used accros the site collections.

Enterprise Integration Patterns Part-3 – Resequencer

At times ordered delivery of multiple messages are required and these messages are received in a non-sequential order. This problem is solved by the resequencer enterprise integration pattern. The resequencer integration pattern implies the following.

All the incoming messages must have the BatchID, their sequence and the last message of the sequence must be determined.

All the incoming messages are stored in a correct order in a temporary cache and when all of the messages are received the messages are sent in order.

You can read more about resequencer pattern here.

In the example shown the incoming messages have a BatchID which works as a correlation set. The property is promoted to the message context and the correlation type is configured. The correlation is initialized by the first receive shape and the orchestration is started when the first message arrives. The message is saved in the SortedList of type System.Xml.XmlDocument so that we can save the message as a key value pair.

The sequence of the message is determined from the message payload and each message has the field for its sequence. To receive all the messages a listen shape with a receive which follows the correlation based on the batchID is placed and on the other branch a delay of 20 secs is configured. The loop terminates when all the messages in the message stream are received. Each message has a field of TotalMsgs which determines the total messages. A loop variable is incremented when each message arrives.

In the next loop all of the messages are returned to the send port in the right sequence.

Note:
Resequencing pattern must have a cache where the messages should be stored temporarily and when all the messages are received they must be sorted and sent in a sequence. However in this simple example we have used memory for storing messages whereas this approach is not good if the message size are very large. Another approach to store messages would be in a Database or on the Disk as a file. Storing on disk would involve efforts to retreive the files again. However storing in a database would also require connection to database, involving SQLServer/Oracle Ports and handling failure exceptions. I haven’t still found the best approach to cache messages, though I think storing the messages in the database would be best considering the performance factor.

You can download the sample here.

Enterprise Integration Patterns Part-2 – Recipient List

In the recipient list enterprise pattern a message is sent to multiple recipients in the list through different channels. The recipient list inspects the incoming message determines the list of recipients based on the incoming message and forwards the message to all the recipients through different channels. You can read more about recipient list pattern here.

Determining Recipient List

From the impletation point of view in the first round the incoming message is inspected and then depending on the data the recipient list is determined. For this we create a parent orchestration who receives an order message and if the order amount is greater that 10,000 it forwards the message to its different suppliers for quote otherwise buys it from its favourite supplier. I have also applied the filter for recipients in the child orchestration. In the parent orchestration we have a variable amount which is passed as a parameter to the child orchestration. This variable determines the recipient of the message. The value of this variable is extracted from the incoming message.

Forwarding the message to recipients through different  channels

In the first part we use the parent orchestration in which a loop is started to call the child orchestration asynchronously using a Start Orchestration shape. A self-correlating port, the request message and the filter variable is passed to the child orchestration as a parameter. You can read more about self correlating ports in my previous post about Scatter and Gather Enterprise pattern. In the second iteration all the response messages are collected back using the self correlating port. Remember to set the child orchestration long-running scope timeout so that if the orchestration fails to return the message the parent orchestration does not waits for it. Then the response messages can be processed and a response can be returned back from the orchestration.

You can download the sample from here.

Enterprise Integration Patterns Part-1 – Scatter and Gather using Self Correlating Port

Scatter and Gather Integration pattern implies that a message is sent to the Scatter-Garther Operation and the message is either debatched and sent to multiple recepients or is sent as it is depending upon the requirements. The recepients process the message request and when the processing is completed a response is sent back to Scatter-Gather where all the responses are collected from all the recepients and then processed and aggregated into a single message. You can read more about Scatter-Gather Enterprise pattern here.

There can be many approaches for building this pattern but I would suggest to follow a loosely coupled design where you can add more systems. I have used Self correlating ports where the parent orchestration calls the partner orchestrations.

Self Correlating ports – Response messsage returned by Start Orchestration shape:

Self correlating ports are used by orchestrations where it calls another orchestration by passing a port instance as a parameter to the child orchestration. The child orchestration sends the message back to this port instance. In the parent orchestration you can create a new one-way send communication port and use direct binding and select “self correlating” in the port configuration wizard. In the child orchestration drag a port shape and select use existing port type and select the self correlating port you created in the partent orchestration. Use one way communication and send in the port configuration wizard. The self correlating port is part as a port parameter to the child orchestration and the orchestration is called by Start Orchestration shape. In this way a response message is returned by Start orchestration shape.

 

The diagram below shows the design of the parent orchestration where the parent orchestration calls the partner orchestrations using Start Orchestration shape within the parallel shape. Remember that the Start Orchestration shape executes asynchronously and it does not returns a message.  So we use self correlating ports to return the message back to the parent orchestration. The advantage in this approach is that it is an asynchronous operation and the child orchestration are called concurrently and their responses are also collected independently irrespective of the order of response messages are received. With looping the parent orchestration has to wait for the first child orchestration to complete before calling the next orchestration.

 

At the end you must apply the business logic to aggregate all the responses into one consolidated response message. I have used a simple transformation map that outputs a response message. In the partner 1 orchestration I have placed a delay shape with delay of 10 seconds and the partner1 orchestration is started in the first branch of the parallel shape. Therefore the response is received after the execution and response of the other two orchestrations.

 

Partner orchestrations cannot be invoked by start orchestration shape

However if the child/partner orchestrations cannot have a self-correlating port as a parameter or you cannot start orchestrations (if the orchestration has receive shape activate property value “True”) then you have to place send shapes in place of start orchestration shapes while the rest of the design will remain the same. When sending the message you have to ensure that the child or partner orchestration is invoked and also you have to use correlation in this design. The send shape will initialize the correlation set while the receive shape will follow the correlation.

 

Web Services as partners for Scatter-Gather

If the partner is a web service then the calling and receiving of the message will be a synchronous operation therefore the send and receive shape will be in the parallel shape branches. The self correlating ports or correlation will not be required for this design.

You can download the Scatter and gather integration pattern sample from here.

Verifying Schemas after deployment

After a developer builds and deploys the BizTalk application one verifies that the assembly of the application is present in the GAC. Sometimes when we test our solution we often get these errors below.

There was an error executing receive pipeline: Finding the document specification by
message type [MessageType] failed.Verify the schema deployed properly.

Or

There was a failure executing receive pipeline: Cannot locate document specification
because multiple schemas matched this message type [MessageType]

This error occurs when the BizTalk runtime cannot find the schema which conforms to the type of the message. To BizTalk the type of message is a combination of target namespace of the schema and the root node separated by pound (#) sign. for e.g. http://mycompany.schema#orders.

The BizTalkMgmtDb is the database where the application information is saved. The error mentioned above means that the schema is not deployed in BizTalk. The records of all the schemas that are deployed to BizTalk are in the bt_DocumentSpec table.  To check that the schema exists and is deployed properly open the query analyser on your BizTalk Server and use the BizTalkMgmtDb and run the query below. In the where clause place the correct target namespace and root node name which is being reported in the error log from the Event Viewer.

Select msgtype, assemblyid, clr_namespace, clr_assemblyname
from bt_documentspec
where msgtype = ‘http://mycompany.schema#orders’

If the record is present this verifies that the schema is deployed properly to BizTalk, if not the project containing the schema should be redeployed.

Recommended Platform Configuration for BizTalk Server

In order to ensure that the live environment is secure, stable, performing well and is highly available following are the recommendations from Microsoft which you need to consider.
1) Operating System Service Pack and Critical Updates.
To ensure that all the live environment operating systems for BizTalk and SQL Server has the latest service packs and has latest security updates. The Operating Systems must be checked weekly for latest updates in order to keep the OS environment healthy and secure. The MBSA (Microsoft Baseline Security Analyzer) tool can identify that if your system is outdated and needs to be updated. Following links would be helpful

Windows Service Pack Road Map
Microsoft Baseline Security Analyzer (MBSA)
Microsoft Support Lifecycle | Service Pack Support
Microsoft TechNet Security Center

2) COM and DTC updates
BizTalk uses COM and Microsoft Distributed Transaction Coordinator (MSDTC) and COM+ for its operations both internally and when interacting with external systems. So to benefit from these you should have their latest updates from here

3) General Network Health
To ensure that your network connectivity is good between the SQL Servers and BizTalk Servers Run the Pathping <SQL Sever Name> command on the command prompt on the BizTalk Servers. Also Find a 100 MB File and copy it on a shared Directory on each of the SQL servers from each BizTalk Servers. The result should be 0% packet loss and the file should be copied within 5 secs on a 1 Gigabit network.

PathPing Reference
How to troubleshoot network connectivity problems
How to troubleshoot TCP/IP connectivity with Windows XP

4) Avoiding TCP/IP Port Exhaustion
When the range of available TCP ports on the Windows operating systems get exhausted the Client applications or BizTalk may not be able to connect with with via TCP and the following symptoms may be observed:

• Client applications may fail to connect to the BizTalk Server.
• The BizTalk Application service may fail to connect to a remote SQL Server.
• BizTalk Server adapters may fail to connect to a remote server.
• Each port reservation that is made by a client application consumes kernel memory. If an unusually high number of client port reservations are made then Windows kernel memory use will increase accordingly.

To verify that one or more client applications are not generating excessive TCP/IP socket connections perform the following on the BizTalk servers in the group:

• From a command prompt window execute “netstat -ano -p tcp”.
• Count the number of unique Local Address TCP ports open above 1024 for each IP address.

If a large number of client applications initiate the expected number of TCP/IP socket connections but there are not enough available short-lived ports to satisfy the connection requests, then implement one or more of the registry modifications go to Avoiding TCP/IP Port Exhaustion.

If less than 3000 ephemeral ports are currently in use then its fine.

5) Avoiding DBNETLIB Exceptions
Most often the DBNetLib error is when the MessageBox database servers are busy and the biztalk runtime cycles every minute to check the database for availability. Whenever the runtime attempts to connect to the messagebox databases fails the exception below can be found in the event log.

Event Type: Warning
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5410
Computer: BIZTALKSERVER
Description:
An error occurred that requires the BizTalk service to terminate. The most common causes are the following: 1) An unexpected out of memory error. OR 2) An inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.
Error message: [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation.
Error source:
BizTalk host name: BizTalkHost
Windows service name: BTSSvc$BizTalkHost

To avoid DBNETLIB Exceptions

6) Time Synchronization
For the servers in the BizTalk group to operate correctly with their supporting SQL servers and with external systems it is important that the system time for all servers are kept synchronized with the domain. On each of the Biztalk servers in the group and SQL servers hosting the BizTalk databases, open a command prompt window and type the following command:

w32tm /stripchart /computer:<domain controller>

Where <domain controller> is the name of one of the domain controllers for the domain.

If a discrepancy is detected then clocks can be resynchronized by running the following command at a command prompt:

w32tm /resync /computer:<domain controller>

Where <domain controller> is the name of one of the domain controllers for the domain.

Windows Time Service Tools and Settings

7) NetBIOS over TCP/IP
If the BizTalk servers reside in a vulnerable network location then check to ensure any internet/public facing network adapters have NetBIOS over TCP/IP disabled. It should be disabled and should not be exposed to the public as it could be a high security risk.

Open Control Panel, Network Connections, and locate the network adapters which are internet facing. Open the properties of the network adapter(s), open the properties of TCP/IP, and on the General tab click “Advanced…”. Click the WINS tab and review the NetBIOS settings.

technorati tags : BizTalk

Microsoft announces BizTalk Server 2009

After the BizTalk Server 2006 R2 Microsoft announces BizTalk Server 2009 which will serve enterprises to build integration solutions with a more powerful Microsoft suite which is the main attraction for me to get my hands dirty with BizTalk Server 2009.
The main features which I found useful on the BizTalk Server Roadmap page on Microsoft’s site are below.

Enhanced development platform:
It supports the Windows Server 2008, VS 2008, .NET Framwork 3.5 and SQL Server 2008 which I think is a major change and will have advantages of the new advanced platform. The main feature of Windows Server 2008 is the Hyper-V virtualization which can also be a standalone product to use virtualization on their costly gigantic machines. The BizTalk Server 2009 Hyper-V guide is available for download. The ESB Guidance 2.0 is also available at Microsoft site and the ESB Guide package is available at CodePlex ESB page. With VS 2008 we have new features for debugging artifacts such as Maps, Pipeline components and Orchestrations. With SQL Server 2008 Analysis Services we can get more functionality out of BAM.

Adapters:
Unlike the release of BizTalk Server R2, BizTalk 2009 comes with two new adapters Oracle E-Business Suites and SQL Server, while there has been an enhancement in the existing adapters which I still have to explore.

Other Enhancements:
On the MS Roadmap for BizTalk Server page on Microsoft’s website you can see enhancements in B2B integration, SOA and web services, Device Connectivity and of course the enhancement in message processing by adding recoverable interchange processing support in the disassembling and validation stages in pipeline. There are also new queries for message tracking in the BizTalk Management Console.

 For more information read MS Roadmap for BizTalk Server.

codeproject
technorati tags : BizTalk