Enterprise Integration Pattern Part -6- Envelope Wrapper

Sometimes batches are sent with common header information related to messages in the batch. This information is useful for routing purposes whereas the information is also useful for processing the messages. This information may or may not be useful to other client application or processes other than the process which is processing the message. Sometimes the header is preserved if the header is needed sometimes it is stripped off from the batch.

You can read more about Envelope Wrapper Integration pattern here.

Thinking and implementing in terms of BizTalk we have to consider the disassembler component properties specially the preserve header property. I have seen a lot of BizTalkers getting in trouble with the header information and they have no idea where the header goes once they specify the header schema and the document schema property in the disassembler component.

Following is the input flat file which I have used as an example. It contains two items for body schema and two items in the header schema. We will mainly focus on the header items and how to extract them separately and access them in a meaningful way.

 

First thing you notice and should implement is to set the preserve header property to true. With this the header schema value is promoted to the message context and it can be extracted in the orchestration. In the figure below you can also see two other properties HeaderValue1 and HeaderValue2 in the expression editor. These values are much more useful than just extracting the raw header schema value. The raw header value can be extracted by the XNORM.FlatFileHeaderDocument context property of the message.

 

 

 
Now to extract the fields separately from the header you have to create a property schema. In my case I have defined two header fields (HeaderField1 and HeaderField2).

 

 

 

After creating the property fields make sure to change the Property Schema Base property to MessageContextPropertyBase for each of the header fields.

 

 

Next step is to promote these fields from the header schema as shown below.

 

 

 

Now when the project is built and deployed we test the solution by placing the input file in the input folder. What we expect is the two header values in the variables which are written in the event log along with the raw header XML which you can see below.

 

 

 

 

In this way we can use the header values in a more meaningful way and even if we need we can route the message based on the header values. If you could see the output file the header items would be stripped off from the batch.

You can download this sample from the box.net widget available on the right of the page.

Enterprise integration pattern part -5- Aggregator (Sequential Convoy)

In integration it is a very common task to receive different messages from different locations process them individually and produce a result in the form of a single consolidated message. This task is performed by an aggregator. In the aggregator a Singleton orchestration with sequential or parallel convoys is implemented. There are variations in the collection of input messages i.e. it can be a parallel convoy or a sequential convoy with different or same message types.

You can read more about Aggregator Enterprise integration pattern here.

While implementing the aggregator orchestration you need to ensure that the single instance of the orchestration receives the convoy of messages. There can be two types of convoys.

Sequential Convoy

In sequential convoy all the types of messages are received in an ordered sequence. The first receive shape initializes the correlation set while the other receive shapes follow the correlation set. You can have the first receive shape’s activate property to true and set the initialize correlation property. While the other receive shapes will have the follow correlation set property set and activate property false.

Parallel Convoy

In the parallel convoy the messages arriving are not in a sequence and their sequence does not matter but the processing should be started when all the three messages are received. Configuring the parallel convoy is straight forward. Just drag the parallel shape and add the receive shapes in the parallel shape branches. If the parallel shape is the first shape in the orchestration then each parallel receive shape must have activate property to true and initialize the correlation set otherwise follow the correlation set initialized by any first receive shape or send shape.

In my example I have used a sequential convoy and used a transform shape to aggregate all the three messages into a single message and then send this message through the file port.

You can download the example from here.

Enterprise Integration Pattern Part 4 – Splitter (Debatching multiple records)

In many business processes we receive a consolidated group of messages containing multiple messages inside it and we call it a batch. It is a common task to debatch each single message from the batch and process it separately. We can then transform or route these messages for further processing.

You can read more about Splitter pattern here.

This would be really common for people working on BizTalk for some time however on the MSDN forums I still find beginners running into problems with debatching and using xpaths. I had this in mind to have a blog post explaining debatching, however I have explained debatching quite a time on the forums. Guess from now on I will be redirecting them to this post.

This is an overview of the debatching orchestration.

The orchestration works in the following steps.

1-      First we count the number of items to debatch in the original message. We assign the count into an integer variable and use a count function in the xpath. Remember to copy the xpath of the repeating record node from the message schema.

TotalOrders = System.Convert.ToInt32(xpath(OrdersRqMsg,”count(/*[local-name()=’Orders’ and namespace-uri()=’http://Splitter.OrderRq’%5D/*%5Blocal-name()=’Order’ and namespace-uri()=”])”)); 

2-      Then we extract each message and assign it in the message of type single message. You have to create a new schema that would define the single message in the batch and in the construct shape assign the Nth message from the batch to the single message. I have used the position function in the xpath and specified the index throught the loop count variable which is incrementing by 1 in the loop.

xpath_str = System.String.Format(“/*[local-name()=’Orders’ and namespace-uri()=’http://Splitter.OrderRq’%5D/*%5Blocal-name()=’Order’ and namespace-uri()=” and position()={0}]”,LoopCnt);
SingleOrderMsg = xpath(OrdersRqMsg,xpath_str);

3-      Then we can process, transform or route these single messages. I have just placed them inside a folder.

Enterprise Integration Patterns Part-3 – Resequencer

At times ordered delivery of multiple messages are required and these messages are received in a non-sequential order. This problem is solved by the resequencer enterprise integration pattern. The resequencer integration pattern implies the following.

All the incoming messages must have the BatchID, their sequence and the last message of the sequence must be determined.

All the incoming messages are stored in a correct order in a temporary cache and when all of the messages are received the messages are sent in order.

You can read more about resequencer pattern here.

In the example shown the incoming messages have a BatchID which works as a correlation set. The property is promoted to the message context and the correlation type is configured. The correlation is initialized by the first receive shape and the orchestration is started when the first message arrives. The message is saved in the SortedList of type System.Xml.XmlDocument so that we can save the message as a key value pair.

The sequence of the message is determined from the message payload and each message has the field for its sequence. To receive all the messages a listen shape with a receive which follows the correlation based on the batchID is placed and on the other branch a delay of 20 secs is configured. The loop terminates when all the messages in the message stream are received. Each message has a field of TotalMsgs which determines the total messages. A loop variable is incremented when each message arrives.

In the next loop all of the messages are returned to the send port in the right sequence.

Note:
Resequencing pattern must have a cache where the messages should be stored temporarily and when all the messages are received they must be sorted and sent in a sequence. However in this simple example we have used memory for storing messages whereas this approach is not good if the message size are very large. Another approach to store messages would be in a Database or on the Disk as a file. Storing on disk would involve efforts to retreive the files again. However storing in a database would also require connection to database, involving SQLServer/Oracle Ports and handling failure exceptions. I haven’t still found the best approach to cache messages, though I think storing the messages in the database would be best considering the performance factor.

You can download the sample here.

Enterprise Integration Patterns Part-2 – Recipient List

In the recipient list enterprise pattern a message is sent to multiple recipients in the list through different channels. The recipient list inspects the incoming message determines the list of recipients based on the incoming message and forwards the message to all the recipients through different channels. You can read more about recipient list pattern here.

Determining Recipient List

From the impletation point of view in the first round the incoming message is inspected and then depending on the data the recipient list is determined. For this we create a parent orchestration who receives an order message and if the order amount is greater that 10,000 it forwards the message to its different suppliers for quote otherwise buys it from its favourite supplier. I have also applied the filter for recipients in the child orchestration. In the parent orchestration we have a variable amount which is passed as a parameter to the child orchestration. This variable determines the recipient of the message. The value of this variable is extracted from the incoming message.

Forwarding the message to recipients through different  channels

In the first part we use the parent orchestration in which a loop is started to call the child orchestration asynchronously using a Start Orchestration shape. A self-correlating port, the request message and the filter variable is passed to the child orchestration as a parameter. You can read more about self correlating ports in my previous post about Scatter and Gather Enterprise pattern. In the second iteration all the response messages are collected back using the self correlating port. Remember to set the child orchestration long-running scope timeout so that if the orchestration fails to return the message the parent orchestration does not waits for it. Then the response messages can be processed and a response can be returned back from the orchestration.

You can download the sample from here.

Enterprise Integration Patterns Part-1 – Scatter and Gather using Self Correlating Port

Scatter and Gather Integration pattern implies that a message is sent to the Scatter-Garther Operation and the message is either debatched and sent to multiple recepients or is sent as it is depending upon the requirements. The recepients process the message request and when the processing is completed a response is sent back to Scatter-Gather where all the responses are collected from all the recepients and then processed and aggregated into a single message. You can read more about Scatter-Gather Enterprise pattern here.

There can be many approaches for building this pattern but I would suggest to follow a loosely coupled design where you can add more systems. I have used Self correlating ports where the parent orchestration calls the partner orchestrations.

Self Correlating ports – Response messsage returned by Start Orchestration shape:

Self correlating ports are used by orchestrations where it calls another orchestration by passing a port instance as a parameter to the child orchestration. The child orchestration sends the message back to this port instance. In the parent orchestration you can create a new one-way send communication port and use direct binding and select “self correlating” in the port configuration wizard. In the child orchestration drag a port shape and select use existing port type and select the self correlating port you created in the partent orchestration. Use one way communication and send in the port configuration wizard. The self correlating port is part as a port parameter to the child orchestration and the orchestration is called by Start Orchestration shape. In this way a response message is returned by Start orchestration shape.

 

The diagram below shows the design of the parent orchestration where the parent orchestration calls the partner orchestrations using Start Orchestration shape within the parallel shape. Remember that the Start Orchestration shape executes asynchronously and it does not returns a message.  So we use self correlating ports to return the message back to the parent orchestration. The advantage in this approach is that it is an asynchronous operation and the child orchestration are called concurrently and their responses are also collected independently irrespective of the order of response messages are received. With looping the parent orchestration has to wait for the first child orchestration to complete before calling the next orchestration.

 

At the end you must apply the business logic to aggregate all the responses into one consolidated response message. I have used a simple transformation map that outputs a response message. In the partner 1 orchestration I have placed a delay shape with delay of 10 seconds and the partner1 orchestration is started in the first branch of the parallel shape. Therefore the response is received after the execution and response of the other two orchestrations.

 

Partner orchestrations cannot be invoked by start orchestration shape

However if the child/partner orchestrations cannot have a self-correlating port as a parameter or you cannot start orchestrations (if the orchestration has receive shape activate property value “True”) then you have to place send shapes in place of start orchestration shapes while the rest of the design will remain the same. When sending the message you have to ensure that the child or partner orchestration is invoked and also you have to use correlation in this design. The send shape will initialize the correlation set while the receive shape will follow the correlation.

 

Web Services as partners for Scatter-Gather

If the partner is a web service then the calling and receiving of the message will be a synchronous operation therefore the send and receive shape will be in the parallel shape branches. The self correlating ports or correlation will not be required for this design.

You can download the Scatter and gather integration pattern sample from here.