Request-Response messaging via Queues, Web services and SOA technologies


The topic is an old debate in the software industry, we have been listening to SOA experts to use modern technologies like web services, SOA middleware products while from an archaic we would listen terms like queues, reliable messaging etc.

This post will try to differentiate between these concepts, their advantages and disadvantages and probably bridge the generation gap of messaging.

Today web services are widely used due to SOAP, WS Messaging and WSS Security protocol standards. They also tend to provide synchronous mode of messaging communication between the client and the server. Almost of the modern technologies and vendors can call and consume web services easily.

Although web services have an advantage of having synchronous communication and easier way to be consumed they lack asynchronous and one way messaging. I am not sure about other technologies but WCF has a one-way asynchronous communication mechanism.

However in online transactions synchronous request-response messaging is essential and usage of web services is inevitable. Due to asynchronous processing nature of queues they should be avoided. Queues do have a request-response pattern as given below but the latency, client implementation of session management and reliability management (handshakes of messages) is an overhead.

In this post we will briefly look at request-response synchronous messaging via Queues and SOA technologies. I hope the conclusion would be interesting.


Reliability, atomicity and retry submit messages in

Web services

Reliability, retrying mechanism and atomicity is not possible in web services since they tend to be synchronous and stateless in nature. Once the request is processed there is no record of the message. This can be fragile and the messages can get lost. For example if the host process is restarted or got crashed and the messages were in-flight and were being processed by the web services. The transaction in the web services will be inconsistent in this case.

Queues

There are a lot of vendors for queues (JBOSS A-MQ, active MQ, RabbitMQ, ZeroMQ etc)  but we’ll focus on MSMQ, IBM MQ Series and JMS queues request-response pattern. All of the queue technologies expose APIs where clients can access the queues and put messages in the queues for processing. This requires clients to handle session states, implement reliability mechanism by checking ACK/NAK messages etc. The advantages and disadvantages of queues are same regardless of different vendors and technologies. The queues can be transactional, allows messages to persist and retrying of message processing can be implemented.

The queues can be of two types.

1- Point-to-point: It means that each queue will be connected to only one subscriber/destination.

2- Publish-subscribe: It means that a queue can be connected to multiple subscribers but the publisher has to publish the message by topic and each subscriber gets the copy of message.

IBM MQ Series

IBM MQ Series is a system/Message Oriented middleware (MOM) in which applications can talk to each other via queues. In the scenario below SOAP messages can also be exchanged between the applications and for application A it is a request-response pattern. In the below the same queue can also be used if the applications and queues use the publish-subscribe model.

 

image

MQ Series: Request response pattern


WCF- Reliable messaging via MSMQ

WCF offers the capability of reliable messaging via WS-RM protocol. It uses reliable sessions and MSMQ to queue messages. The disadvantage is latency, session state handling from client application and queues communication which is a different implementation from a client application than just calling a SOAP web service. However it will serve the purpose of reliability (messages are persisted) and retrying. The following figure illustrates the structure of a WCF service and client using MSMQ as a transport.

 

Queued Application Diagram

WCF service and client using MSMQ as a transport

JMS Request-Response messaging via Queues

I will not go deep into JMS architecture and capabilities you can read here for more information. I am focusing on request/response synchronous pattern implementation via Queues.

In the below diagram you can see that the publisher has to put message in the queue. The advantage is reliability but it comes at the cost of latency.

A publisher sends a message to two subscribers via a
topic destination and receives replies via a queue. Figure is explained in
text.

JMS Request-Response pattern with publish/subscribe model

SOA products:

There are a number of SOA products in the market worth to mention is IBM Message broker, Oracle middleware fusion, TIBCO and Microsoft BizTalk Server. This genes of messaging system has a different set of features than the classical queuing systems like Orchestrations, Business Rules Engines, Business Activity Monitoring, Enterprise Service Bus, Dynamic Routing, Transformation and Error Recovery capabilities. They are far more capable than the classical queues.

In this post we are looking at the request-response message pattern between web services, queues and SOA products. We are concerned about how the message flows between these three set of message processing and their reliability, resilience, error recovery, retrying and atomicity. Lets take a look how message flows between SOA products in request-response pattern then we can compare with Queues and Web services. The message flow is same in all the SOA products we’ll look at the message flow in BizTalk server.

 

BizTalk Server messaging architecture

SOAP Request-Response Message flow along with an extra File message processing

In BizTalk server the client requests a SOAP message which is accepted by the receive location and then published into the Message Box. The subscribers (orchestrations or send ports) picks up the message. If there are multiple subscribers they will get a copy of the message.

If the subscriber is orchestration it will be the processing unit of the message. Orchestrations has persistence points so the state of the transaction will also be persisted to the store. When the orchestration has finished processing it will again publish the message to the database to be picked up by another subscriber. In SOAP request-response pattern it will be routed back to the receive location.

If the subscriber is a send port then the message is probably meant to be sent to another system outside the middleware server. If there is any transmission failure in the message it will be stored in the message box awaiting for actions to be taken (retry submission or suspend).

 

Conclusion

 

Reliability

In all the SOA technologies a persistence data store is present like an Oracle, SQL database or file. This guarantees reliability of the messages and messages never gets lost within the engine since it is persisted to and from memory to data store.

The data store are used as a single queue while the model is the same publish/subscribe where the publisher publishes the message to the data store and relevant subscribers pick the message for processing. In SOA products publishers are receive locations (SOAP, File) and subscribers can be orchestrations (business processing units) or Send Ports (output from the middleware system). It can be a file a destination system or even another orchestration.

Advantages over queues & web services:

The use of publish/subscribe model in analogous to the publish/subscribe model in queues. The advantage is that the clients can just call SOAP messages and the queue and session handling is abstract.

Atomicity

The SOA technologies implement transactions (Long running/Atomic) which leverages persistence capability by which transactions can recover or resume from their last state in case of a server crash, restart or operating system process failure.

Advantages over queues & web services:

The state of the orchestration is persisted in the store and the transaction state can be retrieved and appropriate actions can be taken. Web services & queues lack this feature.

Retrying/Resubmission of messages

SOA technologies can resume the transactions from the last persisted state or even the message can be retried from beginning since it is persisted in the store.

Advantages over queues & web services:

Queues can retry the message since the message when received is stored however the state of the transaction is not stored since the subscriber will be processing the message which is disjoint from the queue. Web services lack retrying and resubmission of messages.

Publishing Windows Azure Solutions from VS2012


I will publish the MVC 4 solution which we created in the previous post to the Windows Azure. You should have a Windows Azure account from http://www.windowsazure.com. Once you have got your account after entering the payment information and selecting your subscription you can see the Windows Azure portal as below.

1

To deploy our application to Windows Azure we would require the following steps.

1- Create a new Hosted Service.
2- Create a new Storage Storage Account.
3- Create and configuring management Certificates.
4- Pre-publishing steps.
5- Publishing the solution from Visual Studio 2012.
6- Changing runtime configurations after deployment (Not mandatory for publishing)

1- Create a new Hosted Service

To deploy our application containing a Web and a Worker role we would need Storage Service and Hosted Service. First we would create a Hosted Service which is to host our Web and Worker role services. Click on New Hosted Service and the below popup would open. In the dialog you can choose a subscription and enter the name of your Hosted Service. Choose a URL prefix for your service I am naming it HelloWorldAzureAbdul. For the solution I am creating and affinity group which is going to have my Hosted Service in the East Asia Data Center region.

2  3

Choose Do not deploy and click OK. You can see in the Hosted Services we have a Hosted Service called “HelloWorldAzureAbdul” which we created.

4  5

2- Create a new Storage Storage Account

To create a new storage account go to “Storage Accounts” and then on the top click “New Storage Account”. In the dialog give choose a subscription and enter the URL. I will enter the same URL helloworldazureabdul and choose the affinity group which was created in Step 1. Click OK and then you can see that the new Storage Account is created.

3

3- Create and configuring management Certificates.

To enable Visual Studio deploy the solution to my Windows Azure solutions we need to configure the management certificates so that I can enable my account to accept deployment from my machine which has the SSL certificate. In Visual Studio go to the Server Explorer (CTRL+W, L) where you can see the Windows Azure Compute and Windows Azure Storage Node which is installed by the Windows Azure SDK.

Right Click “Windows Azure Compute” and Click “Add Deployment Environment”.

6

Since I don’t have any existing deployment environment Click “Manage”. In the Authentication Settings dialog click “New” which would ask you to create a new certificate on your local machine.

7  8

Enter a friendly name of the certificate, I would name it “HelloWorldAzureAbdulCert” and then click OK. You can enter the name of these credentials I would name them “HelloWorldAzureAbdulAccount”. In the second textbox you need to enter your subscription ID which you can get from the portal.

9  15

In the above dialog click “Copy the full path” of the certificate you created. Click OK and you will be able to see the Account with the Solution Environment.

16

In the management portal go to Management Certificates and select your subscription. Then on the top click “Add Certificate”.

11

Click Browse to upload the certificate from your local PC. Since we already copied the full path of the certificate created we can just paste it and click Open.

12

Click Ok and then in the portal you will be able to see the uploaded certificate.

13  14

4- Pre-publishing steps

Before publishing the application to the cloud we need to make sure about the dependencies that are locally present on the machine but they won’t be available in the cloud. We need to check for the referenced assemblies connection strings and application configurations. If there are any third party assemblies or your local project assemblies so set the Copy Local property of the assemblies to True.

If you look at the worker and web role settings in the Windows Azure project it is having connection strings being used by diagnostics so you can uncheck the option to remove these diagnostic connection strings so that there are no dependencies.

5- Publishing from Visual Studio 2012

Now that everything is set up we can publish the solution to the cloud from VS. Right click the Windows Azure project and then click Publish. It will open the Windows Azure publish wizard in the first step Choose the subscription.

20

21

Click Next and in the Common Settings tab choose the cloud service solution, choose the staging/production environment. In the Advanced Settings tab label the deployment and select the Storage Account. Click next to finish the publishing steps and go to the summary page.

22

23

24

The deployment will start and the deployment progress can be seen in the Windows Azure Activity log window in Visual Studio.

27

After sometime you can notice in the Server Explorer that the Web and Worker role VM’s are created in the staging environment of the deployment but will be in the stopped state.

28

In the Server Explorer after some time it can be noticed that the VM’s are starting.

29

Once the deployment is completed the status will be updated to completed and the Website URL with a GUID will be created.

32

You can then go the the website URL generated and see the application. In the portal you can see that the solution is deployed.

33

34

6- Changing runtime configurations after deployment.

In the previous post we added a configuration key “Message” so that it can be changed after deployment into the cloud and re-deployment can be avoided. We can change the configuration from the portal, Select the Deployment and then click Configure in the top menu. A dialog having the configuration XML will appear in which the Message value can be changed.

36

After changing the value click OK as shown below and the changes will take effect in each instance of the web role.

37

38

You can refresh the browser where the application in the cloud was opened and see the updated message.

39

Creating my first Hello World cloud solution


I was very happy with the news that recently Windows Azure was available in my region which I was waiting for. So this would be my first cloud solution which I would deploy in the cloud. To start with I will make a simple MVC 4 application and deploy in the cloud using VS 2012.

The Windows Azure SDK for .NET can be downloaded from here for VS2012/VS2010. After installing the SDK you would get the Windows Azure tools and Emulator to develop and test cloud solutions locally. After installing the SDK you can see the cloud template under C# as shown below. You can see the cloud service by selecting framework 4.0 and then selecting “Windows Azure Cloud Service”.

1   2

In the dialog box below you can select the different roles that are required for the cloud solution. In our scenario since we are making a simple MVC 4 application so a MVC 4 web role would be enough but to do more with the solution in the future I am adding a worker role. A web role is the front end of the azure service which can be assumed that it will be on a web server (IIS) in the cloud whereas a worker role can be assumed as a Windows Service which can be used for background processing.

3   4

The web and worker roles will be renamed from the default and click OK. After that the template for MVC 4 application will appear and we will choose the internet application.

5

As you can see in the solution there are three more projects along with the MVC 4 project. One is the Unit test project for the MVC. In the MVC project you can see that most of the files are the same but with one more additional file WebRole.cs. If you open this file you can see the entry point. This is the entry point which is going to start everything before the web application starts executing.

If you go to the worker project and open the WorkerRole.cs you can see the Run method and the OnStart method. The Run method will have the implementation of your code in which you can fire some events like call WCF service, monitor for storage etc.

The third Windows Azure project you can see a Web Role and a Worker Role configuration file. If you open the HelloWorldMvcWebRole file you can see the below configurations. In this configuration files we can specify how many web role instances we want in the cloud and the size of the VM. The VM size has different billing cost so we will specify the extra small VM since we don’t need much memory and processing. I will specify two instances for high availability and load balancing the web role.

In the settings tab I will add another configuration called “Message” in which I will specify a string value “Hello Windows Azure”. The advantage of this configuration is after deploying the solution in the cloud we can change the value and it would reflect in the application at runtime. Otherwise without configuration these changes would require redeployment in the cloud. So keeping these kind of settings in the configuration is an important decision while developing Azure solutions.

7

8

If you open the ServiceConfiguration.Cloud.csfg file you can see all the settings in the XML format which we specified above in the interface.

9

Now lets open the HelloWorldWorkerRole file and you will find the same GUI and configurations. Right now we are not doing any background processing so I will keep the instance count 1 and the VM size as Extra Small.

10

Now lets change some code in the Home Controller in the MVC Web role project as in the below screen and put a break point. We are just reading the “Message” configuration key from the Windows Azure project as shown in the code below. Lets hit F5 and run the project. Make sure that the “Compute Emulator” and the “Storage Emulator” which was installed with the SDK is started and the status is in the Running State in the task bar.

13

The output can be seen on the browser and note the URL that it is running locally on the PC with the Windows Azure Emulators.

15

You can open the Windows Azure Compute Emulator from the taskbar and see the deployment. You can see the two instances of the Web Role and one instance of the Worker Role as shown below. The Emulator runs on the local PC emulating the cloud and using the local storage. You can open the local store as shown below in the screen.

17

18

BizTalk 2010 Cookbook Released


I am very happy to announce that BizTalk 2010 Cookbook written by a great MVP Steef Jan is released.

He has done a marvelous job writing this book, the contents of the book can be viewed at Packt Website.

The book is mainly for Developers and Administrators who can gain their knowledge in Development, Maintaining, Monitoring & Deployment of BizTalk Solutions. The book has step by step implementations (recipe) of BizTalk Solutions which follows industry best practices & patterns.

I had the honor to be one of the official technical reviewer of the book along with Randal van Splunteren, Sandro Pereira & Rene Brauwers

I have enjoyed reviewing the book specially the recipes related to Monitoring and deployment, of which most of the developers and administrators are unaware.

Congratulations Steef, it has been an outstanding effort from you authoring the book so well.

BizTalk2010Cookbook

BizTalk Server MVP 2012


So another year starts with a great news from Microsoft. I would like to thank Microsoft, my fellow MVPs, my MVP Lead-Ruari Plint, community members and my wife. Thank you all for your support without you people this would not be possible.

Hope this year brings the same success to me as the previous years. Happy New Year to everyone.

MVP_Horizontal_FullColor

I used to wait for the news on the new year but this time I was really busy and my family reminded me of the award renewal date. So today I just checked my inbox for the email from Microsoft.

BizTalk 2010 Web Console


I am very glad to announce the BizTalk Web Console version which is published on codeplex (http://btswebconsole.codeplex.com/) for public now.

The demand for the web version of the BizTalk Administration console has been around by the community so I decided to develop it.

Project Description
BizTalk 2010 Web Console allows administrators/operators to perform operations below from the browser.

  • Stop/Start/Restart Host Instances
  • View Status of Service Instances by applying queries to the group.
  • Stop/Start/Unenlist Orchestrations, Send Ports and Send Port Group
  • Stop/Start Receive Locations
    It works on Single/Multi-Server environment and has been tested on BizTalk Server 2009 and BizTalk Server 2010.

Demo

 

Passed BizTalk 2010 Exam (70-595)


Yesterday I passed BizTalk 2010 Exam – 70-595 (TS: Developing Business Process and Integration Solutions by Using Microsoft BizTalk Server 2010) which was released on 30 March 2011. The questions were interesting and most of the questions were quite easy and I was able to answer them from my experience. Apart from my professional projects I have learnt a lot from participating in the MSDN forums that is why the exam questions were quite familiar.

Because I read the NDA I can’t tell the level of simplicity Smile Well I prepared a lot for EDI and RFID and I was a bit better than before. For the rest of the sections I didn’t had to prepare as I could answer them easily. In my opinion anyone working for 3+ years can pass the exam.

I have passed BizTalk 2006 and BizTalk 2006 R2 previously but they were retired on 30 June 2011.

Follow

Get every new post delivered to your Inbox.

Join 140 other followers

%d bloggers like this: