Back To Basics: Why We Love MSMQ

Note: This post was co-authored by Chad Michel.

Clever == Complexity

I often say this because I wonder if we would be better served by simply using “old” technology and focusing on being better software designers. I really think we can accomplish this without becoming masters of the latest frameworks (although it is hard to imagine tackling today’s complex software problems without them).

If I were to look back and identify “old” technology that has made great contributions to the improvement of our software design I would have to start with Microsoft Message Queuing, or MSMQ. I love MSMQ. In fact, I think it is the coolest tool in the Windows platform. Why? Because it enables us to make our system/software design much better by reducing the amount of coupling between components and services.

MSMQ has been part of Windows operating systems since NT 4/ Windows 95, but I first used it in the early 2000s to create asynchronous, decoupled subsystems within an eCommerce platform.

Before MSMQ, submitting an online order involved:

1. Storing the order
2. Performing a fraud check
3. Processing the credit card
4. Sending a customer email
5. Submitting order data to a third-party for fulfillment

The only thing that really needed to happen synchronously was step 1. With MSMQ we went from a complex synchronous workflow to a simple and fast order placement workflow. This new workflow asynchronously spawned a variety of secondary workflows by having the primary order placement workflow post a message in the transactional queue of each of the isolated processors for steps 2-5. Since these queues were transactional we knew that each message would get processed and we could also build in retry mechanisms when failures occurred, especially with third parties like mail servers and credit card processors.

In terms we use today, we went from a tightly coupled system to a very loosely coupled, scalable set of services that were involved in processing an order. If any one of these processes (e.g., customer emails) had a tendency to get backed up in a queue, we could simply add a second processor and scale out this one part of the order system completely separate from everything else. We refer to this as having many knobs and dials to be able to turn as opposed to one big knob.

An MSMQ Example for the Greenhorn

Let’s look at a message queuing example. We have a single sender and a single listener. The sender and the listener must both listen to the same queue. We are using a queue name of “testQueue”, but this name could be almost anything. We use private queues (and not public queues) since the queuing occurs on a single machine.

The code below creates or opens an MSMQ object. When creating a new queue, you will usually make it transactional. Transactional queues allow for commit and rollback operations, which are essential for preventing data loss.

        private static MessageQueue GetQueueReference()
        {
            const string queueName = @".\private$\testqueue";
            MessageQueue queue;
            if (!MessageQueue.Exists(queueName))
                queue = MessageQueue.Create(queueName, true);
            else
                queue = new MessageQueue(queueName);
            return queue;
        }

Note the MessageQueue object. It is a disposable object. Normally when creating any IDisposable object I like to wrap it in a using statement. But in the code above we are not doing that. That is an intentional omission because we will be wrapping the result of GetQueueReference in a using statement.

To send a message using MSMQ, you must do the following:

1. Create an MessageQueueTransaction object and a MessageQueue object.
2. Begin the transaction.
3. Call MessageQueue Send.
4. Commit the transaction.
5. If something goes wrong, abort the transaction. We don’t want things left in half-finished states.

If you look at the code below a few things will stand out. First, note the using statements. Both the MessageQueueTransaction and the MessageQueue objects are enclosed in using statements. Second, we only call commit after the message has been written to the queue.

        static void SendOneMessage()
        {
            // Create a transaction because we are using a transactional queue.
            using (var trn = new MessageQueueTransaction())
            {
                try
                {
                    // Create queue object
                    using (var queue = GetQueueReference())
                    {
                        queue.Formatter = new XmlMessageFormatter();

                        // push message onto queue (inside of a transaction)
                        trn.Begin();
                        _messageNum++; // increment the message number
                        queue.Send("[Message content here]", String.Format("Message {0}",_messageNum), trn);
                        trn.Commit();

                        Console.WriteLine("Message {0} queued", _messageNum);
                    }
                }
                catch
                {
                trn.Abort(); // rollback the transaction
                }
            }
        }

Receiving from a message queue is a lot like sending. We will require a MessageQueue object. In the example below we don’t put it inside of a using statement but instead manually call dispose.

For production examples I recommend using a using statement whenever possible. The PeekCompleted event will fire when a new message enters the queue. Once we have our eventing set up, we must call BeginPeek to start the async waiting for PeekCompleted events.

            MessageQueue mq = GetQueueReference();
            mq.Formatter = new XmlMessageFormatter(new[] { typeof(String) });

            // Add an event handler for the PeekCompleted event.
            mq.PeekCompleted += OnPeekCompleted;

            // Begin the asynchronous peek operation.
            mq.BeginPeek();

            Console.WriteLine("Listening on queue");

            Console.ReadLine();

            Console.WriteLine("Closing listener...");

            // Remove the event handler before closing the queue
            mq.PeekCompleted -= OnPeekCompleted;
            mq.Close();
            mq.Dispose();

The OnPeekCompleted method will create a new MessageQueueTransaction. We will then receive the message from the queue (which will read the message and remove it from the queue). After we have successfully received the message we call commit to finalize the removal from the queue.

        private static void OnPeekCompleted(Object source, PeekCompletedEventArgs asyncResult)
        {
            // Connect to the queue.
            MessageQueue mq = (MessageQueue)source;

            // create transaction
            using (var txn = new MessageQueueTransaction())
            {
                try
                {
                    // retrieve message and process
                    txn.Begin();
                    // End the asynchronous peek operation.
                    var message = mq.Receive(txn);

                    // Display message information on the screen.
                    if (message != null)
                    {
                        Console.WriteLine("{0}: {1}", message.Label, (string)message.Body);
                    }

                    // message will be removed on txn.Commit.
                    txn.Commit();
                }
                catch (Exception ex)
                {
                    // on error don't remove message from queue
                    Console.WriteLine(ex.ToString());
                    txn.Abort();
                }
            }

            // Restart the asynchronous peek operation.
            mq.BeginPeek();
        }

The code above is available from this GitHub repository: https://github.com/dnsdurham/MessageBusPatterns. The repository also contains other message bus examples.

As I look back at the subsequent systems we have designed over the last 15 years, I can see how impactful that first experience with MSMQ was. We have consistently built applications via a set of decoupled services or subsystem that are integrated by some sort of queue. In many cases, the queue in use is still MSMQ — 20 years old and still working great.