Getting Started with Windows Azure Scheduler – Part 2: Storage Queue

In part 1 I discussed using Windows Azure Scheduler with an HTTP or HTTPS endpoint. In this second and final part I’ll discuss using the scheduler with a Storage Queue. The advantage of using a Storage Queue instead of HTTP(S) is that it allows for more reliable scenarios. If an HTTP call fails, the scheduler will not retry. It will just log failure and try again next time the schedule fires. With a Storage Queue, you place an item in the queue, and then get it out of the queue on the other end. If processing of the item fails, the item is not taken out of the queue.

Creating a queue

Before you can create a job, you need to have a queue. Creating a queue is a two-step process. First you need to create a storage account. You can do this from the Windows Azure Management Portal. Then you can create a queue, which you can’t do from the management portal. You have to do that in code.

Creating a storage account

To create a storage account login in the Windows Azure Management Portal and click +NEW in the bottom left corner. If you’re not already in Storage, select Data Services, followed by Storage and Quick Create. That will reveal input fields to create an account, as shown below.

Create Storage Account

Notice that besides a name you have to specify the region and whether the storage is geo redundant (default). Geo redundant storage is of course more expensive, because it gives you a higher availability and assurance that you data is safe.

Getting an Access Key

To do anything with the storage account you created you need an Access Key, basically a password to access the account. To get it you have to select the storage account and then click MANAGE KEYS at the bottom, as shown below.

Manage Access Keys

Clicking MANAGE KEYS will show you the dialog below.

Manage Access Keys Dialog

From the dialog above you need to copy one of the keys to put in your code (or better still in the configuration). The key itself of course is not enough. You need to create a connection string that the Cloud Storage library can use to access the storage account. This connection string looks like this:

DefaultEndpointsProtocol=https;AccountName=[accountname];AccountKey=[accountkey]

For example:

DefaultEndpointsProtocol=https;AccountName=michieldemo;AccountKey=ChnU8fmFvS3y9vT7wYLew0Nl6dZ7ABGw2Ne/uQ/tgPZ6yKBNbibszPxiiFt1EhVedkIQvWfijT3719J2TrYqmw==

Creating a queue
You can create queues in your storage account. You can do this the first time a queue is needed in an application, regardless if this is the sender or the receiver. In this case, the sender is Windows Azure Scheduler, can’t create a queue. You have to select an existing queue. This means you either need to create a queue with a tool or make sure the receiver is already running. In either case, you can use the C# method below to create a queue, and (if needed) return it to the caller.

public CloudQueue GetQueue(string connectionstring, string queueName)
{
    \\ Create a queue client for the give storage account
    var storageAccount = CloudStorageAccount.Parse(connectionstring);
    var queueClient = storageAccount.CreateCloudQueueClient();

    \\ Get a reference to the queue with the given name.
    var queue = queueClient.GetQueueReference(queueName);

    \\ If the queue doesn't exist, this will create it with the given name.
    queue.CreateIfNotExists();
    return queue;
}

You can run the above code from any application that has access to Windows Azure. Because you need some Windows Azure assemblies, the easiest way is to create a Windows Azure WebRole project. You can then insert the code above and call it from the startup task like this:

public override bool OnStart()
{
    var connectionstring = CloudConfigurationManager.GetSetting("StorageConnectionString");
    var queueName = "jobqueue";
    GetQueue(connectionstring, queueName);
    base.OnStart();
} 

Create a job

In part 1 I already explained how to create a job for HTTP(S). You follow the same steps for a Storage Queue, except that you select the latter when the time comes. The wizard then changes to give the details to connect to the queue, as shown below. To do this you need to have created the queue with the code shown previously, otherwise you can’t select a queue and you can’t finish the wizard. Now you can see that the wizard automatically selected the queue I created, because it’s the only queue in the storage account.

Create Storage Queue Job Dialog

In the dialog above you also need to create a SAS token. This is an access token that allows Scheduler to write to the queue. Just click the button to generate one, add some content you want to send to the target and you’re good to go.

Getting the Next Message From the Queue

Getting a message from the queue is easy. Just get a reference to the queue with the GetQueue method shown earlier, and then call GetMessage. If you received a message you can then read it as a string, as shown below.

public string GetMessage(string connectionstring, string queueName)
{
    var queue = GetQueue(connectionstring, queueName);
    var message = queue.GetMessage();
    if (message == null) return null;
    return message.AsString;
} 

You need to call the above method with a certain frequency to get messages. How quickly you need to process a job message determines how often you should call the method to see if there is a message.

What’s in the Message?

The information in the message is similar to the headers discussed in Part 1, but it is formatted as XML. Below is an example of an XML message received through a Storage Queue.

<?xml version="1.0" encoding="utf-16"?>
<StorageQueueMessage xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ExecutionTag>c3b67e748b93b0bac3718f1058e12907</ExecutionTag>
  <ClientRequestId>2fb66b67-e251-4c09-8d61-8627b8bf9bfd</ClientRequestId>
  <ExpectedExecutionTime>2014-01-13T22:32:30</ExpectedExecutionTime>
  <SchedulerJobId>DemoStorageQueueJob</SchedulerJobId>
  <SchedulerJobcollectionId>DemoCollection</SchedulerJobcollectionId>
  <Region>North Europe</Region>
  <Message>Job Queue Demo Content</Message>
</StorageQueueMessage>

Leave a Reply

Your email address will not be published. Required fields are marked *