Monthly Archives: January 2014

Getting Started with Windows Azure Scheduler – Part 2: Storage Queue

In part 1 I discussed using Windows Azure Scheduler with an HTTP or HTTPS endpoint. In this second and final part I’ll discuss using the scheduler with a Storage Queue. The advantage of using a Storage Queue instead of HTTP(S) is that it allows for more reliable scenarios. If an HTTP call fails, the scheduler will not retry. It will just log failure and try again next time the schedule fires. With a Storage Queue, you place an item in the queue, and then get it out of the queue on the other end. If processing of the item fails, the item is not taken out of the queue.

Creating a queue

Before you can create a job, you need to have a queue. Creating a queue is a two-step process. First you need to create a storage account. You can do this from the Windows Azure Management Portal. Then you can create a queue, which you can’t do from the management portal. You have to do that in code.

Creating a storage account

To create a storage account login in the Windows Azure Management Portal and click +NEW in the bottom left corner. If you’re not already in Storage, select Data Services, followed by Storage and Quick Create. That will reveal input fields to create an account, as shown below.

Create Storage Account

Notice that besides a name you have to specify the region and whether the storage is geo redundant (default). Geo redundant storage is of course more expensive, because it gives you a higher availability and assurance that you data is safe.

Getting an Access Key

To do anything with the storage account you created you need an Access Key, basically a password to access the account. To get it you have to select the storage account and then click MANAGE KEYS at the bottom, as shown below.

Manage Access Keys

Clicking MANAGE KEYS will show you the dialog below.

Manage Access Keys Dialog

From the dialog above you need to copy one of the keys to put in your code (or better still in the configuration). The key itself of course is not enough. You need to create a connection string that the Cloud Storage library can use to access the storage account. This connection string looks like this:

DefaultEndpointsProtocol=https;AccountName=[accountname];AccountKey=[accountkey]

For example:

DefaultEndpointsProtocol=https;AccountName=michieldemo;AccountKey=ChnU8fmFvS3y9vT7wYLew0Nl6dZ7ABGw2Ne/uQ/tgPZ6yKBNbibszPxiiFt1EhVedkIQvWfijT3719J2TrYqmw==

Creating a queue
You can create queues in your storage account. You can do this the first time a queue is needed in an application, regardless if this is the sender or the receiver. In this case, the sender is Windows Azure Scheduler, can’t create a queue. You have to select an existing queue. This means you either need to create a queue with a tool or make sure the receiver is already running. In either case, you can use the C# method below to create a queue, and (if needed) return it to the caller.

public CloudQueue GetQueue(string connectionstring, string queueName)
{
    \\ Create a queue client for the give storage account
    var storageAccount = CloudStorageAccount.Parse(connectionstring);
    var queueClient = storageAccount.CreateCloudQueueClient();

    \\ Get a reference to the queue with the given name.
    var queue = queueClient.GetQueueReference(queueName);

    \\ If the queue doesn't exist, this will create it with the given name.
    queue.CreateIfNotExists();
    return queue;
}

You can run the above code from any application that has access to Windows Azure. Because you need some Windows Azure assemblies, the easiest way is to create a Windows Azure WebRole project. You can then insert the code above and call it from the startup task like this:

public override bool OnStart()
{
    var connectionstring = CloudConfigurationManager.GetSetting("StorageConnectionString");
    var queueName = "jobqueue";
    GetQueue(connectionstring, queueName);
    base.OnStart();
} 

Create a job

In part 1 I already explained how to create a job for HTTP(S). You follow the same steps for a Storage Queue, except that you select the latter when the time comes. The wizard then changes to give the details to connect to the queue, as shown below. To do this you need to have created the queue with the code shown previously, otherwise you can’t select a queue and you can’t finish the wizard. Now you can see that the wizard automatically selected the queue I created, because it’s the only queue in the storage account.

Create Storage Queue Job Dialog

In the dialog above you also need to create a SAS token. This is an access token that allows Scheduler to write to the queue. Just click the button to generate one, add some content you want to send to the target and you’re good to go.

Getting the Next Message From the Queue

Getting a message from the queue is easy. Just get a reference to the queue with the GetQueue method shown earlier, and then call GetMessage. If you received a message you can then read it as a string, as shown below.

public string GetMessage(string connectionstring, string queueName)
{
    var queue = GetQueue(connectionstring, queueName);
    var message = queue.GetMessage();
    if (message == null) return null;
    return message.AsString;
} 

You need to call the above method with a certain frequency to get messages. How quickly you need to process a job message determines how often you should call the method to see if there is a message.

What’s in the Message?

The information in the message is similar to the headers discussed in Part 1, but it is formatted as XML. Below is an example of an XML message received through a Storage Queue.

<?xml version="1.0" encoding="utf-16"?>
<StorageQueueMessage xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ExecutionTag>c3b67e748b93b0bac3718f1058e12907</ExecutionTag>
  <ClientRequestId>2fb66b67-e251-4c09-8d61-8627b8bf9bfd</ClientRequestId>
  <ExpectedExecutionTime>2014-01-13T22:32:30</ExpectedExecutionTime>
  <SchedulerJobId>DemoStorageQueueJob</SchedulerJobId>
  <SchedulerJobcollectionId>DemoCollection</SchedulerJobcollectionId>
  <Region>North Europe</Region>
  <Message>Job Queue Demo Content</Message>
</StorageQueueMessage>

Getting Started with Windows Azure Scheduler – Part 1: HTTP

Ever since we’ve had (web) applications we’ve had a need for tasks that are executed on a regular basis or at specific points in time. A common way to do this is through some sort of scheduler, like the Windows Task Scheduler or from some custom (Windows) service. In a web based or cloud scenario you can now also use the Windows Azure Scheduler to do this. Scheduler basically offers two options to kick off a task in an application: with an HTTP(S) call or using a Windows Azure Storage Queue. In this post I will focus on the former.

Getting Started

Right now Scheduler is in preview, so you’ll have to request it before you can use it. To do so, go to http://www.windowsazure.com/en-us/services/preview/ and click try it now and follow the process until Scheduler is enabled for you.

Creating a job

Once you can use Scheduler you can create new jobs. Just click +NEW at the bottom left of the page and select Scheduler, as shown below.

Creating a new job

When you click CUSTOM CREATE a wizard pops up to guide you through the process of creating a job. First you have to select or create a Job Collection, as shown below.

Create a job collection

A Job Collection is tied to a specific region, so if you select a region where you don’t have a collection yet, it will default to creating a new one. Next, you need to specify the job details, as shown below.

Specifying job details

You can select three Action Types: HTTP, HTTPS, and Storage Queue. Here I’ve selected HTTP, which gives you four method types: GET, POST, PUT and DELETE. Although you can use these differently, these correspond to Read, Insert, Update, and Delete in most REST based APIs. Above I’m creating a HTTP GET job. You just have to specify the URL that gets called when the job fires.

The last thing you have to do is specify the schedule. You have a choice for a one time job that fires immediately or at a specified time, or a recurring job as shown below.

Specifying the schedule

When you create a recurring job you also have the choice of starting it immediately or at a specific time. You also have to specify when the schedule ends. Above I’ve set that to the year 9999, which is effectively indefinitely.

Getting Job Information

When you’ve created your first job, you can go to the Scheduler section in the Management Portal. It will show you all collections you’ve created, in my case just the one, as shown below.

The job collections

When you click the collection you go to the dashboard, which shows you what’s been happening, as you can see below.

Job collection dashboard

For more details you can go to HISTORY, where you can select the job you want information about, and filter all the by status. You see a list of all jobs that have been executed and their result. As shown below for one of my jobs.

Job history overview

When you select one of the jobs you can click on VIEW HISTORY DETAILS to get details about the exact response you received. For a successful job that looks something like the figure below, just the full HTTP response from the server.

Succeeded job details

For a failed job it’s not much different, as shown below. Notice that the body contains more information, so if you have control over the endpoint the scheduler is calling, you can add a comprehensive error message that enables you to debug the endpoint.

Failed job details

Managing Jobs

For now, editing jobs is not possible. You can only create jobs, delete jobs, are enable/disable all jobs. You can do the latter by clicking UPDATE JOBS at the bottom of the dashboard of a Job Collection, as shown below.

Updating jobs

Scaling

There are two plans for the scheduler. The preview defaults to Standard, which allows for a maximum of 50 jobs and an interval up to a minute. The free plan allows for a maximum of 5 jobs, which can run at most every hour. You can change your plan under SCALE, as shown below.

Scaling the scheduler

What happens exactly?

So you’ve created a job, now what? If it’s a get job, basically it’s going to call the URL you specified at the interval you specified. At your endpoint you can run a page or a Web API get request method, or something similar. The request sent to the endpoint looks like this:

GET http://schedulerdemoenpoint.cloudapp.net/api/job/ HTTP/1.1
Connection: Keep-Alive
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: c912f04ea3d225912c8e9dcc82090fe3
x-ms-client-request-id: 6009d929-587c-4051-b588-0ad2f9b14f16
x-ms-scheduler-expected-execution-time: 2014-01-01T17:16:13
x-ms-scheduler-jobid: DemoGetJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: North Europe

As you can see Azure Scheduler adds several headers with information about the job. Part of it is static information about the job, but the execution tag, request id, and execution time are unique for each request.

Notice that the region is North Europe, despite the fact that I defined the Job Collection in West Europe. This is not a fluke on my part. As you can see in the different responses to the POST, PUT, and DELETE job the region is different sometimes. In fact, if you go into the management portal you will see a different region sometimes. I assume this has something to do with high-availability between data centers, and I also assume that the two data centers closest to one another are used for this.

POST

Creating a post job

POST http://schedulerdemoenpoint.cloudapp.net/api/job/ HTTP/1.1
Connection: Keep-Alive
Content-Length: 17
Content-Type: text/plain
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: 728d411206720536d592f1f2cde52e8a
x-ms-client-request-id: 134dea00-e323-4832-9aae-e847ed3884ba
x-ms-scheduler-expected-execution-time: 2014-01-01T19:21:04
x-ms-scheduler-jobid: DemoPostJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: West Europe

Demo POST content

PUT

Creating a put job

PUT http://schedulerdemoenpoint.cloudapp.net/api/job/1 HTTP/1.1
Connection: Keep-Alive
Content-Length: 16
Content-Type: text/plain
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: d62c789c2574f287af9216226d7e48a2
x-ms-client-request-id: 7003fe19-e127-4004-a9e1-1973f066155c
x-ms-scheduler-expected-execution-time: 2014-01-01T19:19:54
x-ms-scheduler-jobid: DemoPutJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: North Europe

Demo PUT Content

Delete

Creating a delete job

DELETE http://schedulerdemoenpoint.cloudapp.net/api/job/1 HTTP/1.1
Connection: Keep-Alive
Content-Length: 0
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: 5eb0e16e3eb9e880ee6edf969c376014
x-ms-client-request-id: 5d2b18e5-4e45-48f4-bf64-620393195c56
x-ms-scheduler-expected-execution-time: 2014-01-01T17:20:48
x-ms-scheduler-jobid: DemoDeleteJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: West Europe

Continue with Part 2.