Getting Started with Windows Azure Scheduler – Part 2: Storage Queue

In part 1 I discussed using Windows Azure Scheduler with an HTTP or HTTPS endpoint. In this second and final part I’ll discuss using the scheduler with a Storage Queue. The advantage of using a Storage Queue instead of HTTP(S) is that it allows for more reliable scenarios. If an HTTP call fails, the scheduler will not retry. It will just log failure and try again next time the schedule fires. With a Storage Queue, you place an item in the queue, and then get it out of the queue on the other end. If processing of the item fails, the item is not taken out of the queue.

Creating a queue

Before you can create a job, you need to have a queue. Creating a queue is a two-step process. First you need to create a storage account. You can do this from the Windows Azure Management Portal. Then you can create a queue, which you can’t do from the management portal. You have to do that in code.

Creating a storage account

To create a storage account login in the Windows Azure Management Portal and click +NEW in the bottom left corner. If you’re not already in Storage, select Data Services, followed by Storage and Quick Create. That will reveal input fields to create an account, as shown below.

Create Storage Account

Notice that besides a name you have to specify the region and whether the storage is geo redundant (default). Geo redundant storage is of course more expensive, because it gives you a higher availability and assurance that you data is safe.

Getting an Access Key

To do anything with the storage account you created you need an Access Key, basically a password to access the account. To get it you have to select the storage account and then click MANAGE KEYS at the bottom, as shown below.

Manage Access Keys

Clicking MANAGE KEYS will show you the dialog below.

Manage Access Keys Dialog

From the dialog above you need to copy one of the keys to put in your code (or better still in the configuration). The key itself of course is not enough. You need to create a connection string that the Cloud Storage library can use to access the storage account. This connection string looks like this:

DefaultEndpointsProtocol=https;AccountName=[accountname];AccountKey=[accountkey]

For example:

DefaultEndpointsProtocol=https;AccountName=michieldemo;AccountKey=ChnU8fmFvS3y9vT7wYLew0Nl6dZ7ABGw2Ne/uQ/tgPZ6yKBNbibszPxiiFt1EhVedkIQvWfijT3719J2TrYqmw==

Creating a queue
You can create queues in your storage account. You can do this the first time a queue is needed in an application, regardless if this is the sender or the receiver. In this case, the sender is Windows Azure Scheduler, can’t create a queue. You have to select an existing queue. This means you either need to create a queue with a tool or make sure the receiver is already running. In either case, you can use the C# method below to create a queue, and (if needed) return it to the caller.

public CloudQueue GetQueue(string connectionstring, string queueName)
{
    \\ Create a queue client for the give storage account
    var storageAccount = CloudStorageAccount.Parse(connectionstring);
    var queueClient = storageAccount.CreateCloudQueueClient();

    \\ Get a reference to the queue with the given name.
    var queue = queueClient.GetQueueReference(queueName);

    \\ If the queue doesn't exist, this will create it with the given name.
    queue.CreateIfNotExists();
    return queue;
}

You can run the above code from any application that has access to Windows Azure. Because you need some Windows Azure assemblies, the easiest way is to create a Windows Azure WebRole project. You can then insert the code above and call it from the startup task like this:

public override bool OnStart()
{
    var connectionstring = CloudConfigurationManager.GetSetting("StorageConnectionString");
    var queueName = "jobqueue";
    GetQueue(connectionstring, queueName);
    base.OnStart();
} 

Create a job

In part 1 I already explained how to create a job for HTTP(S). You follow the same steps for a Storage Queue, except that you select the latter when the time comes. The wizard then changes to give the details to connect to the queue, as shown below. To do this you need to have created the queue with the code shown previously, otherwise you can’t select a queue and you can’t finish the wizard. Now you can see that the wizard automatically selected the queue I created, because it’s the only queue in the storage account.

Create Storage Queue Job Dialog

In the dialog above you also need to create a SAS token. This is an access token that allows Scheduler to write to the queue. Just click the button to generate one, add some content you want to send to the target and you’re good to go.

Getting the Next Message From the Queue

Getting a message from the queue is easy. Just get a reference to the queue with the GetQueue method shown earlier, and then call GetMessage. If you received a message you can then read it as a string, as shown below.

public string GetMessage(string connectionstring, string queueName)
{
    var queue = GetQueue(connectionstring, queueName);
    var message = queue.GetMessage();
    if (message == null) return null;
    return message.AsString;
} 

You need to call the above method with a certain frequency to get messages. How quickly you need to process a job message determines how often you should call the method to see if there is a message.

What’s in the Message?

The information in the message is similar to the headers discussed in Part 1, but it is formatted as XML. Below is an example of an XML message received through a Storage Queue.

<?xml version="1.0" encoding="utf-16"?>
<StorageQueueMessage xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ExecutionTag>c3b67e748b93b0bac3718f1058e12907</ExecutionTag>
  <ClientRequestId>2fb66b67-e251-4c09-8d61-8627b8bf9bfd</ClientRequestId>
  <ExpectedExecutionTime>2014-01-13T22:32:30</ExpectedExecutionTime>
  <SchedulerJobId>DemoStorageQueueJob</SchedulerJobId>
  <SchedulerJobcollectionId>DemoCollection</SchedulerJobcollectionId>
  <Region>North Europe</Region>
  <Message>Job Queue Demo Content</Message>
</StorageQueueMessage>

Getting Started with Windows Azure Scheduler – Part 1: HTTP

Ever since we’ve had (web) applications we’ve had a need for tasks that are executed on a regular basis or at specific points in time. A common way to do this is through some sort of scheduler, like the Windows Task Scheduler or from some custom (Windows) service. In a web based or cloud scenario you can now also use the Windows Azure Scheduler to do this. Scheduler basically offers two options to kick off a task in an application: with an HTTP(S) call or using a Windows Azure Storage Queue. In this post I will focus on the former.

Getting Started

Right now Scheduler is in preview, so you’ll have to request it before you can use it. To do so, go to http://www.windowsazure.com/en-us/services/preview/ and click try it now and follow the process until Scheduler is enabled for you.

Creating a job

Once you can use Scheduler you can create new jobs. Just click +NEW at the bottom left of the page and select Scheduler, as shown below.

Creating a new job

When you click CUSTOM CREATE a wizard pops up to guide you through the process of creating a job. First you have to select or create a Job Collection, as shown below.

Create a job collection

A Job Collection is tied to a specific region, so if you select a region where you don’t have a collection yet, it will default to creating a new one. Next, you need to specify the job details, as shown below.

Specifying job details

You can select three Action Types: HTTP, HTTPS, and Storage Queue. Here I’ve selected HTTP, which gives you four method types: GET, POST, PUT and DELETE. Although you can use these differently, these correspond to Read, Insert, Update, and Delete in most REST based APIs. Above I’m creating a HTTP GET job. You just have to specify the URL that gets called when the job fires.

The last thing you have to do is specify the schedule. You have a choice for a one time job that fires immediately or at a specified time, or a recurring job as shown below.

Specifying the schedule

When you create a recurring job you also have the choice of starting it immediately or at a specific time. You also have to specify when the schedule ends. Above I’ve set that to the year 9999, which is effectively indefinitely.

Getting Job Information

When you’ve created your first job, you can go to the Scheduler section in the Management Portal. It will show you all collections you’ve created, in my case just the one, as shown below.

The job collections

When you click the collection you go to the dashboard, which shows you what’s been happening, as you can see below.

Job collection dashboard

For more details you can go to HISTORY, where you can select the job you want information about, and filter all the by status. You see a list of all jobs that have been executed and their result. As shown below for one of my jobs.

Job history overview

When you select one of the jobs you can click on VIEW HISTORY DETAILS to get details about the exact response you received. For a successful job that looks something like the figure below, just the full HTTP response from the server.

Succeeded job details

For a failed job it’s not much different, as shown below. Notice that the body contains more information, so if you have control over the endpoint the scheduler is calling, you can add a comprehensive error message that enables you to debug the endpoint.

Failed job details

Managing Jobs

For now, editing jobs is not possible. You can only create jobs, delete jobs, are enable/disable all jobs. You can do the latter by clicking UPDATE JOBS at the bottom of the dashboard of a Job Collection, as shown below.

Updating jobs

Scaling

There are two plans for the scheduler. The preview defaults to Standard, which allows for a maximum of 50 jobs and an interval up to a minute. The free plan allows for a maximum of 5 jobs, which can run at most every hour. You can change your plan under SCALE, as shown below.

Scaling the scheduler

What happens exactly?

So you’ve created a job, now what? If it’s a get job, basically it’s going to call the URL you specified at the interval you specified. At your endpoint you can run a page or a Web API get request method, or something similar. The request sent to the endpoint looks like this:

GET http://schedulerdemoenpoint.cloudapp.net/api/job/ HTTP/1.1
Connection: Keep-Alive
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: c912f04ea3d225912c8e9dcc82090fe3
x-ms-client-request-id: 6009d929-587c-4051-b588-0ad2f9b14f16
x-ms-scheduler-expected-execution-time: 2014-01-01T17:16:13
x-ms-scheduler-jobid: DemoGetJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: North Europe

As you can see Azure Scheduler adds several headers with information about the job. Part of it is static information about the job, but the execution tag, request id, and execution time are unique for each request.

Notice that the region is North Europe, despite the fact that I defined the Job Collection in West Europe. This is not a fluke on my part. As you can see in the different responses to the POST, PUT, and DELETE job the region is different sometimes. In fact, if you go into the management portal you will see a different region sometimes. I assume this has something to do with high-availability between data centers, and I also assume that the two data centers closest to one another are used for this.

POST

Creating a post job

POST http://schedulerdemoenpoint.cloudapp.net/api/job/ HTTP/1.1
Connection: Keep-Alive
Content-Length: 17
Content-Type: text/plain
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: 728d411206720536d592f1f2cde52e8a
x-ms-client-request-id: 134dea00-e323-4832-9aae-e847ed3884ba
x-ms-scheduler-expected-execution-time: 2014-01-01T19:21:04
x-ms-scheduler-jobid: DemoPostJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: West Europe

Demo POST content

PUT

Creating a put job

PUT http://schedulerdemoenpoint.cloudapp.net/api/job/1 HTTP/1.1
Connection: Keep-Alive
Content-Length: 16
Content-Type: text/plain
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: d62c789c2574f287af9216226d7e48a2
x-ms-client-request-id: 7003fe19-e127-4004-a9e1-1973f066155c
x-ms-scheduler-expected-execution-time: 2014-01-01T19:19:54
x-ms-scheduler-jobid: DemoPutJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: North Europe

Demo PUT Content

Delete

Creating a delete job

DELETE http://schedulerdemoenpoint.cloudapp.net/api/job/1 HTTP/1.1
Connection: Keep-Alive
Content-Length: 0
Host: schedulerdemoenpoint.cloudapp.net
x-ms-execution-tag: 5eb0e16e3eb9e880ee6edf969c376014
x-ms-client-request-id: 5d2b18e5-4e45-48f4-bf64-620393195c56
x-ms-scheduler-expected-execution-time: 2014-01-01T17:20:48
x-ms-scheduler-jobid: DemoDeleteJob
x-ms-scheduler-jobcollectionid: DemoCollection
x-ms-scheduler-execution-region: West Europe

Continue with Part 2.

ASP.NET OpenID/OAuth Login With ASP.NET 4.5 – Part 7

With ASP.NET 4.5 it is very easy to enable users to login to your site with their accounts from Facebook, Google, LinkedIn, Twitter, Yahoo, and Windows Live. In this 7 part series I’ll show you how for each of the identity providers.

Note: Out-of-the-box this only works with WebForms and MVC4. MVC3 is not supported by default.

Part 7: Logging in with Windows Live

If you want to enable users to login to your application with Windows Live, you to register an App like you have to with Facebook, LinkedIn, or Twitter. This means you first need a Windows Live account (which you get with Outlook.com among others). Follow the steps below to get your application running:

  1. Go to https://account.live.com/developers/applications/index and login if needed.
  2. If you have not created any Apps, you’re automatically redirect to the page to create an App.
  3. Give your application a name, as shown in the figure below.
    Application name dialog
  4. Click I accept. This brings up the API Settings, as shown below.
    API settings
  5. Enter the URL of the application in the Redirect domain textbox, and click Save.
  6. Open Visual Studio (if you don’t have already).
  7. Open the project created in Part 1 (or quickly create a project in the same manner).
  8. Find the App_Start folder and open AuthConfig.cs.
  9. Register the identity provider:
    1. In MVC go to the RegisterAuth method and add the following line of code:
      OAuthWebSecurity.RegisterMicrosoftClient("0000000048104C52", "Jm45Zcvj.........");
    2. In WebForms go to the RegisterOpenAuth method and add the following line of code:
      OpenAuth.AuthenticationClients.AddMicrosoft("0000000048104C52", "Jm45Zcvj.........");
  10. Save the file.
  11. Run the project.
  12. Click the Log in link. You will notice Microsoft has automatically been added next to the other providers you added under Use another service to log in.
  13. Clicking the Microsoft button will send you to Microsoft to log in.
  14. After you login, you are asked whether you want to allow the App access (see image below), and what the App may be able to see from your profile. In this case we aren’t doing anything with that information, but the App will receive a key that would allow it to get this information.

    Consent form

  15. When you click Yes, you are automatically sent back to the web application, where you will be asked to register the account as you learned in previous parts.

ASP.NET OpenID/OAuth Login With ASP.NET 4.5 – Part 6

With ASP.NET 4.5 it is very easy to enable users to login to your site with their accounts from Facebook, Google, LinkedIn, Twitter, Yahoo, and Windows Live. In this 7 part series I’ll show you how for each of the identity providers.

Note: Out-of-the-box this only works with WebForms and MVC4. MVC3 is not supported by default.

Part 6: Logging in with Yahoo

If you want to enable users to login to your application with Yahoo, you don’t have to register an App like you have to with Facebook, LinkedIn, or Twitter. All you have to do is enable Yahoo as a provider. Assuming you already have a project setup, you do this as follows:

  1. Open Visual Studio (if you don’t have already).
  2. Open the project created in Part 1 (or quickly create a project in the same manner).
  3. Find the App_Start folder and open AuthConfig.cs.
  4. Register the identity provider:
    1. In MVC go to the RegisterAuth method and add the following line of code:
      OAuthWebSecurity.RegisterYahooClient("Yahoo!");
    2. In WebForms go to the RegisterOpenAuth method and add the following line of code:
      OpenAuth.AuthenticationClients.Add("Yahoo!", () => new DotNetOpenAuth.AspNet.Clients.YahooOpenIdClient());

    Note that is both cases you have to specify a display name. This is what’s shown in the page where the user selects the identity provider.

  5. Save the file.
  6. Run the project.
  7. Click the Log in link. You will notice Yahoo has automatically been added next to the other providers you added under Use another service to log in.
  8. Clicking the Yahoo! button will send you to Yahoo to log in.
  9. Login with a Yahoo account on the page shown below.

    Yahoo! Login screen

  10. When you sign in you are asked whether you want to sign in to the application with your account. The name of the application is shown in the text (red highlight).

    Yahoo! Consent page

  11. When you agree, you are automatically sent back to the web application, where you will be asked to register the account as you learned in previous parts.

Windows Store App demo with OAuth 1.x and OAuth 2.0

There are several demo’s online that connect a Windows Store App to Facebook, Twitter, etc. using OAuth 1.x and OAuth 2.0.  Although these demo’s show how this works, the code is hard to reuse across applications, because it is tightly coupled to the main app page. I’ve completely rewritten the code to make the code reusable, and to make the OAuth 1.x and OAuth 2.0 interface almost identical, so you can use a single codebase to connect with both protocols. You can download the OAuth demo (71 KB), which includes an OAuth 1.x library you can use in the same manner as with OAuth 2.0.

Book Review – Arduino Workshop: A Hands-On Introduction with 65 Projects

Before reading Arduino Workshop: A Hands-On Introduction with 65 Projects (John Boxall, No Starch Press) I knew what Arduino was, but other than that knew nothing about it. This book for me was the perfect starting point, because it not only tells you what Arduino is, but effectively demonstrates its capabilities. Along the way you get a crash course in electronics (resistors, transistors, switches etc.) and programming. I personally didn’t need the latter, but for people with no programming experience this book will do the trick. That said, I believe the learning curve is quite steep, so you may want to look at another book for the basics of programming (preferably in a language like C#, Java or Javascript), as that comes close to the language used with Arduino). Overall this book is a good read to get started with Arduino. It is written fairly well and the projects give you a good idea of the possibilities. You can also pick stuff from the projects to use in your own. Because some projects build on previous projects, you get a sense of how to build something with Arduino.

Migrated my blog to WordPress

I’ve been wanting to migrate my blog to WordPress for quite a while, and I finally did it.

Why?
Even though dasBlog is a nice blogging engine, it is not quite as feature rich and user friendly as WordPress. There are tons of plugins and templates for WordPress that make my life as a blogger easier, and give the reader a better user experience.

I also wanted my blog to be more than just my blog, and you’ll see some additional pages appear of the course of time. This too I could have done with dasBlog, but it is much easier with WordPress.

How?
Reeves Little has an awesome post on migrating from dasBlog to WordPress, which I followed to the letter. Many thanks to him. It took me about 4 hours to do with some trial and error, mainly because of my large BlogML file.

Now what?
As I just mentioned I will add some pages to the site. This will mostly be self gratified boasting of what I’ve done as an author. I will also be tinkering with the layout, in search of the ultimate look & feel. Last but not least, I will be checking all the posts for broken stuff and to clean up the code listings which are filled with formatting no longer needed, a few posts at a time. Feel free to drop me a line of you find a post with errors.

Book Review – Disruptive Possibilities: How Big Data Changes Everything

Even though this is just a little book (just 80 pages), Disruptive Possibilities: How Big Data Changes Everything (Jeffrey Needham, O’Reilly) is a very good and insightful read. Jeffrey Needham explains very well what Big Data is and how it differs from “traditional” computing. He effectively shows you need to approach Big Data differently, because the “old school” approach to data just doesn’t scale. In that sense he echo’s my view on the subject of data: not all data needs to be normalized and transactional and you can save a lot of effort and money (on expensive hardware and software) by picking the right requirements for types of data you are dealing with. For instance, you would need a pretty good reason to store a file in a RDBMS. Because most current IT staff has been brought up with in the RDMBS paradigm, it is often the tool of choice, without thinking about it. This book effectively breaks with that way of thinking, and I would encourage developers, architects, database administrators etc. to read this book to get a sense of perspective. It would greatly help us in not making the mistake of tackling “new world” problems with “old world” solutions. The book is very easy to read, with good examples, funny stories, and insightful comments. There’s also a side step into neuro science and the future of supercomputing, which is not only good to know, but interesting in itself.

Book Review: Quiet: The Power of Introverts in a World That Can’t Stop Talking

Quiet: The Power of Introverts in a World That Can’t Stop Talking by Susan Cain is a very insightful book into introversion and extroversion. Because we deal with people at different positions on the introversion-extroversion spectrum every day, the book may open your eyes to how certain people react to you or to specific situations. IMHO this especially helps to understand children, because they are no yet adapt at hiding their true nature. So if you’re a mother, father, teacher, coach etc. this book can really help you. The book is very thorough, with a lot of personal insights backed up by scientific research. It is also pretty easy to read because it is told around experiences of the author. The book starts of with exploring extroversion and introversion and how these traits are perceived in western culture, and continues with the discussing the (dis)advantages of both traits. It the continues with giving you deeper understanding of where these traits come from and how they affect the way people think. Finally, the book gives pointer on how to deal with people at different ends of the spectrum.