Wednesday, 29 July 2015

The Mysterious Case of the Missing Milliseconds

Strings are the lingua franca of many distributed systems, and Spotlight is no different. Earlier today, we hit a weird head-scratching bug in one of our services, and - surprise, surprise - turns out it's all do with strings. To work around limitations of an old line-of-business application, we have a database trigger (no, really) that captures changes made to a particular table, serializes them into an XML message, and pushes this into a SQL Service Broker queue; we then have a Windows service that pulls messages off the queue, parses the XML, breaks it into nicely manageable chunks and publishes them all to RabbitMQ using EasyNetQ. SImple. Except, once in a while, it blows up and starts complaining about FormatExceptions.

Now... within the database trigger, we're doing this:


which returns 2015-07-29T20:55:21.130 as you'd expect.

There's then a line of code in the Windows service that says:

var format = "yyyy-MM-ddTHH:mm:ss.fff";
DateTime.ParseExact(d, format, CultureInfo.InvariantCulture, DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal);

Now, this is the code of somebody who knows that turning datetimes into strings and back again can get a bit tricky, and so has left absolutely nothing to chance - they've supplied an exact date format, they've specified a culture, they've even gone so far as to specify the DateTimeStyles. There's unit tests and integration tests, and everything looks absolutely lovely. And then it blows up. Very occasionally,

Except... SQL Server does something weird.

SELECT @DateTime = '2015-07-29 21:59:15:123'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:15.123 (fine!)

SELECT @DateTime = '2015-07-29 21:59:15:000'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:15

SELECT @DateTime = '2015-07-29 21:59:15:999'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:16

SELECT @DateTime = '2015-07-29 21:59:15:001'
SELECT CONVERT(VARCHAR(128), @DateTime, 126) -- returns 2015-07-29T21:59:15

First, SQL Server doesn't have true millisecond precision - the milliseconds part will often get rounded by +- 0.001 seconds. Second - if the milliseconds part is zero, it'll be omitted from the string representation. Which means our incredibly specific and detailed date parsing routine will choke, because suddenly it has a date that doesn't match the format we've specified, and DateTime.ParseExact will throw a FormatException. Unit tests don't pick it up, because why would you mock such completely bizarre (and undocumented) behaviour, when you don't even know it exists?

What this means is that, since any changes done between .999 and .001 milliseconds will blow up, roughly 0.3% of all our transactions will be failing with a FormatException rather than getting synced to the rest of our systems. Which means fishing them out of the error queue and sorting them out manually - ah, the joy of distributed systems. This formatting weirdness happens on every version of SQL back as far as 2003, but there's no reference to it in the documentation until SQL Server 2012. It's been raised as a bug and closed as 'by design' because "the ISO 8601 spec leaves the conversion semantics for fractional seconds up to the implementation" - which I'm pretty sure didn't mean "go ahead and be internally inconsistent!" but as with so many other issues like this, fixing the bug would change behaviour that's been in place for years and could break things. I've no idea how - or why - anyone would build a system that genuinely relies on this bizarre idiosyncrasy, but I'll bet good money somebody out there has done it.

The beautiful irony, of course, is that if we'd used DateTime.Parse instead of ParseExact, we'd never have had a problem. :)

Friday, 26 June 2015

REST Workshop at Progressive.NET 2015 next week

I'll be delivering a hands-on workshop at Progressive.NET 2015 at SkillsMatter here in London next week, where I'll be talking about advanced REST architectural patterns, and actually implementing some of those patterns in .NET using several of the frameworks available for building HTTP/REST APIs on ASP.NET

I've tried quite hard to avoid any esoteric requirements, so attendees should only need

  • A laptop running Visual Studio 2013, Powershell and IIS
  • A reasonable working knowledge of C# and HTTP
  • A test runner capable of running NUnit tests - personally I love NCrunch deeply but ReSharper or plain old NUnit will do just fine./
  • Some familiarity with Git and GitHub - if you know how to fork a repo and clone it to your workstation, you should be fine.

The repo we'll be working from is - there shouldn't be a great deal of setup required, but if you want to clone the repository, check it compiles, and set up your local IIS endpoints by running deploy.ps1 ahead of time, it'll save a little time on the day.

During the workshop, we'll be discussing advanced HTTP/REST API patterns - hypermedia, pagination, resource expansion, HTTP PATCH, OAuth2 - and showing off some tools that can help us design and monitor our HTTP APIs. Alongside the discussion, we'll be implementing some of the techniques covered using your preferred choice of WebAPI, ServiceStack, OpenRasta or NancyFX - or even all four, if you're feeling productive - and then discussing the relative pros and cons of these frameworks for each of the patterns we're implementing.

See you there!

Friday, 19 June 2015

Slides and code from NDC Oslo 2015

I’m here at the Oslo Spektrum in Norway at NDC 2015, where I’ve been talking about the machine code of the web, SASS, TypeScript, CoffeeScript, bundle transformations, web optimisation in ASP.NET, ReST, hypermedia, resource expansion, API versioning, OAuth2, Apiary, NGrok, RunScope – and that’s just the stuff I actually managed to cover in my two talks. It’s been a really great few days, and huge thanks to the organisers for going to such great lengths to make sure everything has gone so smoothly.

A couple of non-software highlights that I really liked:

  • The catering has been excellent, and having food available throughout the day is a great way to avoid the lunchtime rush. (And the free coffee at the Twilio stand is excellent!)
  • The overflow area – where you can tune into any of the 9 talks currently in progress via a wireless headset, or just sit and channel-surf – is a great idea. (But remember it’s there if you’re doing live demos with audience participation – I’m pretty sure the “winner” of my NGrok demo was one of the people in the overflow area!)
  • If you ever get the chance to see the Norwegian band LoveShack, do it. They played the conference after-party last night, and closed their set with a note-perfect 20-minute medley which went through (I think!) Jump, Celebrate, Girls! Girls! Girls!, Welcome to the Jungle, Paradise City, the theme from Baywatch, Livin’ on a Prayer, Radio Gaga and a half-dozen more before dropping back into Jump mid-guitar-solo without skipping a beat. They’re playing the John Dee bar in Oslo this evening, and I’m almost tempted to change my flight just to stick around and see them again…

Slides, Links and Code Samples

The slides and code samples for the talks I’ve given are up on GitHub: the repo is at or if you want to download the slide decks directly, links are:

Front-End Fun with Sass and Coffee

The Rest of ReST

I also want to follow up on one specific question somebody asked after my ReST talk this morning, which can be  paraphrased as “are you comfortable recommending that people use HAL, seeing as it’s basically a dead specification?” An excellent question, and one that probably slightly more detailed answer than the one I gave on the spot. To put this in context, the HAL specification was submitted to the IETF as a draft-kelly-json-hal-06 in October 2013; that draft expired in 2014 and hasn’t been updated or ratified since, so I can see how you could argue that HAL is “dead”.

First – I’d disagree with that. Although the specification itself hasn’t changed in a while, the mailing list and community is still relatively active, and I’ve no doubt would still welcome engagement and contributions from anybody who wished to participate. Second – the spec still provides a perfectly valid approach. It’s a specification, not a tool or a framework, and in terms of delivering working software, if HAL helps you solve your problem then I say go go for it. Third – and I should have made this more obvious in this morning’s talk – HAL is just one of several approaches for delivering hypermedia over JSON. I used HAL in my examples because I think it’s the most readable, but that doesn’t mean it’s the best choice for your application. (Remember, one of my requirements for a hypermedia language in this context was “looks good on Powerpoint slides”.) If you’re interested, I would recommend also looking at JSONAPI, JSON-LD, Collection+JSON and SIREN. There is a great post by Kevin Sookocheff, which succintly summarises the difference between four of them – it doesn’t cover JsonAPI - and concludes “there is no clear winner. It depends on the constraints in place on your API”

Right. I’m going to watch Troy Hunt making hacking child’s play for an hour, and then head to the airport. Thank you, Oslo. It’s been a blast.


Friday, 12 June 2015

Restival Part 4: Deployment and Fun with Case Sensitivity

Before we dive into the next phase of API development, I wanted to make it a little easier to install and run the Restival app on your own machine, so I've added a Powershell Deploy.ps1 script to the project which will:

  • Create a new local IIS website called Restival, bound to http://restival.local/
  • Create applications for each of our API implementations
  • Configure the whole thing to run under the ASP.NET v4.0 application pool.

One interesting quirk I discovered whilst doing this is that OpenRasta appears to be case-sensitive when it comes to request URLs. I'd initially created the applications like this:

The test project uses lowercase URLs - so http://restival.local/api.nancy/ - and for some strange reason, the OpenRasta implementation just doesn't work if the IIS application name differs in case from the URL in the unit test. I'll dig into this a little further but for now, I've just modified the deploy script to do a .ToLower() on the application name and everything's working. Code for this instalment is in

Tuesday, 19 May 2015

Restival Part 2 Revisited: Attribute Routing in WebAPI

(Code for this instalment is version 0.0.3 on GitHub if you're following along.)

Mike Thomas commented on my last post, asking "any reason why you are not looking at attribute routing in WebAPI"? To which my answer is "yes - I didn't know it existed", which I'd argue is a pretty good reason why I hadn't looked at it! But Mike's absolutely right to bring it up - if we're comparing frameworks, it makes a lot of sense to really explore the full capabilities of those frameworks. So I've been reading up on attribute routing, and have to say it looks rather nice - and will, I suspect, help out with a lot of the more advanced stuff that's coming up in future instalments.

According to the documentation on

Web API 2 supports a new type of routing, called attribute routing. As the name implies, attribute routing uses attributes to define routes. Attribute routing gives you more control over the URIs in your web API. For example, you can easily create URIs that describe hierarchies of resources.

The earlier style of routing, called convention-based routing, is still fully supported. In fact, you can combine both techniques in the same project.

Sounds good, right? So what do we need to do to make Restival's WebAPI implementation run on attribute routing instead of convention-based routing?

As it turns out, not much. Well, apart from a little light yak-shaving. Attribute routing is in the Microsoft.AspNet.WebApi.WebHost package - so let's install it:

PM> Install-Package Microsoft.AspNet.WebApi.WebHost
Attempting to resolve dependency 'Microsoft.AspNet.WebApi.Core (≥ 5.2.3 && < 5.3.0)'.
Attempting to resolve dependency 'Microsoft.AspNet.WebApi.Client (≥ 5.2.3)'.


Install failed. Rolling back...
Install-Package : Could not install package 'Microsoft.AspNet.WebApi.Client 5.2.3'. You are trying to install this package
into a project that targets '.NETFramework,Version=v4.0', but the package does not contain any assembly references or content
files that are compatible with that framework. For more information, contact the package author.At line:1 char:1
+ Install-Package Microsoft.AspNet.WebApi.WebHost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Install-Package], InvalidOperationException
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

OK, no problem - we're currently targeting .NET 4.0 and it looks like WebApi.WebHost wants .NET 4.5. Right-click, properties, Target Framework to .NET Framework 4.5.1, done. Shift-Ctrl-B... what's this?


Oh. OK. Let's enable NuGet Package Restore so it'll reinstall packages when we compile the solution... oh dear:


Oh, joy. Right. This just stopped being fun, because now the solution has entered a sort of weird limbo-state where it's not restoring packages, but the option to enable package restore has disappeared. Time for a tried and tested troubleshooting routine:

  1. Close Visual Studio. Completely. SHUT IT DOWN. Yes. And the other instance you've got open. In fact, reboot the machine. DO IT.
  2. Whilst it reboots, get something to drink. Coffee if you're on the clock (did I mention Spotlight has a bean-to-cup espresso machine? We're hiring, you know...) - or something a little stronger if you're not.
  3. Put on "Turn Up the Radio" by Autograph.
  4. Take a deep breath.
  5. Re-open Visual Studio, re-open your solution, try building it again.

This time, it builds. It gives a warning about assembly version conflicts, and then settles down to 0 errors and 1 warning:

Warning: Some NuGet packages were installed using a target framework different from the current target framework and may need to be reinstalled. Visit for more information.  Packages affected: EntityFramework, Microsoft.Net.Http

Well, we're not using Entity Framework so I can just remove it. Except I can't, because Microsoft.AspNet.Providers.Core uses EntityFramework, and Microsoft.AspNet.Providers.LocalDB uses Providers.Core... but since I'm not using ANY of those, we can remove LocalDB, which removes Core, which removes EntityFramework, and we're down to a single warning about Microsoft.Net.Http, which we can fix with a NuGet package reinstall. Simple.

PM> Update-Package -reinstall Microsoft.Net.Http
Removing 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.WebApi.
Successfully removed 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.WebApi.
Removing 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.ServiceStack.
Successfully removed 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.ServiceStack.
Removing 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.OpenRasta.
Successfully removed 'Microsoft.Net.Http 2.0.20710.0' from Restival.Api.OpenRasta.
Uninstalling 'Microsoft.Net.Http 2.0.20710.0'.
Successfully uninstalled 'Microsoft.Net.Http 2.0.20710.0'.
Installing 'Microsoft.Net.Http 2.0.20710.0'.
You are downloading Microsoft.Net.Http from Microsoft, the license agreement to which is available at Check the package for additional dependencies, which may come with their own license agreement(s). Your use of the package and dependencies constitutes your acceptance of their license agreements. If you do not accept the license agreement(s), then delete the relevant components from your device.
Successfully installed 'Microsoft.Net.Http 2.0.20710.0'.
Adding 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.WebApi.
Install failed. Rolling back...
Update-Package : Unable to uninstall 'Microsoft.Net.Http 2.0.20710.0' because 'Microsoft.AspNet.WebApi.OData 4.0.30506'
depends on it.At line:1 char:1
+ Update-Package -reinstall Microsoft.Net.Http
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Update-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.UpdatePackageCommand
'Microsoft.Net.Http 2.0.20710.0' already installed.
Adding 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.ServiceStack.
Successfully added 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.ServiceStack.
'Microsoft.Net.Http 2.0.20710.0' already installed.
Adding 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.OpenRasta.
Successfully added 'Microsoft.Net.Http 2.0.20710.0' to Restival.Api.OpenRasta.


Hmm. I don't even know what WebApi.OData is, and I'm pretty sure I'm not using it... Let's remove it. Which, of course, means removing a bunch of other things, including Microsoft.Net.Http... which requires another Visual Studio restart. And this time, Update-Package fails because Microsoft.Net.Http is actually gone... so let's install it:

PM> Install-Package Microsoft.Net.Http

Zing! Done. Clean build, zero errors, zero warnings, it works. Now we can actually implement attribute routing.

First, we're going to remove our existing Hello route and enable attribute routing:

public static class WebApiConfig {
    public static void Register(HttpConfiguration config) {
        //    "Hello", // route name
        //    "hello/{name}", // route template
        //    new { Controller = "Hello", Name = "World" } // defaults
        //    );


Next, we need to decorate our HelloController with the route attribute:

public class HelloController : ApiController {
    public Greeting Get(string name) {
        return (new Greeting(name));

FInally, we need to change the way we register our configuration, because the old WebAPI 1.x convention isn't compatible with attribute routing:

protected void Application_Start() {
    // Old WebAPI 1.x syntax - not compatible with attribute routing:
    // WebApiConfig.Register(GlobalConfiguration.Configuration);

    // New WebAPI 2.x configuration via delegate instead of direct method call


That works - well, everything works except our default "Hello, World" scenario - so let's add a default value to the {name} parameter in our route attribute:

public class HelloController : ApiController {
    public Greeting Get(string name) {
        return (new Greeting(name));

And there you go. Attribute routing works, all tests are passing - and a yak so impeccably shaved you could use it to sell cologne. Not bad.

Thursday, 7 May 2015

Restival Part 3: Hello, World!

So far, we've got a /hello service up and running in four different frameworks (five if you include Andrew 'kolektiv' Cherry's F#/Freya implementation, which looks really interesting). Last time, we looked at how our frameworks handle routing; in this instalment, we're going to look at adding default values to our route parameters. Specifically, if you just call /hello, the service should return "Hello, World!"

GET /hello

should return

200 OK

{ "Message" : "Hello, World!" }

OK, so how we do this? There's three places we can potentially do it - the routing, the handlers, or the underlying Greeting object itself - but not all our frameworks support all three options.

Only WebAPI supports explicit defaults in the routing configuration, using this syntax:

    "Hello", // route name
    "hello/{name}", // route template
    new { Controller = "Hello", Name = "World" } // defaults

ServiceStack doesn't support routing defaults per se, but because the request is decoupled from the rest of our implementation, we can specify the default inside our request DTO - and thanks to the single responsibility principle, we know this isn't going to affect any other details of our implementation

public class Hello {
    private string name = "World";
    public string Name {
        get { return name; }
        set { name = value; }

OpenRasta's routing maps our requests directly onto the Get method of our HelloHandler, which makes it really easy to use .NET's optional parameter syntax to supply a default value to the handler method itself:

public class HelloHandler {
    public object Get(string name = "World") {
        return (new Greeting(name));

Finally, NancyFX has such sparse routing that there's only really one line of code we could modify - after all, our entire API's only two lines of code, and the other one's doing our /hello/{name} implementation.

public class HelloModule : NancyModule {
    public HelloModule() {
        Get["/hello"] = _ => new Greeting("World");
        Get["/hello/{name}"] = parameters => new Greeting(;

Astute readers will be going "but hang on - you've just defined the same default in four different places! That's not very SOLID! What if it changes?" - and yes, you're exactly right. Because this is a demo application, I'm favouring readability over abstraction; if you had to define this kind of default behaviour in four different places in a real app, you'd do it using constants, or implement it on the Greeting object itself so the code is reused between all four endpoints.

Code for this post is on GitHub as v0.0.2.

Now we've got a basic skeleton API up and running, it's probably time to share my to-do list - things we'll be implementing, in no particular order

  • List resources, and retrieving multiple objects
  • Support for PUT and DELETE (easy), POST (interesting) and PATCH (really interesting)
  • Hypermedia links using HAL
  • Pagination and resource expansion, as described in this excellent post on the Stormpath blog and this blog post from Etsy's Venkata Mahalingam
  • API versioning - because, sooner or later, we're going to break something and need to maintain backward compatibility for older clients. Check out Troy Hunt's great post on API versioning for a preview of how we'll be getting this wrong in three different ways.
  • OAuth2 and bearer authentication - this one's going to take some work, because before you can build an OAuth2 API, you need an OAuth2 authentication server, but I've wanted to build a really lightweight .NET OAuth2 server for a while so I'm quite looking forward to it.
  • API documentation using tools like Swagger, which generates documentation based on metadata exposed by your API itself.

See you next time.

Monday, 4 May 2015

Restival Part 2: All Aboard The Routemaster

Hello, and welcome to the second instalment of Restival: The Great .NET ReST Showdown (part 1 is here if you missed it)  So far, our API defines a single very simple method – “hello”. Making a GET request to /hello will return { "Message" : "Hello, World!" }, and making a GET request to /hello/chris will return { "Message" : "Hello, Chris!" }

The code we're discussing here is on GitHub as release v0.0.1. This release supports /hello/{name}, which demonstrates routing and parameter binding. I've deliberately not implemented "Hello, World" at /hello yet,   because I want to do that by using the various frameworks' conventions for specifying default parameter values and that logically can't happen until you've defined your routes. Even at this incredibly early stage, there's some interesting design decisions going on.

Routing and Route Binding

Routing controls how your app will map incoming HTTP requests to methods - it's the set of rules that say "when you get a request that looks like X, run method Y on class Z"

Nancy has a really lightweight routing syntax inspired by Sinatra - by inheriting from NancyModule, you get access to a RouteBuilder, a sort of dictionary that maps routes to anonymous methods, for each supported HTTP verb (DELETE, GET, HEAD, OPTIONS, POST, PUT and PATCH) - to add a route, you supply the pattern to match and the method implementation:

public class HelloModule : NancyModule {
    public HelloModule() {
        Get["/hello/{name}"] = parameters => new Greeting(;

Note the Nancy convention whereby we use an underscore to indicate "we're not using this variable for anything" in handlers that don't actually use their parameter dictionary. It's also worth noting that Nancy's lightweight syntax won't stop you defining multiple handlers for the same route - but this can lead to non-deterministic behaviour, so don't do it :)

WebAPI uses an explicit routing table that's configured during application startup - in WebAPI, there's a call to WebApiConfig.Register(GlobalConfiguration.Configuration) in Application_Start, and routes are mapped by specifying the name, the URL pattern and the defaults to use for that route. (If you're familiar with routing in ASP.NET MVC, WebAPI uses a very similar routing configuration, but with the 'action' mapped to the HTTP verb instead of to a path segment.)

    "Hello",   // route name
    "hello/{name}", // route template
    new { Controller = "Hello" } // route defaults

OpenRasta and ServiceStack are both far more explicit about the relationship between resources, routes and handlers. OpenRasta uses a fluent configuration interface to declare your resources (i.e. the bits of data we're interested in), your URIs (routes), handlers (the bits of code that actually do the work), and contracts (which handle things like serialization and content types)

public class Configuration : IConfigurationSource {
    public void Configure() {
        using (OpenRastaConfiguration.Manual) {

Finally, ServiceStack requires you to explicitly define requests (DTOs representing the incoming request data), services (analogous to handlers in our other frameworks) and responses. This is far more verbose than the other frameworks, but providing these abstraction layers between every aspect of your ReST API and your underlying codebase gives you more flexibility to evolve your API independently of the underlying business logic. You map your routes to your request DTOs using the Route attribute, and inherit from ServiceStack.Service when implementing your handlers. ServiceStack maps HTTP verbs onto service method names - HelloService.Get(Hello dto), HelloService.Post(Hello dto), etc. - but also supports a catch-all Any() method which will map incoming requests regardless of the request verb.

public class Hello {
    public string Name { get; set; }

public class HelloResponse {
    public string Message { get; set; }

public class HelloService : Service {
  public HelloResponse Any(Hello dto) {
    var greeting = new Greeting(dto.Name);
    var response = new HelloResponse() { Message = greeting.Message };
    return (response);

So there you go. /hello/{name} takes one line in NancyFX, a couple of lines in OpenRasta and WebAPI, and three entire classes in ServiceStack. Before you draw any conclusions, though, try pointing a browser at the root URL of each API implementation.

Nancy gives you this rather splendid 404 page - complete with Tumbeast:


Running under IIS, WebAPI and OpenRasta both interpret GET / as a directory browse request, and give you the all-too-familiar IIS 7.5 HTTP error screen:


But the pay-off for the extra boilerplate required by ServiceStack is this rather nice API documentation page, describing the services and encoding formats supported by the API and providing WSDL files for adding our API as a service endpoint. Now, we're not actually using any of that yet... but as our API grows, it's going to be interesting to see how much extra work the other frameworks require to do things that ServiceStack provides for free. (Or for $800 per developer, depending on what you're doing with it.)


Now, it's important to remember that we're trying to reflect the conventions and idioms of our chosen frameworks here. You could, without too much difficulty, implement the request/service/response pattern favoured by ServiceStack on any of the other frameworks, or to get your ServiceStack services to return raw entities instead of mapping them into Response objects - but if you're trying to make framework A behave like framework B, you might as well just switch to framework B and be done with it.

In the next episode, we're going to make GET /hello return "Hello, World!", and in the process look at how to define default values for our route parameters in each of our frameworks. Until then, happy hacking!