It’s clear from the explanation that you give, that all four of my assumptions are some way off the mark: I made the assumtion that we’re dealing with expensive, persistent resources, and you have explained that your goal is to produce inexpensive, ephemeral ones.

In terms of my initial post, this does not trouble me: I was interested in working through a problem, rather than telling you how to build your system(!), and the fact that the problem I set out to solve is not the same as the scenario that inspired it needn’t undermine the value of the exercise.

But the scenario you describe is possibly more interesting to model, and I can see various ways this might be done:

1. Treat the results of the scan as normal resources, but destroy them on read. This would follow a POST — 201 Created — GET pattern.

This model is a poor match, as we would need to create an explicit mechanism for storing the data over the 201 Created redirect; also, addressability and findability would fail, as a) the GET resource is ephemeral and cannot be persistently linked to, and b) it’s not possible to link with a POST verb.

2. Treat the results of the scan as a latent resource, which only comes into being when you request it. This would follow a simple GET pattern, and should include all the information about the resource in the hierarchical part of the URL.

There are various reasons you might create a latent resource:

a) There are too many resources for it to be practical (or possible) to store them. If you write a service that calculates square roots, you won’t store every square root, you’ll just calculate it on the fly.
b) The specific resource is a refinement of more generic resources, which can be stored. If you’re showing a weather forecast map for a specific postcode and time of day, you might actually store hourly maps for each region, and then produce the customised map based on the stored data.
c) You do not own the resources in question. A social media hashtag aggregator might be an example here: given any input hashtag, the system can fetch the results from elsewhere, but it would be pointless to pre-fetch the results.
d) You have specific security reasons not to store the resources. This is the case with your scenario.

A model like this maintains addressability, as the URL format is clear, and also emphasises findability, as the user can feed any valid data into the URL and expect a response. If I’m providing a square root service for numbers >= 0, then the user can expect that http://mathsservice/square-roots/n will deliver a valid resorce for any n∈ℕ.

I can see two small problems:

a) In the specific case of running a scan against a URL, there are the technical problems you have outlined in embedding one URL in the hierarchical part of another URL.
b) With some of the examples I have given, especially the weather example, it’s disputable whether we are referring to a single, highly specified resource, or whether in fact we are offering a customised view of a more general resource. Perhaps in this example, the resource is simply ‘the weather forecast’, and we want a view on it that focuses on such-and-such a location and such-and-such a point in time.

This leads us to a third possibility:

3. You treat the URL of the scan as the refinement of an existing resource. This would allow the user to make a simple GET request, with further information (viz the URL to scan) in the query string.

This is the standard model for searches, and is a reasonable fit for the weather example above. It maintains addressability and findability, and as a bonus avoids the technical issues of embedding a URL in another.

The risk is that this approach fails to identify enough resources, and can lead to a RPC style of programming, where a small number of endpoints are overloaded with query refinements. For the weather example, it may be reasonable to see ‘The weather at 16:30 on Saturday for WC1A 1AA’ as a refinement of ‘the UK weather forecase for the next 5 days’, but with your original example, I’m not sure ‘a security scan of http://foo.bar‘ can be as a refinement of anything else; certainly not ‘a security scan of the entire internet’!

I’m not sure if any of these models is absolutely correct for your scenario. I think the notion of latent resources in model 2. is rather interesting, and certainly merits further thought, and I’m also interested in the questions model 3. raises about what counts as a refinement of a resource, and what should be considered another resource entirely.

What I think can be said for certain is that the POST — 201 Created — GET model, which I describe in my original post, while absolutely suitable for persistent resources, is completely unsuited to ephemeral resources, and therefore doesn’t fit your scenario.

There are a few non-URL related constraints I’ve consciously decided to work within which change the discussion a little. Firstly, scans must be fast. typically they should be within several seconds. This means being very sparse with HTTP requests from ASafaWeb to the site being scanned. When they’re running this fast, the user can wait for a response rather than needing to queue them and return a bit later. At present the median scan duration is 2.5 seconds across 4 HTTP requests – and that’s issuing requests synchronously and not using HTTP compression so it should come way down yet.

Secondly, I don’t want to store any identifying information about either the requestor or the site being scanned which means I don’t want to keep the URL anywhere. This is primarily for privacy – people shouldn’t feel that I’m building a list with the mother load of vulnerable sites! But of course it also significantly mitigates my responsibly; the entire site and DB can be compromised and I’ll just redeploy it without suffering any disclosure fallout.

Of course the performance issue remains but without having yet profiled this, I think I’ll find the bottleneck is not in computing resources but rather in the number of simultaneous HTTP requests that are running. One alternative to persistent storage as a means of overcoming this is caching. I’m using AppHarbor and I’m very keen to try out the Memcacher service that they offer. This could greatly mitigate the scenario where the URL of a scan is sent around, say, via Twitter and gets a heap of hits in a short time frame.

All that said, it’s very, very early days (still in private beta) and I know I’ll inevitably chop and change things as I go, particularly once it has public exposure. I may well change course on the above principles, but I think it’s a cautionary way to begin and I’d always rather start simple and scale it up after that.

Your post is fantastic and great food for thought. I do actually have an item on the backlog related to JSON based services and would love to get your input so will get in touch with via Twitter.