Number crunching WhatDoTheyKnow

Posted on
Categories Internet, Politics and NewsTags ,

I’ve mentioned before that I’m a big fan of, the mySociety website which makes it easy for anyone to submit a Freedom of Information (FoI) request to any organisation which is obliged to respond to them.

One thing it doesn’t do, though, is give any kind of top-level overview of the requests being made, so you can’t, for example, easily see which authorities get the most requests or which are the best at answering them. So I decided to generate my own statistics.

Using a combination of the API and some judicious screenscraping, I’ve built up a database of FoI requests made through WDTK and then written a front end to display a selection of stats. You can see the results here.

Most of the stats should be fairly self-explanatory, but a little explanation of the methodology is probably in order. FoI requests at WDTK can have a number of different statuses, including “Successful”, “Partially Successful”, “Refused”, “Information not held”, Internal review” and “Waiting for response” as well as a few other, more esoteric, ones. To obtain the lists of most and least responsive authorities, I’ve added together all the statuses which indicates that a request is complete – that is, the information has either been supplied or a final reason for non-supply has been given – and called them “Completed” requests. I’ve then added together the figures for “Successful” and “Partially successful” (since, in most cases, the reason for most partially successful outcomes is that the requester has either asked for too much information or worded the request badly, which isn’t the fault of the authority) and calculated that as a percentage of completed requests. I’ve further limited this to authorities which have had at least 100 completed requests, in order to minimise the distortion caused by authorities which only have a few requests (or even just one) and a 100% success rate.

It’s pretty tight at the top, with less than 1% difference between the top three, but at the time of writing the most open authority in the UK is York City Council. In fact, it’s notable that all of the top ten, and most of the top one hundred, are local authorities. That’s almost certainly partly because local authorities get fewer of the unanswerable (and, frankly, stupid) requests that go to some central government departments, but, even so, I think this is something which reflects well on local government in the UK.

To get the list of least open authorities, I’ve taken the number of refused requests as a percentage of those complete. This makes more sense than simply taking the lowest number of successful or partially successful requests, since a request can be unsuccessful because the information is not available rather than because the authority refuses to give it. And, while some authorities undoubtedly do evade openness by falsely claiming not to hold information, on the whole most “not held” outcomes are the result of misdirected or badly worded requests.

I’ve also excluded the University of Salford from the list, despite the fact that, on the raw figures alone, they would be at the top. Unfortunately, the UoS has been a victim of a series of maliciously motivated FoI requests which all appear to stem from a single former employee with a grudge. The Information Commissioner’s Office (ICO) ruled that the UoS was entirely justified in refusing these requests on the grounds that they were vexatious, so it would be unfair to include them here. Excluding them takes the UoS below the 100 completed requests threshold for inclusion in the stats.

Unlike the most open authorities, here we have a clear winner (or should that be “loser”?). Standing more than 5% clear of the next worst, the least open authority in the UK is the BBC.

This is probably unsurprising, and some at the BBC would probably argue that it’s unfair, given that the BBC is permitted by law to refuse requests which relate to material held for journalistic or artistic reasons. But, reading through some of the requests which have been refused, I’m not at all convinced that this is being used as much more than a convenient loophole – just because you can refuse to supply some information, it doesn’t mean that you should. The second worst offender, Her Majesty’s Courts Service, has a much stronger justification for refusal in many cases as it gets a lot of requests which relate to specific, named individuals. These are generally barred from FoI disclosure under data protection rules.

Also, despite the generally good showing of local authorities, some do appear in the least open list as well. I wonder why Elmbridge Borough Council finds it necessary to reject 30% of requests?

The next two sets of stats on the page, for authorities with the most overall requests and authorities with the most currently unanswered requests, are pretty much what they say on the tin. They give an idea of the workload faced by authorities dealing with FoI requests.

The stats for authorities with the greatest percentage of “information not held” outcomes is titled “Most Ignorant Authorities”, but that really is unfair for many of them – particularly those at the top of the list. The Prime Minister’s Office, for example, gets huge numbers of requests that either should have been directed to a different department or are not FoI requests at all – they are just political rants and moans. So “Authorities who get the most requests from numpties” is a perfectly good alternative title!

Finally, the table for the “Tardiest Authorities” shows which currently have the largest percentage of overdue requests. Again, there are some very different reasons why authorities appear in this list. The presence of the UK Border Agency in second place, for example, is almost certainly a reflection of excessive workload and a shortage of staff rather than deliberate obstructionism. But it’s hard to see why there should be any excuse for the local authorities in the list.

Alongside the number-crunching, I also decided to use the data I’ve obtained to create my own interface directly to the requests themselves. While the principle behind WhatDoTheyKnow is absolutely brilliant, it has to be said that the user interface could do with some improvements. I’ve taken the liberty of creating an index to FoI requests which, in my opinion, addresses some of those weaknesses. I’d be interested in any comments on this.

  • Paul Cardin

    Great stuff.  I was banned from making FOI requests by Cheshire West and Chester Council – but not via an exemption within the Act e.g. vexatious requests.  They also reserve the right to do it again in the future.  Would such a ban / threat take them out of the top 10, or can we introduce a penalty to such councils?  Brent Borough Council has also used it, and I’m waiting on a few others.

    Brilliant work though.

  • I find that HMCS are fast at responding to requests but are hopeless at providing any sensible answers. It’s almost like they can’t read properly. Their responses can be comical at times – Her Majesty’s Comedy Service:

    • That’s a typical example, though, of what I meant by poorly worded questions. That isn’t a request for information, it’s a query about the law. 

      • No, the requester did ask for information:

        “Who, other than the courts, can lawfully issue a court summons?”

        It would be odd if HMCS didn’t hold that information. It’s not a qurey about the law. However, here’s my re-worded request:

        “Please send me a copy of the document that lists the names of
        Public Authorities that are authorised to issue a Magistrates’
        Court Summons.”

        However, I’m also having the same problems in getting the information I asked for.

  • Tommi Komulainen

    Very interesting. However I think to get a better picture the “Tardiest Authorities” should be based on the number of (business) days it takes to provide the final response. Would also be interesting to break it down further between (Partly) Successful, Refused, Not Held, etc.

    “Most often late” is also interesting, but different IMO.

  • Surplus Labour

    Thanks.  Interesting and usefull work.

    How many qualifying authorities are there in total?