Posts Tagged ‘Government’

It’s been a while since I liberated any data and that’s because I’ve been wrestling with a scraper of Government Salaries. I’ve only looked at pay floor for the tables below. This is the minimum pay with ceiling pay being £4,999 more than the floor. Salaries ranged from £35,000 to £235,000.

There was the coding to deal with of course. I’ve had my third formal lesson. Lists, ooh er. But the main difficulty came from the awful state the data was in. All CSV files but spread over the web with a blatant disregard for consistency. Anyway, you can download the data for yourself. There are 440 rows so looking back at the poor quality of the data one could easily have done a copy and paste job, but a scraper that trawls a site for the links to the pages that has the download ensures you get all the data sets and makes the next collection a matter of hitting a button (hopefully).

Here is the top 10 for pay:

If you’re looking to climb your way up to the top pay band then here’s the top 10 departments:

And since the Information Commissioner has ordered release of names of high-paid civil servants who did not want their salaries disclosed the data shows 48 names have been withheld amounting to £508,000 in cumulative pay. Most of these redactions came from the Cabinet and Home Office. I can understand why names from the Office for Security and Counter-Terrorism Unit were not disclosed. One consistency with the withholding of names is from the lawyers. The entire list of names for the Office of the Parliamentary Counsel is N/D as is the legal advisers branch of the Cabinet Office. Also not named are the Media Director and Communication and Change Director. Here are the details where the names are not disclosed:

If you follow Scrape_No10 on twitter you will be receiving all the meetings, gifts and hospitalities received at No.10 by ministers, special advisers and permanent secretaries. If you follow #Scrape10 then you should be getting those as well as all the tweets relating to any item of data. The current database should be tweeted out by July.

The information contained in each tweet comes from the data published by the Cabinet Office. I scraped each data set and sorted the aggregated information chronologically, so that it can be tweeted out in the order in which the events happened (not every day was given). The links to the original data source and the scraped data are given at the bottom of this post.

With respect to what I have done, I would like to remind people that Freedom of Information does not equate to contextual knowledge regarding the information or making it useful. I am a data journalist. Data is my beat. But data is a public right. Not just your data but also the data of the people who work for you – government data.

Journalism involves information but also conversation. Each data entry now has the ability to start a conversation. Just use #Scrape10. If a tweet is interesting or someone somewhere has added a piece of news relating to the tweet, then #Scrape10 should trend and the tweet should be sent around to the community where the information matters. That’s the theory.

Information is now socially enabled and should be socially enabling. What you would like to know, what matters to you or your wider social community should not only be made available to you but should be made useable in a way that matters.

I am an experimental data journalist, playing with code. All my source code for collecting this data is open and you can download the entire dataset. The code to get it onto twitter is not available, as the publication of the authorisation keys would allow people to hack into the account. I have also written a scraper to store #Scrape10 tweets into a database everyday, so you can catch them all here if you want to.

You can also read a previous post on the Special Advisers’ gifts and hospitality dataset here.

Source Scraper
Permanent Secretaries’ Meetings Permanent Secretaries’ Meetings
Ministerial Meetings Ministerial Meetings
Ministerial Hospitality Ministerial Hospitality
Ministerial Gifts Ministerial Gifts
Special Advisers’ Gifts and Hospitality Special Advisers’ Gifts and Hospitality