Posts Tagged ‘No. 10’

If you follow Scrape_No10 on twitter you will be receiving all the meetings, gifts and hospitalities received at No.10 by ministers, special advisers and permanent secretaries. If you follow #Scrape10 then you should be getting those as well as all the tweets relating to any item of data. The current database should be tweeted out by July.

The information contained in each tweet comes from the data published by the Cabinet Office. I scraped each data set and sorted the aggregated information chronologically, so that it can be tweeted out in the order in which the events happened (not every day was given). The links to the original data source and the scraped data are given at the bottom of this post.

With respect to what I have done, I would like to remind people that Freedom of Information does not equate to contextual knowledge regarding the information or making it useful. I am a data journalist. Data is my beat. But data is a public right. Not just your data but also the data of the people who work for you – government data.

Journalism involves information but also conversation. Each data entry now has the ability to start a conversation. Just use #Scrape10. If a tweet is interesting or someone somewhere has added a piece of news relating to the tweet, then #Scrape10 should trend and the tweet should be sent around to the community where the information matters. That’s the theory.

Information is now socially enabled and should be socially enabling. What you would like to know, what matters to you or your wider social community should not only be made available to you but should be made useable in a way that matters.

I am an experimental data journalist, playing with code. All my source code for collecting this data is open and you can download the entire dataset. The code to get it onto twitter is not available, as the publication of the authorisation keys would allow people to hack into the account. I have also written a scraper to store #Scrape10 tweets into a database everyday, so you can catch them all here if you want to.

You can also read a previous post on the Special Advisers’ gifts and hospitality dataset here.

Source Scraper
Permanent Secretaries’ Meetings Permanent Secretaries’ Meetings
Ministerial Meetings Ministerial Meetings
Ministerial Hospitality Ministerial Hospitality
Ministerial Gifts Ministerial Gifts
Special Advisers’ Gifts and Hospitality Special Advisers’ Gifts and Hospitality

Seeing as I like to fly in the face of tradition, I’m going to turn things on it’s head and write a blog post of how I did it before I publish what “it” actually is. That is, I have scraped all the Cabinet Office spending data, cleaned it up and extracted it. But before I tell you what I’ve found (indeed, I haven’t got around to that properly yet!), I’m going to tell you how I found it.

Firstly, I scraped this page to pull out all the CSV files and put all the data in the ScraperWiki datastore. The scraper can be found here. It has over 1,200 lines of code but don’t worry, I did very little of the work myself! Spending data is very messy with trailing spaces, inconsistent capitals and various phenotypes. So I scraped the raw data which you can find in the “swdata” tab. I downloaded this and plugged it into Google Refine. I used the text facet functions to clean up the suppliers’ names as best I could (I figured these were of the most interest and would be more suitable for cleaning). This can be done by going into the “Undo/Redo” tab and clicking on “Extract…”. Select the processes you want the code for, then copy the right hand box. I pasted this prepackaged code into my scraper.

So if you want the cleaned data make sure you select the “Refined” table by hitting the tab and selecting “Download spreadsheet (CSV)”. If you want to use the amount as a numerical field (it was not put in as such in the original!) to get totals for each supplier, for example, you’ll have to use the refined table as I had to code to get the “Amount” as numbers. Or if you know a bit of SQL and want to query the data from ScraperWiki you can use my viewer to be found here. Either way, here is the data. I have already found something of interest which I am chasing but if you’re interested in data journalism here is a data set to play with. Before I can advocate using, developing and refining the tools needed for data journalism I need journalists (and anyone interested) to actually look at data. So before I say anything of what I’ve found, here are my materials plus the process I used to get them. Just let me know what you find and please publish it!

————————

Here is a table of the top 10 receivers of Cabinet Office money. I’ve put the image in here but the original is a view that feeds off the scraper so as the data gets published, this table should update. So the information becomes living information not a static visual. The story is being told not catalogued.

Oh and V is V inspired youth volunteering. They received nearly £44 million over a nine month period. On their website they say they have received over £48 million from the private sector. I imagine £44 million of that has come straight from the Cabinet Office. The Big Society seems to be costing the government a lot of money at the moment even though they say it will be mostly funded by the private sector.

The road to No.10 is paved with advisers, they lead you in, they open doors. Often for themselves. Previous advisers include Alastair Campbell, Ed Balls and the Miliband brothers. Until they’re in the door they generally don’t command the political spotlight. That is, unless they’re on the way out like Andy Coulson. What they do command is fine wining and dining.

The Cabinet Office publishes Special Advisers’ gifts and hospitality in various Excel sheets that are filled-in depending on how much coffee the civil servant had that morning i.e. inconsistently. They weren’t even consistent with the appointed minister the adviser falls under. So I scraped it and put all the files into one download which covers May to September 2010. You can get it all by hitting the ‘Download spreadsheet (CSV)’ link here.

The Trends:

Here are the advisers listed according to the amount of hospitality they received:

Note that Nick Clegg’s chief adviser, Jonny Oates, has been taken out the most followed by the then PM’s communications chief, Andy Coulson. Most hospitality is provided by media organisations (see table below) and by using Google Refine I dug deeper into the data to look for a bias between advisers for the Prime Minister and Deputy Prime Minister (seeing as there’s a party split). It turns out the BBC only court Cameron’s advisers (15 times in 5 months). The same is true of the Daily Mail. Whereas The Financial Times dine only with those close to Clegg. The Guardian similarly enjoy Lib Dem company, inviting them to their table twice as many times as they did the Tories.

What’s very noticeable from this information is that Special Advisers are wined and dined mostly by media organisations. Here is a list of the top 10 hospitality givers:

If you add up all of Rupert Murdoch’s empire, they account for 20 occasions split 13:7 Cameron’s to Clegg’s.

The close relationship between advisers and media organisations (this is all within a five month period) makes me wonder: when a ‘No.10 insider’ or ‘someone close to the Prime Minister’ is quoted, how often is that piece of information plucked from the lips of these well-fed advisers? A lot I imagine.

The Outliers:

In fact, media and PR are so predominant in hospitality for advisers, I’ve decided to list the rest of the givers in order of how many times they appear in the data: Bell Pottinger (mostly business clients, Airbus, Sky, Unilever, etc), News Corporation, Tetra Strategy (clients include the Government of Dubai and the jailed Russian billionaire,
Mikhail Khodorkovsky), The Daily Telegraph, The Mail on Sunday, The Sunday Times, The Telegraph, Alexander Kutner, Baron Wolfson of Aspley Guise (Conservative life peer and CEO of Next), Business in the Community, Center for Court Innovation (New York think-tank), Citi, Connect Communications (lobbying), Demos (think-tank), General Sir Richard Dannatt, ITN, ITV, Ian Osborne and Partners, Institute for Public Policy Research (think-tank), Islamic Relief, James Kempton, Lansons Communications (clients include J.P. Morgan, Lloyds TSB and Barclays), London Palladium (Whoopi Goldberg), Malaria No More, Martyn Rose, Medley Global Advisors (“provider of macro policy intelligence service for the world’s top hedge funds, institutional investors, and asset managers”), News International, Lawn Tennis Association, Not to Scale, Open Road, Pakistan International Airlines, Policy Exchange (think-tank), RSA, Ramesh Dewan, Richard Thaler, Royal Bank of Scotland, SAB Miller (De Klerk Foundation Event), Save the Children, Taxpayers’ Alliance, The Daily Telegraph and The Daily Mirror, The Economist, The Evening Standard, The Spectator, The Sun, The Sunday Express, UK Music, Wall Street Journal and Wellington College.

Bell Pottinger and Martyn Rose are now with the Big Society Network.

Only six entries weren’t lunch or dinner dates. Steve Hilton was given champagne from Not to Scale, Steve Chatwin received concert tickets from Malaria No More, Naweed Khan got his flights upgraded by Pakistan International Airlines, Andy Coulson was given theatre tickets by Whoopi Goldberg and a bottle of wine by a one Alexander Kutner.

The Anomalies:

Now the only Alexander Kutner I can find happens to have been the Vice President and Principal Engineer of Software Development at Electronic Evidence Discovery. They reduce the risk of electronic discovery, a process which involves digital forensics analysis for recovering evidence.  See comment below regarding the identity of Alexander Kutner

Also, Ian Osborne and Partners, who dined Tim Chatwin, has no existence according to Google.

The MetaData:

What’s missing is what went on at these meals. Who attended. What was said, or agreed upon. Who was being represented. What goes on is not an entry in the data sheets and it never will be. But this data should make you more aware of the existence of these meals on deals.

You can find a list of Special Advisers and their salaries here.