Rt Hon Margaret Hodge, chair of the Public Accounts Committee, has pulled the government's transparency agenda into question by criticising its unwillingness to publish data about struggling IT projects, which in turn undermines its plans for taking advantage of big data.
Speaking at an EMC roundtable this week on how the public sector can best utilise big data technologies, Hodge was particularly critical of the Treasury and its secrecy surrounding the Universal Credit system.
"The government still remains selective about which data it chooses to make available. If you want proper accountability and choice [via big data analytics], then the government has got to be more open," said Hodge.
"For example, the Major Projects Authority is doing some incredibly helpful work trying to assess the quality of major projects as they move through government departments. But they have been told they can't publish the traffic light findings [green, amber, red ratings] on whether or not the projects are at risk or not - because they might be damaging to government."
She added: "I would imagine it's because Universal Credit has got a big red on it."
Universal Credit will merge benefits such as jobseeker's allowance, income support, housing benefit, child tax credit, and working tax credit. The Universal Credit IT system will require real-time data on the earnings of every adult, from a new Pay as You Earn (PAYE) system being developed with HM Revenue & Customs.
The DWP wants the system to be built by April 2013, followed by six months of testing before going live.
When asked if she thought the government would ever publish the traffic light ratings from the Major Projects Authority's work, Hodge replied "no". She also implied that government officials in the Treasury are considering pushing back the go live date for Universal Credit.
She said: "When I have looked at the government's attempts to exploit data to a good public policy purpose, all too often they fail. I think the worst example of this is the National Programme for IT. We wasted billions on a very sensible idea because there was a failure to understand the complexity of the policy they were trying to implement."
Hodge also commented on the government's complex datasets, which could make it difficult to achieve the benefits made possible by analysing big data. A recent Policy Exchange report estimated that cutting edge-performance could in time save the government between £16 billion and £33 billion a year.
"There is a problem with consistency of data. Take the Department for Education as an example. The rules that govern the data out of academies is different from the rules that govern the data out of local authority schools. We can't make a comparison of value added investment per pupil. That consistency needs to be secured," she said.
"Definitions also change so quickly for all sorts of reasons. Academies have now moved from being funded by local government, to being funded by central government, for example. This makes it very difficult to make meaningful comparisons over time."
She added: "Also, as the government increasingly chooses to use private providers to deliver public services, it's very difficult to access data because of commercial confidentiality."
Bill McCluggage, a former deputy government CIO and now chief technologist for the UK public sector at EMC, said that to overcome these challenges the government needs to be a 'rapid follower' to the private sector and set up centres of excellence.
He said: "The key, from my perspective, is that you have to understand whether government's role is to be at the bleeding edge or a rapid follower. Because it would be wrong to build the expectation that the government is going to be an early adopter of the big data environment."
"Take the cyber security model, which uses centres of excellence to develop skills. We should do this to develop big data and data scientists. If the UK isn't starting to roll the ball on generating new data scientists we will be behind the curve internationally."
It was announced earlier this year that Eight UK universities were to benefit from a capital investment of £50,000 and additional funding after being awarded 'Academic Centre of Excellence in Cyber Security Research' status by GCHQ.
The universities include University of Bristol, Imperial College London, Lancaster University, University of Oxford, Queen's University Belfast, Royal Holloway, University of Southampton, University College London.
McCluggage said: "The creation of centres of excellence in cyber was driven by a threat that was recognised. Big data needs to do the same, but it mustn't come from a threat perspective but a need perspective. The need to tackle fraud and the need to tackle escalating costs."