Tag Archive for records management

Valuable information

FOIMan on literally giving your information value.

coinsWe often hear people talk about information or data being valuable. But in the last 24 hours I’ve heard two separate speakers, ostensibly on two separate topics, discuss attributing actual monetary cost to information. So perhaps there’s something in it.

First, yesterday evening David Ryan, who was hired several years ago to establish the National Archives’ digital preservation department (and a declaration of interest, he also gave me my first information management job 20+ years ago – don’t hold it against him), was talking about the future of records management at the Information and Records Management Society’s London Group meeting. Amongst other things, David noted the move of many organisations to cloud storage, meaning that there is a noticeable increase in cost if more data is stored each month. He gave the example of Amazon’s cloud storage service, AWS, which now offers customers a retention scheduling tool to help them manage the cost by ensuring that stored data is automatically deleted or archived. He asked if anyone included a monetary cost for record series identified in their records retention schedules. Nobody did, but he speculated that that might become a feature of retention schedules and information asset registers in the future. An invoice might have an intrinsic value to a business in much the same way as a chair.

Which was fascinating but to some probably seemed a long way off. Then today I attended the Direct Marketing Association’s (DMA) Data Protection update, a conference aimed at informing marketers in particular about the General Data Protection Regulation (GDPR). It was an enjoyable event and I found it useful to hear about GDPR from a different perspective.

One session was delivered by Nicholas Oliver, a youthful entrepreneur who talked about “Unified decentralisation & the future of a consumer-led data economy”.

Yes, I know – I was fully prepared to spend that half-hour catching up on email. But it was very interesting.

Nicholas identified that most of us are rather unnerved by the growing trend towards creating unified profiles of us. The fact that Facebook appears to know what we just bought from Amazon and suchlike. He compared this practice to what Edward Snowden revealed about the US security services and concluded that there was little difference between that and what companies are doing to better target their marketing. Having collected all this data, the companies think they own it, and there have even been suggestions that individuals who try to prevent its use are somehow at fault (John Whittingdale, former Culture Secretary, being a notable proponent of this view in relation to ad-blocking).

Nicholas is a businessman and having identified the problem, was there of course to provide us with the answer – or at least his answer. His company, people.io, provides an online platform for people to choose what marketing they receive. And interestingly, given what David Ryan had to say, they actually get paid for their personal data. So you sign up, indicate your preferences, and at some point you or a charity of your choice, receive a payment. Meanwhile, the advertising you receive is more targeted (so in theory less irritating), and more likely to result in you spending money on products so the companies who sell things to you get more value from their advertising budget. What’s more, Nicholas stressed the fact that consumers have control over their data at all times – once they decide not to receive marketing anymore, their data is deleted. We’re used to our data being a valuable commodity to the companies that collect it. We’re maybe not so used to the idea that it might have monetary value to us.

I haven’t looked at Nicholas’ service and I’m not endorsing it (there may well be other products out there that do something similar), but I did think the approach he described was interesting and seemed very much in line with the GDPR’s emphasis on individual control over data. Elizabeth Denham, the new Commissioner, said yesterday that it’s not about privacy OR innovation, it’s about privacy AND innovation, and this sounded a lot like the kind of thinking that she has in mind. Put together with David’s talk yesterday, it has made me think about how literally to take the phrase “valuable information”.

FOI-avoidance or good records management? Cabinet Office Email Policy

FOIMan wonders if 90 day retention of email by the Cabinet Office is a conspiracy, or if it could be a (clumsy) attempt at governance.

The Cabinet Office

The Cabinet Office

The FT has a piece this morning stating that the Cabinet Office routinely deletes email after 90 days. It argues that this is evidence of the Cabinet Office deliberately avoiding FOI. Cue outraged voices from all quarters.

However, this may not be the dark conspiracy that everyone supposes. Email causes significant problems for those responsible for managing information in all organisations. Despite the fact that it has been around for years, there is no generally accepted approach to managing it. Records management consultant James Lappin wrote a really good piece a couple of years ago outlining the options and their shortcomings. Amongst them is the option of automatically deleting email after a short period.

The argument in favour of this policy is that most email is ephemeral. Keeping it all forever isn’t an option for various reasons (including data protection requirements). But some emails are important records. What you need is to identify a way to encourage staff to file the important emails. But staff tend to put off filing. So the aim of the 90 day retention policy is to force staff to file key email because if they don’t it will be gone forever.

There are of course weaknesses with this policy. Staff may still fail to file things and then they are lost. It is usually easy for staff to find ways to get round the policy by creating folders and moving all their email into them in bulk before the 90 days is up. Their intention is to get round to weeding and filing them at some point but it never happens. As the FT article demonstrates, the policy can be readily misinterpreted. But it is a policy widely adopted because at least it is an attempt to manage the chaos of the email server.

My understanding is that this policy was adopted by many government departments in 2004. It was a policy that we adopted at the Greater London Authority possibly at my suggestion (it’s a long time ago). The intention – at least on my part – was to manage email, not to bypass FOI. As I’ve previously argued, retaining everything would be impractical – it might even reduce the likelihood of information being disclosed, as the section 12 limit would more often be an issue. And I should be clear – the intention was never that all emails would be deleted; it was to encourage staff to take action by identifying and keeping the emails that were formal records. My strong suspicion is that staff in the Cabinet Office are encouraged to do the same. This is not about deleting ALL emails; staff should be saving emails which document key decisions within whatever records management system is in place within the Cabinet Office.

One of the reasons I advocated it was that it was a policy which had been adopted by a previous employer. That employer wasn’t in the public sector, and didn’t face the prospect of receiving FOI requests. I saw it as a legitimate and established way to address the management of email.

None of this is to say that the policy hasn’t become a convenient way for the Cabinet Office to avoid having to answer certain questions. As a policy it has significant weaknesses and it is not necessarily one which I would advocate now. Given the complaints made by the officials quoted by the FT, the approach doesn’t seem to be working well for the Cabinet Office’s staff (although it’s also possible that those quoted ignored the instructions they were given on email filing). If I was advising the Cabinet Office (not very likely, I grant you), I would be advocating a different approach, not least because of the obvious reputational harm that it is causing.

As I’ve written before, records managers do need to consider the political ramifications of their advice and policies. But my strong suspicion is that the 3 month email policy was not – at least initially – proposed as a way to avoid FOI. It was just one of many options for managing email – and every single approach that I’ve ever used or read about has its shortcomings.

Who’s afraid of the dark age?

FOIMan explains why he’s not afraid of the dark age.

In my last post I recounted how pioneers in the UK have contributed to the development of digital preservation solutions over the last 20 years. This was inspired by several news articles at the end of last week reporting on Google Vice-President Vint Cerf’s comments heralding a “digital dark age”. In this piece I want to give my personal reaction to this apocalyptic prediction.

As I indicated in my previous post, the issues raised by Cerf are not new. And indeed he isn’t the first to suggest the dire consequences of a lack of action.

But are such visions realistic? My personal view is that they’re not. Let’s consider what happened in the past.

Acts of Parliament on parchment have survived for over 500 years

Acts of Parliament on parchment have survived for over 500 years

We tend to assume that electronic formats are somehow more fragile than previous media. This isn’t in fact the case. As any archivist or conservator will tell you, failure to keep paper or parchment at the right levels of temperature and humidity can lead to it becoming unreadable. In one job early in my career I found records being stored in damp, dank conditions under the Town Hall steps. They were covered in mould and fungi – to all intents and purposes unreadable. Some of the records were less than 10 years old.

Just as servers can be hacked, intruders or employees with a grievance can access offices and pick up files they shouldn’t have seen. Careless employees can leave files on trains or even in evacuated premises. Fire or flood can destroy whole warehouses of physical records without the insurance of a backup to restore the files.

These risks have always existed. And until more enlightened times, even governments failed to keep their records in suitable storage. Just read Caroline Shenton’s excellent book about the fire that destroyed the Palace of Westminster if you want some illustrations of this.

And yet… Record Offices hold vast quantities of physical records – they complain of lack of space and have significant backlogs requiring cataloguing. Historians will always want more, but the fact is that despite the poor quality of storage in previous eras, the limited literacy of earlier generations, and in some cases the passing of many years, archivists hold vast volumes of evidence on our past.

The problem in our era is not a sparsity of information. It’s a glut of it. And with so much information – whatever format it is originally created in – it is inevitable that a huge proportion of it will survive. Indeed it is fear that information will live on indefinitely that feeds current debate over the right to be forgotten.

It will survive because it is popular – the more copies of a file that exist, the more likely it is that some will remain (take, for example the four copies of Magna Carta recently exhibited together in London). It will survive because people are interested in it. FOI will play its role – now copies of government documents will be found in many personal collections and on websites as well as stored by their creators. A proliferation of information – facilitated in the digital world – will guarantee that vast quantities of it remain accessible to future historians.

The real problem is not whether there will be information that will remain accessible, but which information should do so. As I’ve suggested, lots of it will live on purely through chance. But it is important that organisations (and individuals too) identify the records that have most value – especially long-term value – and take deliberate action to preserve them. This too will happen because there are commercial, governmental or sentimental reasons to retain them. In my last post I explained the need for pharmaceutical companies to retain digital records – so they took steps to ensure that those records would be preserved. Similarly the digital photographs that you look at the most – of your children, your significant experiences – will almost certainly survive because you will regularly look at them and if you have problems accessing them you will do something about it.

Digital records require specific techniques to ensure their preservation (as indeed do records printed on paper or written on parchment). That’s why the work of the pioneers I wrote about in my last post is so important. But in principle at least, preserving digital records is no different to preserving records created in other formats. It requires the organisation or individual to first identify what it needs to keep (a point made by the National Archives’ Chief Executive, Jeff James, on Saturday’s Today programme on BBC Radio 4). How it will keep it is a secondary and technical question, but one that will be answered if the information really does have value.

This is why, short of a nuclear holocaust (in which case I suspect we will have more pressing concerns should we survive), I don’t think a dark age is coming, digital or otherwise.

Preserving our digital past

FOIMan highlights the important work of UK pioneers to preserve digital records for future generations.

Thank goodness for Vint Cerf. Cerf’s up because he has been speaking at a conference in the US about the dangers of a “forgotten century”. He is highlighting the problem of digital preservation which is an important one. And because he’s a “web pioneer” and a Google Vice-President, the media are listening to him.

If you were to read the BBC News website or the Guardian this weekend, you’d be forgiven for thinking that this is a problem that has just dawned on very clever people in the US. But the truth is that whilst it is welcome that this issue has finally attracted the attention of journalists, archivists and people in the IT industry have not only been aware of these issues for some time, but have also been putting forward solutions. Many of them are right here in the UK.

Good luck accessing your 1980s project report (or loading Manic Miner) from this today

Good luck accessing your 1980s project report (or loading Manic Miner) from this today

In a nutshell, the problem is twofold. One, the hardware that runs computer programs is regularly superseded. On my records management courses, I illustrate this by producing a 7″ floppy disk from the 1980s. The hardware that can read that disk only exists now in a handful of museums, but when I was at school, it was used there and in most offices.

Two, the software that is used to create the programs, to manage your email, to retain your photographs, to write letters – also changes all the time. Each time you get a message saying that a new version is available the chances of being able to open a document created in the original version are reduced. Software manufacturers are focussed on creating something that will do lots of new sexy things rather than something that will continue to open your dissertation from ten years ago.

20 years ago this was a problem in the pharmaceutical industry. Research increasingly depended upon technology that produced data which could not be recorded or preserved by traditional methods. This was a concern because regulatory approval for drugs required the experimental data to be available for long periods. And if you wanted to demonstrate that you had discovered a drug for valuable patent purposes then again you needed the records to prove it. A drug like Viagra, say, discovered by scientists in the UK working for Pfizer, is worth billions to the company. So the records proving its discovery are also worth billions.

Back then I was starting my career in records management, and one of the reasons for pursuing it was that at Pfizer I saw it at the cutting edge. Not only did my colleagues invite experts in the field like David Bearman to visit us to discuss the problems, but they were proposing solutions too. They developed, with the help of a UK company called Tessella, a system called the Central Electronic Archive, specifically to retain – and preserve – this important experimental data. The CEA has since been retired, but the work done in establishing it helped to ensure that its contents remained accessible and could be migrated to its successor systems.

This work started in the pharmaceutical industry but its benefits can now be felt in the public sector. My former boss, David Ryan, was headhunted from Pfizer to set up the National Archives’ Digital Preservation Unit. The Unit has produced useful tools such as PRONOM, a database that maps the compatibility of different versions of software so that organisations can work out how to open documents created in older versions of the software. It has also established a programme to extract digital data from central government departments. One example was a 3D reconstruction of a shipping disaster used at an inquest which otherwise could never have been captured and preserved. In establishing the technical infrastructure for these services, the National Archives has continued to rely on Tessella, who won the Queen’s Award for Enterprise in 2011 for their work on this.

The Digital Preservation Unit’s next head, Adrian Brown, has subsequently gone on to establish a digital preservation programme in the UK Parliament, and is widely recognised as an expert in preserving digital formats. Adrian has recently written a handbook on digital preservation to help others looking to ensure their records will be available for decades to come.

There is still a long way to go here, and Vint Cerf is absolutely right to highlight the issue. But much good work has already been done around the world, including here in the UK where pioneers in industry and in the public sector have shown the way.

The politics of records management

FOIMan reflects on how his boring records management work has twice found its way into the national press and considers whether there are lessons for others who manage their organisation’s information.

MPs' expenses are in the news once again

MPs’ expenses claims – no longer a problem for the House of Commons?

The Telegraph reported earlier this week that MPs’ expenses claims prior to 2010 can no longer be investigated as they have all been destroyed. Defending its actions, the House of Commons authorities explained that they had been disposed of in line with a policy called an Authorised Records Disposal Practice (ARDP), and that to retain them longer would have breached the Data Protection Act (DPA).

The story attracted my attention because…I drafted the ARDP in 2003. I don’t recall why we gave it such a rubbish name, but I do remember why MPs’ expenses claims, receipts and other financial records were to be retained for only 3 years. It was because the House of Commons authorities went out of their way to obtain permission to retain them for this period rather than the usual 6 years. Three years was explicitly chosen because they wanted to limit both the scale of information that might potentially have to be searched, and the risk of “smoking guns”.

As it turned out, three years wasn’t enough to avoid the latter, at least not entirely.

Is the Commons’ spokesperson’s statement that to retain the records longer would have breached the DPA true? The DPA requires organisations using personal data to comply with 8 principles. One of those principles requires personal information not to be kept for any longer than necessary. The Act doesn’t specify how long such records should be kept – in practice it depends on a range of things from other legal requirements to business need. It does require, of course, that organisations develop clear policies on how long personal information will be retained and put in place procedures to implement those policies consistently. To this extent, the House of Commons was following good practice in adopting the ARDP, and indeed in disposing of information in line with it.

This doesn’t entirely justify the Commons statement though. As I stated above, it was their deliberate decision to retain financial information for a shorter period than most organisations, albeit with relevant authority. They could have decided to keep records for 6 years. At any point – especially once the public interest in MPs’ expenses was aroused – they could have amended the policy. Given the historical significance of these records, it is even arguable that they should have been retained permanently. An exemption within the DPA would have provided legitimacy for this action.

So whilst the argument made by the Commons authorities was factually correct at least in part, it was perhaps a little…disingenuous.

This isn’t the first time that my records management work has aroused controversy. In preparation for FOI in late 2004, the Greater London Authority (GLA) held a number of “Records Management Days”. Staff were encouraged to dress down and spend part or all of their working day throwing away files that were no longer required. The aim was to reduce the amount of space taken up by storage for records (City Hall never having been designed with physical records storage in mind), and to reduce the volume of information that would need to be searched through when we started to have to answer FOI requests. (This is arguably of benefit to potential requesters as well. The more information that remains to be searched through, the more likely that a public authority will be able to claim that complying with the request will exceed the “appropriate” or cost limit.)

Records management is seen as an activity that busy people can put off. Put that together with the “just in case” mentality and the challenge for records managers like myself is not generally to persuade colleagues to keep information, but to get them to throw it away. FOI presented a fabulous opportunity to make this happen. If I milked the “smoking gun” message it was a means to an end – though perhaps it suited others for different reasons.

The GLA’s records management project, of which this activity was a part, was approved by both the Mayor and the London Assembly. So it was with some surprise that I learnt that London Assembly Members were asking questions about what were now termed “shredding days” in early 2005. And with even more surprise to find them reported in the Sunday Times in March 2005:

“KEN LIVINGSTONE, the London mayor, has admitted that his office had a “shredding week” to destroy documents ahead of new disclosure rules under the Freedom of Information Act.”

Back in 2005, and again this week when reading about my ARDP’s role in the latest MPs’ expenses story, my initial reaction was defensive. As a records manager I was doing my job. It isn’t possible for organisations to keep everything – just imagine what your own house (or even computer, though that’s perhaps less obvious at first sight) would look like if you never threw anything away. If you must dispose of records, good governance requires that it be done in line with an agreed policy grounded in established best practice. For many years that is the argument that records managers have expounded and stood by. I’m sure that like me, they have seen themselves as slightly removed from the fray, an almost neutral observer. So news articles suggesting that our work is “political” in some sense provoke an indignant response. That’s exactly what our work is there to demonstrate – that the management of information is simply a process. Records are destroyed according to our rules not to the whim of a politician or a nervous official.

But is that true? When I reflect on this latest story, and once again on my earlier brush with politics, I begin to realise that what I thought of as apolitical policies were never really anything of the sort. All policies – and retention policies are no different – have to go through some sort of approval process. They have to take on board the needs of the organisation – political as much as financial and practical. In drafting retention policies for both the Houses of Parliament and the GLA (and for that matter, any of my other employers) I documented established and desired practice. If the Commons’ authorities wanted to keep information for 3 rather than 6 years, then that was what was written into my policy. I don’t remember if I challenged it at the time, but even if I had, it is unlikely that my view would have prevailed. It is most likely that at the time I accepted the view of those authorities unquestioningly. It didn’t seem unreasonable at the time, whatever hindsight suggests.

So those managing their employer’s information should remember that whilst it may be tempting to see themselves as objective rule-makers, they are probably nothing of the sort. In truth, records managers are as much a part of the culture of their organisation as anyone else, and it is perhaps more dangerous to continue to deceive ourselves to the contrary.