The Daily Telegraph has highlighted the tricky issue of redaction. FOI Man reflects on the perhaps surprising difficulties of blanking out a bit of text.
Today’s Daily Telegraph features a story about redaction. And for a change, this is not a story complaining about public authorities redacting too much, but about them failing to do so properly.
The Departments for Health and Communities and Local Government, and the Ministry of Defence, are all alleged to have disclosed and published documents containing redactions. But unfortunately, it appears that the redactions were poorly done, and as a result, the material that should have remained secret can be read by requesters and others with very little effort on their part.
Redaction, for those who don’t speak FOI, is the term used to describe blanking out information in documents. It happens when public authorities are disclosing documents but there are particular words or passages that contain sensitive information and are therefore exempt. Rather than refusing to provide the whole document, public authorities will blank out the relevant sections.
It is a difficult process from start to finish. First of all, if the document (or documents) is very long, it can be time consuming (and this time often can’t be included in estimates of the cost). Secondly, as the Telegraph has highlighted, the practicalities of how to redact are not straightforward.
The Telegraph gives two examples of how redaction can go wrong. In the first, it appears that the Civil Servant responsible thought they had successfully blanked out the relevant sections using available software, but when the journalist studied the documents, it was a simple matter to highlight the relevant sections to see what had been supposedly hidden. In the second, rather more prosaic (and familiar) example, the text had been blanked out using a black marker pen, but when the document was held up against the light, again the information was magically revealed.
Another common difficulty occurs with Track Changes™ or similar functionality in office software (or more accurately, with staff understanding how it works). In a previous job, we purchased redaction software in an attempt to overcome these issues, only to find that it didn’t work properly (it tended to blank out more than the section you wanted to cover up).
In the end, less technical solutions tend to be the most effective. The standard one is to use a black marker pen to cover the relevant words, then photocopy the pages, possibly use the pen again on the photocopy, then photocopy the pages again, and so on until you (and usually half a dozen colleagues interrupted to double check it for you) are satisfied that the words or passage can’t be read.
My favoured solution, sometimes complemented by the one above, is to use cut up bits of Post-It ™ note or paper that can be otherwise secured, and place them over the relevant sections before photocopying the pages (taking care not to dislodge said bits of paper en route to the photocopier). You can even indicate the relevant exemptions on the paper covering each section. This is effective, and has the added benefit of making your desk look like the aftermath of a Blue Peter craft session. “And here’s a document I prepared earlier…”
So I feel for my colleagues in central Government. They will no doubt want to read up about redaction, so if they and you want to know more, both the Information Commissioner and the National Archives publish useful guidance for public authorities.
If you’re waiting for my post on exemptions and the public interest test, don’t worry, I haven’t forgotten about it. I’ve just extended the deadline(!) and hope to publish it later this week.
Well, if the point of making information available to the public via FoI is for it to be useful, then the process you have described is a very good way of making that as hard as possible (in other words, verging on use-less). PDF is already a pretty munged version of useful text, but once printed, photocopied and scanned it is likely the only way of getting the (non-redacted) information into a useful form is to transcribe it by hand (many of the documents I see in planning web sites, for example, are pretty poor quality and would be tough tests for OCR).
I can see why you do it, but surely it’s the opposite of the point. What about shouting at your redaction vendor until he gives you a version that works?
Chris, thanks for the comment. First things first – and I hope this is clear in the post (in which I have my tongue firmly in my cheek) – I agree that the way this is done is not great. Aside from anything else, it’s a cumbersome process for us.
Having said that, the point you’re making is true of some situations, but arguably not of most in my experience. If you were collating a lot of data (which I know may be relevant in your case), it may well be a problem to receive information that has been photocopied x number of times and then scanned into a pdf. I think the public sector is increasingly aware of that, and certainly if you’d asked for data in a usable format, I’d be looking to find another way to redact information (in practice, we’d probably be looking to remove whole columns of data, so we can just explain this in our covering response). I think this is a different issue – it’s unlikely that redaction as such would be used for this kind of data.
Many requesters, it seems to me, are perfectly happy with paper copies or pdfs of documents, as long as they are actually physically readable. They aren’t necessarily going to be reusing it – they just want to know for themselves what was discussed internally, or how much was spent, etc. They might not be happy about information being concealed, but I’ve rarely if ever heard anyone complain about the fact that a redacted document was in the wrong format or difficult to read/OCR or whatever.
As for the redaction tool vendor – at the time, there wasn’t much choice as I recall; we had next to no budget, there were only a couple of companies producing the software, they were in the US and making software for the US market, so we pretty much had to go with the off-the-shelf product. The other issue, which is, I suspect, often the case, is that it just wasn’t worth spending a lot on. We only had to redact information about once a month on average, and frankly, on that basis, it was most cost effective to go back to basics.
Oh and I should add about the redaction software we used – it was a plug-in for Adobe, so it only worked if you converted the document into pdf. I’m sure there are better tools on the market now if you’ve got the budget to pay for them.
Thanks, FoiMan! However, “reuse” can be for much more than data; even quoting from the document in (say) an email becomes tedious with the technologies you describe. I do recognise the chronic under-funding however; maybe that’ll change as the volumes slowly ramp up (;-)!
Actually, I would hope that volumes would decrease as we all get better at publishing information online so people don’t have to ask for it!
I suspect many organisations send information as pdf as standard operating procedure, unless it has been specifically requested in another format:
a) pdf is the most efficient format size-wise
b) there are unlikely to be any ‘read’ issues with differing brands/versions of software
c) pdf readers are easy and free to grab online in the unlikely event that the recipient doesn’t have one installed. So dear requester, if you want your info in a particular editable format, please say so!
It’s also possible that we genuinely don’t have the information in editable format ourselves – the last major redactions I had to do were on a report from an external organisation, which had only been supplied to us as pdf. (Black marker pen + photocopy + scan photocopy worked just fine there).
Finally, philosophical question: how far should we go to provide information in the format requested? A requester sends us a spreadsheet with various headings that they want completed (this actually happens quite a lot). We have, or can get (e.g. from database) the information in editable and clearly comprehensible format, but it would take several hours to re-order it exactly how the requester wants it. Should taxpayers pay for that or – as it’s not difficult, just the sheer volume of information makes it time-consuming – should the requester do the work?
America’s National Security Agency has wrestled with this, and published a really comprehensive guide called “Redacting with Confidence” – full link here: http://www.nsa.gov/ia/_files/support/I733-028R-2008.pdf
It assumes you’re using Word 2007 (much of the public sector still uses Word 2003), and Adobe PDF – but the principles will be applicable whatever software you’re using.
Edited highlights: remove metadata, then delete the text from the original document and replace with Xs or just [REDACTED]. Then PDF it.
So you end up with a searchable PDF containing the text (much better for re-use than a scanned PDF), and no chance that anyone can retrieve the redacted text.
Adobe Technical Note:Technical Redaction of Confidential Information in Electronic Documents – How to safely remove sensitive information from Microsoft Word documents and PDF Documents Using Adobe Acrobat
I am not sure what all the fuss is about. A number of people have commented upon the use of redaction technology and it works really well. I introduced the process into my jurisdiction about 9 years ago and wrote an article about it in the FOI Journal a short while after. Time and time again I hear about problems with redacting but if its done properly with the right software nobody should have a problem – ever! I am completely confident about this.