3 compelling factors feeding the digital heap within government departments

Introduction: 

The 20 Year Rule mandates Government departments every year must perform an appraisal, selection and sensitivity review of records created 20 years ago. ​However, during the past 20 years the volume and types of data being created and stored has grown exponentially, leading to significant challenges for digital appraisal now, and in the future.  

In our recent blog, Automated Intelligence outline what has led to the challenges Government organisations are now facing in relation to digital appraisal and selection of records for long-term preservation. We also discuss why it’s important for departments to implement relevant processes and controls to manage born digital records, and as importantly, why the digital heap needs to be addressed, today.  

What is Digital Appraisal? 

The Public Records Act requires government and certain public bodies to transfer records of historical value for permanent preservation to The National Archives (TNA) or Places of Deposit (for locally based organisations such as a magistrates’ court, prison, coroner or NHS body). Although the PRA in the UK dates from 1958, it provides a mandate to collect, preserve and provide access to public records in any format, and that includes born digital records.  

Digital appraisal is the process of deciding which records to keep for permanent preservation and which records can be deleted. 

What is the Digital Heap? 

In 2013, the government transitioned to releasing records when they are 20 years old, instead of 30. Since that date, until 2022, two years’ worth of government records have been transferred to TNA to be made available to the public. Now, a single years’ worth of government records that are 20 years old are transferred each year.  

In the year 2002, new technologies were on the rise, and across government and public bodies there was an exponential growth of born digital records that has led to what has been described as a digital heap that is still being fed today. In 2015, Sir Alex Allan, as part of his Review of Government Digital Records commissioned by the Cabinet Secretary, stated:  

“Existing systems which require individual users to identify documents that should constitute official records, and then to save them into an EDRMS or corporate file plan, have not worked well. The processes have been burdensome and compliance poor. As a result, almost all departments have a mass of digital data stored on shared drives that is poorly organised and indexed” 

This is the digital heap, and for most, it has continued to grow unabated.   

Government departments and public bodies have developed excellent skills and experience managing appraisal, selection and sensitivity reviews for the transfer of paper-based records. However, applying these traditional approaches to the huge volume of born digital records threatens to be overwhelming and ineffective, especially when considering the reduction of the civil service workforce and budgets in recent years. 

Tackling the digital heap requires new skills, technologies and methodologies now, otherwise the ability for the TNA to open records to public access will be severely limited and this will essentially undermine the purpose of the Public Records Act.  

3 compelling factors feeding the digital heap within government departments 

Three factors feeding into the digital heap which are common across governmental departments:

1. Legacy platform obsolescence 

During the past 20 years the scale and volume of born digital records has been accelerating, and in the wake of technological change, not only have many digital formats become obsolescent, but so too have legacy platforms. Data is often residing in platforms no longer actively used that are expensive to maintain, requiring specialist skills and are difficult to access, or have been moved across systems to avoid obsolescence and now also lack original metadata or context. Many of these legacy datasets have been inherited or transferred as part of organisational changes and the only option has been to store as is.

2. Misaligned policy and technology  

Policy doesn’t change as fast as technology. In recent years, technology has been a game-changer increasing efficiencies such as productivity, collaboration, capacity, costs and enhanced governance and security. However, the reality is that the technology, approaches and policy haven’t been applied consistently or effectively for managing the lifecycle of born digital information. As new solutions for document creation and storage emerged the digital heap became increasingly dispersed and unstructured, and of course, difficult to manage. Even today, most staff do not have the time or the inclination to retrospectively manage the information they create on a daily basis, it’s just not practicable. Without implementing automation and intelligence to information governance, the digital heap simply continues to grow.  

3. Budget, priority and methodology 

A lack of budget, prioritisation and a digital appraisal methodology are continuing to feed the digital heap. The trend of delete nothing balanced with fast and furious migrations to the next low-cost cloud location has come to an end. The perceived benefits of low-cost storage has been wiped out by sheer volume, and risk, and there are alternatives to drive the business case for reallocating budget and prioritisation. Using new technology with human-in-the-loop expertise presents significant opportunities to impact costs and risks while meeting legal obligations for long-term preservation.  

How Automated Intelligence can help 

Automated Intelligence (AI) is a market-leading data management solution provider with significant experience supporting transformation projects in both Central and Local Government, as well as public sector bodies.   

AI can help government departments to bring their data under control in order to address the digital heap.  

This is carried out in a four-stage approach detailed below: 

1. 1. 1. 1. 1. Discovery and indexing of source systems to gain and socialise high-level understanding 
            2. Classification and relevancy scoring to separate redundant, obsolete and trivial (ROT) 
            3. Temporary archiving of ROT to enable volume reduction from active and legacy systems 
            4. Ongoing methodology development for technology-assisted review, transfer and disposal 

 If you’d like further information on how Automated Intelligence can help you prepare for digital appraisal, including current use cases of who we’re currently working with, contact us for further information at info@automated-intelligence.com or you can also download our whitepaper here.