What to Store and When to Remove

Last updated on 2026-07-23 | Edit this page

Estimated time: 55 minutes

Overview

Questions

  • How do I decide which research data to keep and which to delete?
  • How long am I required to retain my research data?
  • What are the risks and hidden costs of keeping everything forever?
  • How do I permanently delete data across different systems?

Objectives

By the end of this episode, learners will be able to:

  • Identify key factors (funder policies, data value, provenance) when evaluating data retention.
  • Evaluate a sample dataset to distinguish between high-value raw data and redundant temporary files.
  • Permanently delete files using both local desktop file managers and remote server environments.
  • Explain the psychological barriers to deleting data and the impact of digital waste.

The Problem: Too Much Data, Too Little Clarity

In the last episode, we looked at how to measure file sizes, compress directories, and move data efficiently. But as our projects progress, we encounter a deeper problem:

We are running out of space, and we don’t know what half of these files actually are.

It is incredibly easy to adopt a “keep everything just in case” mentality. We generate temporary versions, intermediate analysis steps, raw outputs, and duplicate backups. Over time, this creates a digital swamp that makes finding relevant data difficult and increases the risk of mistakes.

As researchers, we need a strategy to determine what data is worth keeping, what must be kept due to policy mandates, and what should be safely removed.

Discussion

To Keep or To Delete?

Before looking at specific rules, let’s think about how we evaluate the value of our data.

Working in pairs or small groups:

Imagine you are cleaning out a project directory from two years ago. What factors or criteria would you consider when deciding whether to retain a file or permanently delete it?

Take 5 minutes to list as many considerations as you can.

After a few minutes, gather responses from the room.

What to Consider: The Retention Checklist

When deciding the fate of your data, expect to balance several competing factors:

  • Funder and Institutional Mandates: Many research funders (e.g., UKRI, NIH, Horizon Europe) explicitly require research data to be preserved for a set period, often 5 to 10 years after project completion or publication.
  • Reproducibility and Provenance: Can this data be easily re-generated? Raw data from an expensive, one-time physical observation or experiment is irreplaceable. An intermediate clean data file generated by a script can be deleted, provided the script and raw data are saved.
  • Legal, Ethical, and Privacy Constraints: If data contains sensitive personal information, or if consent agreements state data must be destroyed at the end of a study, you are legally obligated to delete it. Data can be requested by the person it is about and they can request the date is delected or ammended, see the Personal data access policy This will be discussed in a later section.
  • Data Integrity and Documentation: If a file has no metadata, no clear naming convention, and no documentation explaining its origin, its scientific value drops significantly. Data without provenance is often just digital noise.

Evaluating the Messy Dataset

Let’s look at a snapshot of our project directory and decide what stays and what goes.

Challenge

Challenge

Look at the following list of files found in a messy project folder. Based on what we’ve discussed, decide which files should be Retained and which should be Deleted. Be prepared to justify your choices!

  1. raw_sequencing_data_2024_03_12.fastq (Original machine output from a £5,000 lab run)
  2. temporary_merged_subset_v2.csv (A temporary file used to test a quick plotting idea)
  3. analysis_script_final.py (The code used to clean and process raw data into the final figures)
  4. draft_paper_comments_John.docx (An old manuscript draft from last year with tracked changes)
  5. unlabelled_test_results.txt (A text file with numbers, no headers, and no date; nobody remembers who made it)

Take 5 minutes to categorize these.

  • 1. Raw Sequencing Data: RETAIN. This is original, expensive raw data with clear provenance. Funders generally mandate keeping this.
  • 2. Temporary Merged Subset: DELETE. This was a temporary scratchpad file. It can be easily recreated if the scripts and raw data are kept.
  • 3. Analysis Script: RETAIN. This contains the methodology and workflow. It takes up almost no space but holds massive reproducibility value.
  • 4. Draft Paper Comments: DELETE or ARCHIVE. Once the paper is published, dozens of older internal draft files add noise. If retained, move them out of the active data directory into a dedicated “administrative archive” folder.
  • 5. Unlabelled Test Results: DELETE. Without context, documentation, or headers, this file is useless and potentially dangerous if mixed up with real data later.

How Deletion Actually Works

When we decide to delete a file, we need to understand how different systems handle that process. It isn’t always as instantaneous or permanent as it looks.

1. Local Deletion (Windows and macOS)

When you delete a file using your graphical file manager (File Explorer or Finder), the system doesn’t immediately scrub the data off your hard drive.

The Safety Net: The Recycle Bin / Trash

  1. When you click Delete or drag a file to the bin, the file system simply moves it to a hidden folder called the Recycle Bin (Windows) or Trash/Bin (macOS). Linux will use something similar.
  2. The Catch: The file still occupies space on your computer or hard drive! If you are deleting files to free up room because your drive is full, your drive will remain full until you take the next step. Data can still be recovered, which might be an issue if the data needs to be securely deleted
  3. Action: To truly free up space, you must right-click the Recycle Bin/Trash icon on your desktop and select Empty Recycle Bin or Empty Trash.
Caution

Secure data deletion

Even then data may not be entirely deleted and remains on disk until it is overwritten.

If you need make absolutely sure data is deleted:

  • Combine file deletion with encryption.
  • For Hard Drives use tools like shred (Linux), cipher (Windows) or rm -P in a Terminal (MacOS) .
  • Solid State Drives have their own secure deletion tools.
  • Securely dispose of the disk using confidential waste services.

Contact NUIT for advice if you have concerns.

Hard Drives and Solid State Drives behave differently when it comes to secure data delection.

2. Remote Server Deletion (Linux Storage / Research Data Warehouse)

Many institutions provide remote servers or shared network storage (like Newcastle University’s Research Data Warehouse) for storing massive datasets. Managing files here is fundamentally different.

The No Safety-Net Environment

  • When you access a shared Linux server or an institutional data warehouse via a shared network or file transfer tool, deleting a file is usually permanent and immediate.
  • There is typically no Recycle Bin or Trash folder on a remote Linux server filesystem.
  • Once you confirm you want to delete a file or directory, the pointers are destroyed, and the space is instantly reclaimed by the server.
  • Nothing stops you making your own Recycle Bin by having a folder to move files into, but it will lack the restore feature.
Caution

Double Check Before Clicking

Because remote servers lack a safety net, always double-check the files you have selected before hitting delete. A moment of carelessness on a shared server can wipe out months of collaborative work! Data archiving can function as an intermediate step before deletion.

Some remote servers have backups. Make sure you know if your server makes backups, how regular they are and how long backups are retained. It is useful to know how to access the backups and who to contact for help.

General Principles: The Psychology and Impact of Data Retention

Why do we find it so difficult to delete files?

The Hoarding Trap

Psychologically, deleting data can feel risky. We often worry: “What if I need this obscure intermediate file in three years?”. It feels that consequences of leaving useless data on disc are less than the consequences of accidentally deleting good data. The thought of having to explain why you accidentally deleted data from a £10,000 experiment is not a pleasent one. This fear causes us to hoard data.

However, keeping everything introduces hidden costs:

  • Mental Overhead: It takes longer to navigate folders, copy data, and find what you actually need.
  • Security Risks: Retaining old, sensitive participant data past its expiration date increases the severity of potential data breaches.
Discussion

What strategies could you use to mitigate worries about deleting data?

Discuss as a group.

Digital Waste and the Environment

Data storage isn’t free, and it isn’t completely green. Every gigabyte of data kept forever lives on a physical server spinning in a data center somewhere in the world.

These data centers require significant amounts of electricity to run and massive cooling infrastructure to keep from overheating. Deleting useless, duplicate, or temporary files directly reduces your project’s digital waste and environmental footprint.

Summary

Key Points
  • Research funders often mandate data retention for specific timeframes (typically 5–10 years).
  • Raw data and code have high preservation value; temporary files and undocumented data should be routinely purged.
  • Deleting files on a local desktop moves them to a Recycle Bin, meaning they still occupy disk space until it is emptied.
  • Deleting files on remote Linux servers or Data Warehouses is usually immediate and permanent with no undo option.
  • Over-retaining data creates mental clutter, increases security risks, and contributes to environmental digital waste.