All in One View

Content from Introduction to Data Management

Last updated on 2026-07-28 | Edit this page

Overview

Questions

  • Why is data management important?
  • How do computers organise files and folders?
  • What is a file system?
  • How can I use a graphical file explorer to navigate my data?

Objectives

By the end of this episode, learners will be able to:

  • Describe common data management challenges.
  • Explain how files and folders are organised on a computer.
  • Navigate a file system using a graphical interface.
  • Locate project data using a file explorer.
  • Explain why data organisation matters for research.

Introduction

Research increasingly depends on digital data. Whether you are collecting measurements in the field, analysing simulation outputs, recording interviews, working with images, or managing spreadsheets, the data you create must be stored, organised, documented, and maintained throughout its lifecycle.

Good data management makes research more efficient, more reproducible, and easier to share with collaborators. Poor data management can result in wasted time, duplicated effort, lost information, and difficulty reproducing results.

In this lesson we will work through a realistic scenario that many researchers encounter: inheriting a project from somebody else.

Over the course of the lesson we will begin with a chaotic collection of files and gradually improve it using practical data management techniques.

We will work from the outside in:

  1. Organising folder structures
  2. Creating meaningful file names
  3. Documenting data and workflows
  4. Choosing appropriate storage locations
  5. Managing storage and file transfers
  6. Deciding what data to keep and what to remove
  7. Improving tabular data structure
  8. Choosing appropriate data formats

The focus of this lesson is not programming, data science, or statistics. Instead, we will concentrate on practical skills that help people find, understand, share, and preserve research data.

Scenario

Imagine the following situation.

You are a postgraduate researcher joining a collaborative research project.

One of the project’s researchers has recently left the institution. Before leaving, they copied everything they thought might be important into a single folder and sent it to the team.

Unfortunately, the folder contains hundreds of files.

Some files have names such as:

Data!@#$.csv
data_final_v3_LAST_ONE.xlsx
Untitled.csv
copy of DSCO1023.png

Some folders are called:

New Folder
New Folder (2)
miscellaneous
RAW_DATA_!!!

There are multiple file versions, undocumented datasets, hidden folders, and no README file explaining what anything means.

Your supervisor asks:

Can you work out what this data is, organise it, and make sure we can continue the project?

Throughout this lesson, you will take the role of the researcher trying to rescue this project.

Each episode focuses on solving one specific problem and introduces practical techniques that can be applied to your own research projects.

Why Does Data Management Matter?

Many research projects begin with only a handful of files:

experiment.xlsx
notes.docx
results.csv

Finding information is easy.

However, projects grow. As files accumulate, researchers typically encounter familiar problems:

  • “I can’t find the file I need.”
  • “Which version is current?”
  • “Where did this data come from?”
  • “Why am I running out of storage?”
  • “How do I share this file?”
  • “What does this dataset actually contain?”

Most of these problems are not technical problems.

They are organisational problems.

Good data management practices reduce time wasted searching for data, improve collaboration, and make research easier to reproduce.

Callout

Various research funding organisations have data management and sharing policies, and they can request detailed plans in funding applications:

Guidelines for more funders are included in Newcastle University’s overview of Data Management Expectations.

Newcastle University has its own policies for postgraduates and all researchers.

Discussion

Have you ever inherited files from another researcher or returned to an old project after several months?

What was the most difficult part of understanding the data?

How Computers Store Data

Before we organise data effectively, it helps to understand how computers store information.

Computers store information in files.

Files are organised into folders (also called directories), which can contain both files and additional folders.

Together these form a file system.

Visualising a File System 

Computer
│
└── Users
    │
    └── Alice
        │
        ├── Documents
        │   ├── Report.docx
        │   └── Notes.txt
        │
        ├── Downloads
        │   ├── Dataset.csv
        │   └── Image.png
        │
        └── Pictures
            ├── Figure1.jpg
            └── Figure2.jpg

Folders help organise related files in much the same way as folders in a filing cabinet.

A file system provides:

  • Structure
  • Organisation
  • Navigation
  • Storage

Without folders, every file on your computer would exist in one enormous list.

Storage Locations

Files ultimately live on some form of digital storage. Examples include:

  • Internal hard drives
  • Solid-state drives (SSDs)
  • USB drives
  • Network storage
  • Cloud storage

Regardless of where files are stored physically, they are typically presented to users through the same folder-and-file structure. As a result, learning to navigate folders is a transferable skill regardless of the storage system being used.

Introducing the File Explorer

Most people interact with files through a graphical interface known as a file explorer or file manager.

Examples include:

Operating System File Manager
Windows File Explorer
macOS Finder
Linux Files / Nautilus / Dolphin (varies by distribution)

In this lesson we will demonstrate using Windows File Explorer, but the same concepts apply to other operating systems.

Discussion

Opening File Explorer

The instructor will demonstrate how to open a file explorer in Windows.

Depending on the operating system, you can use any of these options to open a file explorer:

Callout

Further instructions for various file navigation tools:

Anatomy of File Explorer

A typical File Explorer window contains several important components.

+--------------------------------------------------+
| Address Bar                                      |
+--------------------------------------------------+
| Navigation Pane | File List                      |
|                 |                                |
| Documents       | Dataset.csv                    |
| Downloads       | README.txt                     |
| Pictures        | results.xlsx                   |
| Desktop         | raw_data                       |
|                 |                                |
+--------------------------------------------------+

Usually found on the left-hand side and shows commonly used folders such as

  • Desktop
  • Documents
  • Downloads
  • Pictures
  • Shared drives
  • Other folders

You can use it to jump quickly between locations.

File List

The central area of the window. This displays the contents of the currently selected folder.

Address Bar

Located near the top. Shows your current location within the file system. For example:Downloads > legacy_dataset > RAW_DATA_!!! This helps you understand where you are and allows you to move back to previous folders.

Many file managers provide a search box. Search can be useful, but it should not be your primary strategy for finding files. A well-organised folder structure makes data easier to locate without relying on search.

Looking Ahead

In the next episode we will examine the inherited dataset and begin improving its folder structure. Before we can organise the files, however, we first need to locate them.

Resources

Newcastle University provides a number of resources for data management:

Discussion

Finding the Dataset

Download the lesson dataset.

Using File Explorer:

  1. Open the Downloads folder.
  2. Locate the downloaded dataset.
  3. Open the dataset folder.
  4. Explore the contents without modifying anything.
  5. Identify at least three things that seem confusing or poorly organised.

Was it immediately obvious where the important data was? What information would have helped you understand the project more quickly?

Key Points
  • Data management is an essential part of research practice.
  • Many common research frustrations arise from poor organisation and documentation.
  • Computers organise information using files and folders within a file system.
  • File explorers provide a graphical way to navigate stored data.
  • Understanding how files are organised is a foundation for good data management.
  • A well-organised project should be understandable to collaborators and to your future self.

Content from Choosing the Right File Structures

Last updated on 2026-07-31 | Edit this page

Overview

Questions

  • Why does folder organisation matter?
  • What makes a good folder structure?
  • How can I identify problems in an existing file structure?
  • How can I move, rename, and delete folders using a file browser?
  • How can I organise a project so that it is easier to understand and maintain?

Objectives

By the end of this episode, learners will be able to:

  • Navigate the file browser using a mouse.
  • Identify issues with folder names in a poorly organised project.
  • Rename folders using a graphical file browser.
  • Identify issues with folder organisation.
  • Move folders using a graphical file browser.
  • Delete unnecessary folders using a graphical file browser.
  • Design a folder structure that supports collaboration and reproducibility.

File Structures

In the previous episode we located the dataset that was handed over by our departed collaborator. Unfortunately, finding the data is only the first challenge.

The project structure itself is difficult to understand - the files include

legacy_dataset/
│
├── miscellaneous/
│   ├── cat_pic.jpg
│   ├── important_note.txt
│   └── Untitled.csv
│
├── New Folder
│   ├── big_file.txt
│   ├── Image One.png
│   ├── image_1.png
│   ├── image_10.png
│   ├── image_2.png
│   ├── image_8.png
│   ├── image_80.png
│   ├── image_81.png
│   ├── image_9.png
│   ├── image_93.png
│   ├── IMG_000.png
│   ├── IMG_001.png
│   ├── IMG_002.png
│   ├── ...
│   ├── IMG_018.png
│   ├── IMG_019.png
│   ├── June second image.png
│   ├── test.png
│   ├── Untitled 1.csv
│   └── New Folder (2)/
│       └── backup_copy.txt
│
├── penguins
│   ├── palmer_penguins.csv
│   ├── penguin958.jpg
│   └── penguin958.png
│
├── project_data_2025/
│   ├── data_final_v3_LAST_ONE.xlsx
│   ├── analysis_12-05-24.txt
│   ├── data_final_v4_revised_jerrys_comments.xlsx
│   ├── data_final_v2.xlsx
│   ├── script.py
│   ├── Alice_Jones_passport.pdf
│   ├── john_smith_passport.pdf
│   ├── script.py
│   └── analysis_12-05-24.txt
│   └── .processed/
│        └── REAL_ACTUAL_DATA_DO_NOT_DELETE.csv
│
├── processing_data
│   ├── 04-11-2025_penguin_data.csv
│   ├── 10-Nov-26_penguin_withtheirnames_and_weatherreport.txt
│   └── ...
│
└── RAW_DATA_!!!
    ├── Data!@#$.csv
    ├── 03_04_26_results.csv
    ├── raw_images_TEMP/
    └── ...

and more!

Imagine joining this project six months from now.

Could you quickly answer:

  • Where is the raw data?
  • Which files should be analysed?
  • Which files are temporary?
  • Which files should be preserved?
  • Which files can be deleted?

For most people, the answer is “probably not”. Even if details are stored in lab notes, these can easily be misplaced.

Before we worry about individual files, we need to make sure the overall project structure makes sense.

Why Folder Structures Matter

Most research projects start small with only a few files:

project/
├── data.csv
├── notes.docx
└── report.docx

At this stage organisation may seem unnecessary. However, projects rarely stay this simple. More data is added or revised, code is written, plots are generated, and papers/reports/theses are written… and revised.

Callout

As projects grow we often accumulate:

  • Raw data
  • Processed data
  • Analysis outputs
  • Figures
  • Documentation
  • Draft manuscripts
  • Collaborator comments
  • Software scripts
  • Temporary files
  • Meeting minutes
  • Downloaded software
  • Software or equipment manuals
  • Various notes
  • Talks and presentations

A project that starts with three files may eventually contain hundreds or thousands.

Without a clear structure, researchers often experience:

  • Time wasted searching for files
  • Duplicate work
  • Confusion over what files contain
  • Difficulty onboarding collaborators
  • Difficulty reproducing previous work

Good folder structures make projects easier to:

  • Navigate
  • Understand
  • Collaborate on
  • Maintain over time

A useful rule of thumb is

A collaborator should be able to understand where files belong without asking you.

Callout

The best practice is to plan for expansion rather than reorganising files later, which can lead to incorrect file paths and references in files such as software scripts or lab notes.

Establish a possible file structure early in the project, while being flexible if new folders or files need to be added or moved.

Some of this organisation depends on personal preference. For example,

  • Do you want to store a seminar or conference presentation with the project, or with other talks elsewhere?
  • Should a paper, along with generated figures, be kept together with the rest of the project?

If these files are stored elsewhere, can you to find the origin of figures and other results when you write a paper in six months time? If you move a folder, will your file references still work?

Challenge

What’s Wrong With This Structure?

Working in groups of 2–3:

Spend 5 minutes exploring the inherited dataset.

Identify as many problems as you can.

Consider:

  • Folder names
  • Folder organisation
  • Folder depth
  • Duplication
  • Clarity
  • Consistency

Write down your observations.

After 5 minutes, bring the class back together and discuss.

Common observations often include:

  • Default names such as New Folder
  • Unclear names such as miscellaneous
  • Use of special characters (RAW_DATA_!!!)
  • Hidden data locations
  • Mixed file types within the same folder
  • Folders whose purpose is unclear
  • Temporary files mixed with project files

Principles of Good Folder Structures

There is no single correct folder structure because different projects have different requirements.

However, most successful structures share common characteristics.

  • Meaningful
  • Consistent
  • Shallow
  • Categorical
  • Able to archive old files
Discussion

Common Characteristics

After looking at the Poor examples in each subsection below, try and come up with some good examples before clicking the Good tab.

Meaningful

Folder names should communicate purpose. For example,

Callout

If somebody joined the project tomorrow, would they understand what belongs in this folder?

Consistent

Consistency is more important than perfection. Choose a particular style and use it throughout the project. This approach makes the structure easier to understand because all folders follow the same convention. For example,

Shallow

Deeply nested structures can become difficult to navigate.

Avoid situations like:

Project/
└── Data/
    └── NewData/
        └── New Folder/
            └── Updated/
                └── Updated Again/
                    └── Results/

Having to click through many layers increases the chance of losing track of files.

As a general guideline:

  • 3–4 levels deep is often sufficient
  • Create more folders horizontally before creating more levels vertically

Avoid using the folder hierarchy to encode too much information. For example

Project/
└── Data/
    └── Raw/
        └── 2025/
            └── 05/
                └── 06/
                    └── 01/
                        └── Newcastle/
                            └── A_roads/
                                └── UserID12345/
                                └── UserID53546/
                            └── Minor_roads/
                                └── UserID35790/
                                └── UserID53546/
                        └── Gateshead/
                    └── 02/
                        └── Sunderland/
                    └── 03/
                        └── Results/

At best, this approach results in a lot of additional clicks in a file explorer or typing in a terminal/scripts. It can also make analyses unwieldy if you need to collect data from many different subfolders (e.g., if you want to analyse all of Newcastle’s data from the example above).

At worst, file metadata, such as city and datetime in this example, can be irrevocably lost if files are moved from their original location. As such, metadata should ideally be stored in the files themselves or in a reference table with the filenames, rather than in the folder structure.

Categorical

A common source of confusion is mixing different types of data and files together so that plots, data, reports, analyses, and scripts are all stored in the same folder. Instead, group related materials in the same folder.

This structure makes it easier to know where new files belong and find existing files, even if you’re new to the project.

Caution

Know where any private/secure data is to minimise the chance of GDPR violations.

Archive Rather Than Hoard

Many researchers hesitate to delete anything. As a result, they accumulate folders full of obsolete material. Instead, consider creating an archive folder:

archive/

This approach allows old material to be retained without cluttering the active project structure.

We will discuss data retention in a later episode.

Moving Files and Folders

Now that we know what good structures look like, we need the practical skills to improve existing structures.

The first operation is moving content.

How to Move a Folder

Renaming Folders

Renaming is one of the simplest improvements we can make to the file organisation. However, be aware that renaming can have unintended consequences if existing files depend on those names; for example, you may need to update folder references in the project’s documentation or code.

Discussion

Challenge

Choose a better name for the folder in the Poor example, then see one potential better name in the Good tab.

How to Rename a Folder

Caution

Be careful when renaming folders that are used by software or analysis scripts. Changing a folder name may break links elsewhere in a project.

Additionally, project documentation may become out-of-date and confusing if it references the original folder structure.

Deleting Unnecessary Folders

Sometimes the simplest improvement is removing unused material.

Consider:

New Folder (2)

If this folder is empty or no longer required, it may be reasonable to remove it.

How to Delete a Folder

Data deleted using the above methods are not fully removed from the system; they are first moved to the recycle bin. You can permanently delete the file by emptying the bin. Alternatively, files in the recycle bin can be restored and “put back” to their original location. On some systems, the bin permanently deletes files after they have been in the bin a certain amount of time (e.g., 30 days).

Caution

Check your data policies on data deletion, particularly if handling private data and your files are not encrypted. We will discuss data security in a later lesson.

Deleting a file on your computer will not always remove the file from automated backups (e.g., files stored in Newcastle University’s Microsoft OneDrive); this behaviour will depend on your syncing settings. Confirm that these settings support the backup behaviour you need for your project.

Designing a Better Structure

At this stage we know:

  • What good folder names look like
  • How to move folders
  • How to rename folders
  • How to remove unnecessary folders

The next step is deciding what folder structure you need for your project.

Challenge

Choosing a Folder Structure

For one of your projects, consider possible useful folders and how you would arrange them. What folder structures would work well? How would you divide up files?

Try drawing out three possible folder hierarchies for your project and discuss the pros/cons of the different approaches with your neighbours.

There is no one correct answer. It will vary depending on your project or personal preferences. One possible structure might be:

project/
├── data/
│   ├── raw/
│   └── processed/
│
├── analysis/
│
├── documentation/
│
├── outputs/
│
└── archive/

Other possible folders would be:

  • code
  • figures folder inside the analysis folder or outputs folder
  • temp for temporary files you know you do not wish to keep etc.

The most important part of the design is that each folder has a clear purpose.

Discussion

What Else Would You Change?

Looking at the inherited dataset:

  • Which folders would you rename?
  • Which folders would you move?
  • Which folders would you remove?
  • Are there any new folders you would create?

Compare ideas with a neighbour.

Challenge

Repair the Project Structure

Working individually or in pairs:

  1. Rename unclear folders.
  2. Move folders into more logical locations.
  3. Remove any unnecessary empty folders.
  4. Create a structure that clearly separates:
    • Data
    • Documentation
    • Analysis
    • Outputs

You do not need to create a perfect structure.

Focus on making the project easier to understand.

There is no single correct solution.

A reasonable result might:

  • Rename RAW_DATA_!!! to raw_data
  • Replace miscellaneous with more specific folders
  • Remove unnecessary empty folders
  • Create dedicated locations for documentation and analysis
  • Reduce unnecessary nesting

Looking Ahead

We have improved the folder structure, but many files still have confusing names.

In the next episode we will focus on file naming conventions and explore how good file names make datasets easier to understand and process.

Key Points
  • Folder structures should help people find and understand data.
  • Use meaningful, consistent folder names.
  • Separate different types of project content into dedicated folders.
  • Avoid excessive nesting.
  • File Explorer allows folders to be moved, renamed, and deleted without using the command line.
  • Consistency is usually more important than any specific organisational scheme.
  • A good folder structure should be understandable by collaborators and by your future self.

Content from Naming Files Well

Last updated on 2026-07-31 | Edit this page

Overview

Questions

  • What makes a good filename?
  • How much information should be included in a filename?
  • Why are naming conventions important?
  • How can filenames support batch processing and automation?
  • How can I rename one file or many files using a graphical file browser?

Objectives

By the end of this episode, learners will be able to:

  • Identify problems with poorly named files.
  • Explain the characteristics of a good filename.
  • Choose an appropriate file naming convention.
  • Use filenames to support sorting and batch processing.
  • Rename an individual file using a graphical file browser.
  • Rename multiple files using bulk rename tools.
  • Apply a consistent naming convention to an existing dataset.

From Folder Names to File Names

In the previous episode, we improved the folder structure of the inherited project. However, fixing the folders has not solved all of our problems.

Consider the files we have been given:

Data!@#$.csv
Untitled.csv
copy of DSCO1023.png
03_04_26_results.csv
results-new.csv
data_final.xlsx
data_final_v2.xlsx
data_final_v3_LAST_ONE.xlsx

At best, these names are unhelpful. At worst, they actively prevent us from understanding the project or keeping track of progress, particularly when looking back on the work done months later.

Callout

Imagine receiving an email that says:

“Can you update the analysis using the latest data?”

Which file would you use?

data_final_final.xlsx
data_final_final_last_one.xlsx
data_final_final_last_one_test.xlsx
data_final_v21.xlsx
data_final_v2a.xlsx
data_final_v4_revised_jerrys_comments.xlsx

The answer is not obvious.

You do not want to resort to sorting the files by date and hoping that the last created or modified file was the last good file!

Good filenames should give us insight into what a file contains without needing to open it. Your choice of filenames should make subsequent analysis steps as frictionless as possible. If you find you are fighting your files in order to make progress in your work, there is probably something that can be improved. You have to be able to find your data in order to use it!

Your choice of filenames becomes increasingly important as projects grow larger, data is shared between collaborators, or data needs revisiting at a later date.

Discussion

Working in pairs, look through the files in the inherited project. What problems can you identify?

Consider:

  • Can you tell what the file contains?
  • Can you identify the newest version?
  • Are dates presented consistently?
  • Would the names be easy to process automatically?
  • Are there any confusing abbreviations?

Write down as many issues as you can find.

After a few minutes, discuss observations as a group.

Callout

Common issues with file names include

  • Meaningless names (data.csv)
  • Overly long names (1Nov16_polymer_test1_heating_123degree_measure14_userabc_id12345.txt)
  • Special characters (image#1.png)
  • Multiple date formats (23-Nov2024.csv, 2024-15-07.txt)
  • Ambiguous version information (data1bfinalfinal2reallyfinalusethis.txt)
  • Inconsistent separators (10-12_2019-data_ver1.img.csv) - although different separators can be used to split different types of information (e.g., 10-12-2019_data_ver1_img.csv consistently uses - to separate the date elements and _ to separate other text)
  • Difficult to parse names due to lack of separators (test1userabsnovember17.h5)
  • Duplicate information (24Nov_image01_241117.dat)

Why Filenames Matter

A filename serves several purposes simultaneously.

It helps answer

  • What is this file?
  • When was it created?
  • Which sample or dataset does it belong to?
  • Which version is this?
  • Can it be grouped with related files?

Good filenames support

  • Human understanding
  • Sharing and collaboration
  • Automated processing
  • Reproducibility

A filename should be easily useable by both a human reader and a computer. Keep in mind that humans and computers are good at different things and that different software and operating systems (Windows, macOS, Linux) have different conventions. Your filenames should be as portable as possible.

Future-you is one of the most important users of your naming convention, but it should also be easily understandable by colleagues.

Renaming a Single File

Now that we understand good naming principles, we need practical ways to apply them.

Discussion

Rename an example file

In our inherited dataset, find

Data!@#$.csv

A more descriptive name might be

2026-04-03_incidence_data_v01.csv

We can rename it via:

Characteristics of Good Filenames

Callout

There is no universal naming convention.

It is impossible to build an understandable, concise, informative template that accounts for all possible scenarios. Different projects require different information. However, successful file naming schemes usually share several characteristics.

Filenames should be

  • Meaningful
  • Consistent
  • Machine-friendly
  • Sortable
  • Clear

Meaningful

A filename should describe the contents of the file, for example:

The ‘good’ example immediately communicates useful information about the data: it measures the water quality at siteA on the 3rd April 2026. This name can be used by a human to easily find the desired data in a folder containing many files and reference it in code or documentation. For a computer, this filename information can help software load and sort the data without opening the document and reading its content. The ‘poor’ filename communicates almost nothing.

Consistent

Choose a convention and use it across all files.

Even if the files contain the same information, inconsistent naming makes them harder to understand and automatically analyse. If the folder contains many files, how would you be able to find a specific piece of data? For automation, the good filenames each contain the data’s date and sample id, allowing a script to easily loop through and find each file.

Machine-Friendly

Computers prefer predictable patterns.

Recommended characters:

  • letters
  • numbers
  • hyphens (-)
  • underscores (_)

Avoid:

- special characters: \ / : * ? " < > |
- spaces

These characters can have special meanings on different operating systems. Hyphens can cause issues because different word processors can substitute different characters which look almost the same. Similarly, word processors can autocorrect lower case letters into uppercase characters at the start of lines.

Caution

Unlike Linux and macOS file names, Windows file names are case insensitive. This difference can cause data loss when copying files from Linux or macOS to Windows file systems.

Sortable

Since files are usually displayed alphanumerically, choose file names that logically order your files.

Using ISO date format (YYYY-MM-DD) means the files will be grouped from first to last date. This organisation is more useful than sorting by day first, then month within that day, which instead sorts files by the day of the month:

01-02-2025_recording.csv
01-03-2025_recording.csv
01-05-2024_recording.csv
01-05-2023_recording.csv

which is usually, but not always, less useful.

When planning your file names, consider how alphabetical sorting will arrange your files. Place the most important metadata for grouping files (such as the date, subject, country, or sample) at the start of the file name.

Discussion

Ordering your files

Consider the metadata you collect for one of your projects. What metadata would you put first in your filenames to help sort your data files?

Clear

Filenames should be easily understood by humans and computers.

Separators provides breaks for the eye and for parsing using software.

What Information Should Go In a Filename?

Many researchers face a common temptation:

If information is useful, why not put all of it in the filename?

This can lead to names such as:

2026-06-01_12-43-16_sampling_trip_siteA_temperature_sensor_03_processed_final_v2.csv

which becomes difficult to read, particularly in a long list of files with similar names. Additionally, some operating systems and applications limit the allowed length of filenames and file paths.

Acronyms can make the file name smaller, but you don’t want to rely on a set of definitions taped to your office wall to remember them!

A useful guideline is:

Include enough information to identify the file, but not so much that the filename becomes documentation.

The most useful information might include:

  • date
  • sample identifier
  • location
  • experiment identifier
  • version

For example:

2026-06-01_siteA_sample03_v01.csv

Additional details can either be stored elsewhere:

  • README files
  • metadata files
  • data dictionaries
  • laboratory notebooks

or as a header or metadata in the file itself. Additionally, avoiding storing metadata only in the filename, as that information will be lost if the filename is inadvertently changed.

We will discuss documentation in the next episode.

Naming Files for Batch Processing

Consistent filenames allows files to be easily be processed together.

The files in the good example can be processed together by software and scripts, while files in the poor example are much harder to work with systematically.

When files may belong to a sequence, also use consistent numbering with leading zeros:

Leading zeros preserves the numeric order:

Callout

Remember to use enough digits for the expected total number of files. For example, if you expect 99 files or fewer, pad with one zero to format numbers as 01, 02,... 99. If you may have 100 to 999 files, use 001, 002,... 999.

Discussion

Choosing the number of digits

Consider the different types of files you create for one of your projects. Which files might be part of a larger sequence? How many files do you expect in each sequence, and how many digits should you include in the filenames for each one?

Bulk Renaming Files

Renaming one file is easy, but renaming hundreds is not.

Imagine receiving

IMG_001.png
IMG_002.png
IMG_003.png
...
IMG_125.png

from a collaborator.

Renaming each file individually would be extremely time consuming (and boring!).Fortunately, many operating systems provide ways to rename multiple files simultaneously.

For example:

IMG_001.png
IMG_002.png
IMG_003.png

might become:

siteA_image_01.jpg
siteA_image_02.jpg
siteA_image_03.jpg

Tools for bulk renaming:

All of these tools support replacing text as well as adding prefixes, suffixes, and numbering across multiple files simultaneously.

Challenge

Bulk renaming

Using a bulk rename tool, replace

IMG

with

siteA_image_
Challenge

Challenge

What could go wrong if a bulk rename operation is performed incorrectly?

How could you reduce risk?

  • Test on copies first.
  • Rename a small sample initially.
  • Keep backups.
  • Check the preview before applying changes.

Using Hierarchies Effectively

Remember that your folder hierarchy already provides information.

Suppose we have:

project/
└── case_studies/
    ├── scotland/
    ├── england/
    └── wales/

Inside each folder:

2026-06-01_image_01.png

is sufficient.

Avoid naming files:

scotland_2026-06-01_image_01.png

For example:

The location information is already present in the folder structure and so is redundant in the file name.

This example illustrates how each project has different filename requirements. If the ‘country’ layer of the directory structure wasn’t present, it would be useful to put the country in the filename. However, if you move files, the folder names would need to change; for example

scotland/2026-06-01_image_01.png
england/2026-07-01_image_01.png

would need to be renamed if placed in the same folder to avoid data loss.

Callout

Good folder structures and good filenames should work together.

Versioning Files

New versions of files can arise from repeated experiments, fresh downloads of data, and documentation or report edits. For some types of changes, you’ll want to keep non-destructive revisions to documents or an audit trail of changes. Versioning files keeps track of these changes over time, allowing you to manage different versions of files without overwriting previous ones.

Due to a lack of a versioning strategy, many researchers create filenames that making tracking revisions difficult:

The bad example is difficult to follow, while the good example uses explicit versions.

Avoid:

  • latest
  • final
  • newest
  • final_final
  • use_this_version

because these labels eventually become inaccurate. As soon as you edit report_final_final.docx it is no longer final (or even final final!), and you probably won’t go back to rename the earlier versions.

Callout

Various software tools can help with versioning without using multiple files. For example:

  • git for code, documents, and small files (commonly used via GitHub)
  • Microsoft Office (version history tools)
  • OneDrive
Challenge

Rename the inherited files

Working individually or in pairs:

Review the filenames in the inherited dataset and apply the principles discussed in this lesson.

Consider:

  • Meaningful descriptions
  • Consistent formatting
  • ISO dates where appropriate
  • Explicit version numbers
  • Removal of unnecessary special characters

Rename the files using File Explorer/Finder.

If a bulk rename tool is available, use it where appropriate.

There is no single correct answer.

A reasonable outcome might be

or

These different options are all good provided the convention is applied consistently throughout the project.

Looking Ahead

We have improved both the folder structure and the filenames.

However, many questions remain unanswered:

  • Where did the data come from?
  • How was it collected?
  • What do variables mean?
  • Which files should be used?

This information belongs in documentation.

In the next episode we will explore how README files, metadata, file headers, and provenance information make datasets understandable and reusable.

Key Points

Key Points

  • Good filenames help both people and computers understand data.
  • Consistency is more important than choosing a particular naming convention.
  • Use meaningful names that describe file contents.
  • Avoid spaces and special characters where possible.
  • Use ISO dates (YYYY-MM-DD) when dates are needed.
  • Include version numbers rather than labels such as “final” or “latest”.
  • Folder structures and filenames should work together.
  • Bulk renaming tools make large-scale renaming practical.
  • A filename should help identify a file without needing to open it.

Content from Documenting Your Data

Last updated on 2026-07-28 | Edit this page

Overview

Questions

  • How can I remember what my data is and where it came from?
  • What information should I document about a project?
  • Where should documentation be stored?
  • What is a README file?
  • How can I create project documentation using a graphical file browser?

Objectives

By the end of this episode, learners will be able to:

  • Explain the purpose of research data documentation.
  • Identify information that should be captured about a dataset.
  • Create a README file using a graphical file browser.
  • Document project structure, provenance, and data sources.
  • Identify situations where additional documentation is needed.
  • Improve the reusability of a dataset through documentation.
  • Use file headers to store key metadata.

How Do I Remember and Share Where My Data Is Stored?

In the previous two episodes, we improved our folder structure and renamed our files.

The project now looks considerably better than it did when we inherited it.

However, several important questions remain unanswered:

  • What does this dataset contain?
  • Who collected it?
  • When was it collected?
  • How was it collected?
  • What do the variables mean?
  • What units were used?
  • Which files are raw data?
  • Which files were generated during analysis?
  • What changes do different versions capture?
  • Which version should be used?
  • Which files relate to which other files?
  • When was the data downloaded?
  • Where was the data downloaded from?
  • What software and versions of software were used to analyse the data?

Imagine returning to this project six months from now. Would you still remember all of these details? What about three years from now? What if somebody else joins or takes over the project?

Good folder structures and filenames help people navigate data. Documentation helps people understand data.

Challenge

What Makes Data Easy to Understand?

Think about a dataset, project, software package, or repository that you found particularly easy to use.

  • What made it easy to understand?
  • What information was available?
  • How was that information presented?
  • Was there a README file?
  • Was documentation easy to find?

Conversely:

  • Have you ever received poorly documented data?
  • What information was missing?

Discuss responses with your neighbour.

Helpful documentation practices include

  • Clear explanations
  • Consistent organisation
  • Readily available information
  • Definitions of terms and variables
  • Information about data collection
  • Contact details

Documentation Is For Humans

When discussing data management, people often focus on technology:

  • Storage
  • File formats
  • Software
  • Analysis tools

However, one of the biggest barriers to reusing data is simply understanding it. Good documentation reduces the number of assumptions that somebody must make when using a dataset.

Callout

The most important collaborator you will ever have is often future you.

Documentation helps future you understand decisions that seem obvious today, but may be forgotten later.

What Is Metadata?

Callout

Metadata is data about data.

Metadata provides context that helps us understand a dataset.

For example, consider the file:

incidence_data.csv

The filename tells us very little.

Useful metadata might include:

  • What the dataset contains.
  • Who collected it.
  • When it was collected.
  • How it was collected.
  • Units of measurement.
  • Data quality or collection issues.
  • Licensing information.
  • Contact details.
  • Link to the original source of the data if using an external source of data.

Metadata helps transform a collection of numbers into something meaningful.

Some software autogenerates some metadata for you.

For example, Microsoft Word produces

  • Author
  • Company
  • Template
  • Date Created
  • Date Modified
  • Last Saved By
  • Revision number
  • Total editing time
  • etc…

by default. Other software will store different metadata.

Caution

Automated metadata may add information you do not wish to share to the file. Ensure you do not inadvertently share sensitive or private metadata when distributing data.

Microsoft Office provides tools for removing automated metadata if required.

What Should We Document?

Given the huge variety of research and data, there is no universal documentation standard suitable for every project. However, most project-level documentation should include

  • Project description
  • Dataset descriptions
  • Data provenance
  • Data use and analysis
  • Contact details
  • README file(s)

Some projects may also include

  • Data dictionaries
  • Standard operating procedures
  • Project logs
  • Parameter files

These elements will be expanded on below.

Callout

Research should be reproducible. Given the information in the documentation and an accompanying paper or thesis, would someone else be able to repeat your research?

Would future you be able to repeat your research?

Project Description

Provide a short description that summarises the project’s goal; for example,

This project investigates disease incidence in marine mammals
along the Scottish west coast between 2020 and 2025.

A new collaborator should understand the purpose of the project within a few seconds.

It may be tempting to skip this description and instead rely on a long project folder name such as

marine_mammals_disease_scotland_westcoast_2020_to_2025

but this name is long and still ambiguous. Additionally, a project’s scope and goals change over time; what if your focus narrows to investigating seals or dolphins? It is easier to change the documentation than update folder names and references.

Data descriptions

Describe the available datasets and folders.

For example:

raw_data/
Contains original incidence records obtained from collaborators.

processed_data/
Contains cleaned datasets used in analysis.

analysis/
Contains scripts and outputs generated during analysis.

This documentation helps people understand how to navigate your files and the relationships between them.

Data provenance

Data provenance - the full record of a dataset’s origins and modifications - becomes increasingly important as projects grow, and certain projects may have strict rules for tracking providence. Some data sources include provenance information in file headers, while others store provenance in other locations.

If data sources are updated, for example to correct collection errors or accidental changes, it is important to keep track of the new and old files.

Challenge

Documenting data provenance

What things do you need to document to track data provenance for one of your projects?

Think about details you may need to - report in a paper or other publication - reproduce the work in future - assess or fix a data error that is discovered during the project

Possible information to document:

  • Was the data collected by your team? When? Where? Include links to other notes, such as lab manuals.
  • Was it obtained from a collaborator? When?
  • Was it downloaded from a repository? When?
  • Was it generated by software? Which version?
  • Were any corrections or other modifications made? Why, when, and by whom?

Data use and analysis

Many projects involve extensive data analysis. This analysis needs to be well-documented to ensure anyone with the same data can reproduce your results.

If you share your data, you also need to include any access conditions, such as restrictions for publishing or storing the data.

Challenge

Documenting your data analysis

Consider what information you should document on how you used and analysed the data during one of your project.

Useful documentation might include

  • Required software and their versions
  • Processing steps and settings
  • Important assumptions
  • Known limitations of your analysis
  • Access conditions - i.e., how to access the data and any restrictions on using it

These details can save future users significant amounts of time.

Contact details

Include contact information whenever possible, including for a primary contact, so that collaborators and other researchers can ask questions or raise concerns.

For example:

Project Lead (Primary Contact):
Dr Jane Smith

Email:
j.smith@example.ac.uk

Data Collection Lead:
Dr John Doe

johndoe@biguniversity.ac.uk

README File(s)

One of the simplest and most effective forms of documentation is a README file.

A README is usually placed at the top level of a project:

project/
│
├── README.txt
├── data/
├── analysis/
└── results/

It explains the project and helps people navigate it.

A good README provides enough information for someone to understand the project at a high level without opening any other files.

Callout

A README does not need to be perfect. A simple README containing a few useful paragraphs is significantly better than no README at all!

Try to keep README files up to date.

Keep files together with their READMEs if you exchange data with colleagues.

Example README Structure

Example template:

Project Name

Project Description

Project Structure

Data Sources

File Naming Convention

Software Requirements

Contact Information

Last Updated

For example:

Project Name:
    Marine Mammal Incidence Study

Project Description:
    Investigation of marine mammal health records collected
    between 2020 and 2025.

Project Structure:
    data/raw - original data
    data/processed - cleaned datasets
    analysis - analysis scripts
    results - generated outputs

Software Requirements:
    Data was analysed using Python 3.13
    See analysis/pyproject.toml file for Python package dependencies and versions

Contact:
    bob.badgerton@example.ac.uk

Last Updated:
    2026-07-01

Another example can be found here.

Challenge

Creating A README File

Follow along with the demo to make a README file for the inherited project.

1) Navigate To The Top-Level Folder

Open File Explorer.

Navigate to:

legacy_dataset/

2) Create A New Text File

Right-click in an empty area of the window and select

New → Text Document

to create a new file.

3) Rename The File

Rename the file README.txt.

4) Open The File

Double-click the file to open it in Notepad (or a similar text editor).

5) Add Basic Structure

Add headings such as

Project Name

Description

Folder Structure

Data Sources

Contact Information

Last Updated

Save the file. We’ll fill out the fields later in the lesson!

Documentation Beyond The Top Level

Project-level documentation is important, but subfolders sometimes require their own documentation. For example, a results folder may contain dozens of generated outputs and sub-folders. Several months later, someone may reasonably ask

  • Which software generated these files?
  • When were they created?
  • Which parameters were used?
  • Which version of the data was analysed?
  • What is the data access and retention policy of these data?

A README placed inside the results folder can answer these questions.

For example:

results/
│
├── README.txt
├── figures/
└── tables/

The README could document

Generated:
2026-07-01

Analysis Software:
R version 4.6

Input Data:
processed_data_v03.csv

Parameters:
Significance threshold = 0.05

to preserve important contextual information.

Caution

Any READMEs written by hand may also need to be manually updated as your project progresses, which risks the documentation becoming out-of-date. Where possible, automate your documentation (for example, by exporting settings from analysis tools or scripts).

Discussion

Consider one of your projects. What additional README files may you need beyond the top-level project README, and what information should they contain?

Other Useful Documentation

Data Dictionaries

These explain variables in tabular datasets.

For example

Variable Description
incidence_rate Number of cases per 100,000
region Geographic study area
year Observation year

We will revisit these dictionaries in the tabular data lesson.

Standard Operating Procedures

Standard operating procedures (SOPs) describe how data is collected or processed.

Project Logs

Project logs provide a record of major project decisions; they help record not only what you did, but why. This logic can be help justify methods in publications as well as warn future researchers about inefficient or unsuitable approaches that were ruled out in earlier project stages.

Parameter files

Many analyses are highly adjustable, with various different parameters, or settings, that can be changed depending on your needs. Parameter files record these settings to ensure that your data processing and analysis steps can be reproduced.

Challenge

Write A README

Fill out the README.txt file for the inherited project.

Include

  • Project name
  • Project description
  • Folder structure
  • Data sources
  • Contact information
  • Last updated date

Make reasonable assumptions; if you inherit a real, poorly-documented project, you’d probably need to read through old project emails/messages, find the original project/grant proposals, and talk to current lab members to get all the README information.

The goal is to make the project easier for a future collaborator to understand.

There is no single correct answer, but it should allow somebody unfamiliar with the project to begin understanding its contents.

Additional resources:

File-level documentation

Individual files often require additional documentation; for example, you may have many data files that were collected under different circumstances. Where possible, this data should be stored directly in the file in the file header or metadata. We will discuss approaches for storing headers later in the data formats lesson.

The advantage of storing metadata in a file is that it cannot be separated from the corresponding data. File names and folder names do not have this property since files can be easily renamed or moved to different folders. As such, avoid storing important metadata in file or folder names.

Discussion

Reflection

Imagine you are leaving your current project tomorrow. What information would a replacement researcher need to continue your work? Does your project’s current documentation include that information?

Caution

You may not always have much warning before temporarily or permanently leaving a project; plans can unexpectedly change due to new opportunities, emergencies, or illness. Similarly, new people may join a project without much notice and need to get up-to-speed quickly. Keeping your documentation up-to-date can make unplanned transition periods less stressful.

Looking Ahead

Our project now has

  • a clearer folder structure
  • more information file names
  • initial documentation

The next challenge is deciding where data should be stored so it is accessible, secure, shareable, and recoverable.

In the next episode we will explore how to choose appropriate storage locations for different types of research data.

Key Points

Key Points

  • Documentation makes data understandable and reusable.
  • Documentation is primarily written for people, including future-you.
  • Metadata provides context about a dataset.
  • README files are a simple and effective way to document projects.
  • A README should explain what the project is, how it is organised, and where the data came from.
  • Documentation should be stored close to the data it describes.
  • Additional README files can be useful within subfolders such as results or processed data.
  • Good documentation reduces confusion and improves collaboration.

Content from Choosing Where to Store Your Data

Last updated on 2026-07-31 | Edit this page

Overview

Questions

  • Where should I store my research data?
  • What are the advantages and disadvantages of different storage options?
  • How can I make data accessible to collaborators?
  • How can I ensure research data is backed up and secure?
  • What additional considerations apply to sensitive or restricted data?
  • How can I make data accessible when publishing research?

Objectives

By the end of this episode, learners will be able to:

  • Identify different storage options available for research data.
  • Explain the difference between local, network, and cloud storage.
  • Evaluate storage locations based on accessibility, backup, security, and sustainability.
  • Choose an appropriate storage location for different types of research data.
  • Explain why storing research data in personal accounts can create risks.
  • Recognise situations where specialist storage solutions may be required.
  • Identify options for publishing research outputs and datasets.

Where Should I Store My Data?

At this point in the lesson we have:

  • Organised our folder structure
  • Improved file names
  • Added documentation

However, another important question remains:

Where should this data actually live?

Many researchers begin a project by storing files wherever is most convenient, such as their work laptop, a USB drive, or personal cloud storage. This approach often works initially, but many research projects involve

  • Collaborators
  • Large datasets
  • Sensitive information
  • Long-term storage requirements

A storage decision that works for a single person may not scale as the project and team grows. Choosing an appropriate storage location is therefore an important part of good data management.

Challenge

Where Do You Store Your Data?

Spend a few minutes discussing your current storage set-up with a partner:

  • Where do you currently store your research data?
  • Why did you choose that location?
  • How do collaborators access it?
  • Are the data backed up?
  • What would happen if your laptop stopped working tomorrow?
  • What would happen if you suddenly left your current project?

Share some of your current solutions and any concerns with the group.

Researchers often store data on

  • Laptop hard drives
  • USB drives
  • OneDrive
  • Google Drive
  • Shared network drives
  • Institutional storage systems
  • HPC storage
  • External hard drives

What Makes a Good Storage Location?

When deciding where to store data, consider

  • Accessibility: the ability of different users to retrieve and use data
  • Recoverability: the ability to restore damaged or lost data
  • Security: the ability to protect data from unauthorised access or misuse
  • Sustainability: the ability to store and access data for long time periods

Can Other People Access the Data?

Collaborators must be able to find and open the data.

To evaluate data accessibility, consider

  • Can all project members access the data?
  • Can external collaborators access it?
  • Can access be limited to specific people?
  • Can access be easily added for new project members and removed for past members?
  • When you leave the project, will other people have difficulty accessing the data?

Is the Data Recoverable?

Data storage and data backup require different solutions. Many researchers assume that files on a computer are safe. However -

  • Computers fail
  • Laptops are stolen
  • Hard drives stop working
  • USB drives are lost
  • Data can be accidentally deleted or incorrectly modified

Good storage solutions should have reliable backup solutions so that lost data is recoverable.

Challenge

Making data recoverable

Discuss features of a good backup system in your groups.

Possible features of a good backup system:

  • Has regular, automated backups: automation ensures backup versions stay up-to-date; for e.g., backups may be scheduled every night or triggered when you connect to the internet.
  • Off-site storage: stores data off-site (e.g., on the cloud or in a different building) to protect against theft or physical damage due to, e.g., fires or floods.
  • Accessibility: allows you to easily restore lost data from the backups and find your files.
  • Redundancy: maintains multiple copies of your data to protect against accidental deletion.

Is the Data Secure?

Not all data can be shared openly; data that needs to be kept secure includes

  • Personal information (particularly special category data)
  • Commercially sensitive information, such as trade secrets
  • Confidential research

Storage locations must be appropriate for the sensitivity of the data. We’ll discuss these considerations further in the Personal and Confidential Data section.

Caution

You have a legal requirement to keep certain types of data safe; failure to do so can have consequences for yourself and your institution.

In the event of a data breach you are required to take certain actions, such as reporting to your institution. Make sure you know the requirements for your data.

Is the Data Storage Sustainable?

To ensure your data storage is sustainable over the lifetime of the project, consider

  • Will the storage still exist in five years?
  • What happens to the data if a team member leaves?
  • Can file ownership be transferred? This ability is particularly important for cloud services.
  • Is there enough storage capacity for data growth?
  • Is the file format accessible? If the company which makes your software shuts down, will you still be able to access your data?
  • Are any ongoing costs for data storage affordable?

Types of Storage

There are many different types of data storage:

  • Local storage
  • Personal cloud storage
  • Shared network or cloud storage
  • Institution storage (e.g., the Research Data Warehouse)

We’ll discuss the advantages and disadvantages of each one below.

Local Storage

Local storage refers to files stored directly on a device such as a

  • Laptop
  • Desktop computer
  • External hard drive
  • USB drive
Challenge

Local storage pros and cons

Discuss the advantages and disadvantages of local storage with a partner or group.

Advantages

  • Fast access
  • Convenient
  • Works without internet access
  • Good for active work

Disadvantages

  • Data may only exist in one location
    • Difficult to access off site
    • Difficult to share with colleagues
  • Vulnerable to device failure or loss
  • Difficult to collaborate
  • Difficult to manage project-wide access
  • Security issues, e.g., ‘left laptop with personal data on a train’
Callout

Local storage is often useful for temporary working files.

It is not a good long-term storage solution for important project data since it lacks all four desired storage features: accessibility, recoverability, security, and sustainability.

Personal Cloud Storage

Many institutions provide services such as

  • OneDrive
  • Sharepoint
  • Google Drive
  • Dropbox

These services can synchronise files between devices and provide data backups.

Newcastle University has guides for using OneDrive and Sharepoint storage.

Challenge

Personal cloud storage pros and cons

Discuss the advantages and disadvantages of cloud storage with a partner or group.

Advantages

  • Accessible from multiple devices
  • Easy and rapid syncing. OneDrive often works in the background.
  • Simple sharing
  • Automatic backup in many cases
  • Stronger security compared to email
  • Version history
  • Easier to share large datasets using links to the data

Disadvantages

  • Usually linked to an individual account
  • Access often disappears when somebody leaves
  • Shared permissions must be managed carefully
  • Storage may be limited
  • Security/Privacy issues
  • Data sovereignty issues
  • Potentially harder to provide access to external collaborators
  • Understanding University policy
  • Knowing which service to use can be challenging
  • May fail silently
Caution

Newcastle University discourages uploading personal or sensitive data to Google Drive and DropBox (see the cloud storage tab). It suggests Microsoft OneDrive as a more secure option.

Callout

A Common Research Problem

Many researchers store project data in their personal OneDrive. This approach seems convenient until

  • A researcher leaves the project
  • A researcher leaves the institution and their institution OneDrive account is removed
  • You need to transfer ownership of files

These situations can also result in loss of access to important project data.

In some organisations, governance policies may prevent administrators from accessing a past team member’s OneDrive because personal storage is treated as individual, rather than project-owned, data. For this reason, avoid using personal cloud storage as the primary home of important shared project data.

Depending on the cloud service, it can also be challenging to transfer file ownership to another team member.

Shared Network or Cloud Storage

Shared storage is designed specifically for collaboration.

Examples include:

  • Shared network drives
  • Shared project folders
  • SharePoint sites
  • Departmental storage
Challenge

Shared storage pros and cons

Discuss the advantages and disadvantages of shared storage with a partner or group.

Advantages

  • Multiple users can access data
  • Permissions can be managed centrally
  • Better suited to collaboration
  • Less dependent on a single individual

Disadvantages

  • May require administration
  • May have storage limits
  • Large datasets can become difficult to manage
  • May not have permission to access off-site
  • Can be more difficult to understand backup policy (e.g., is something deleted locally also deleted in the shared folder?)

Institution storage

Many institutions provide dedicated research storage services.

At Newcastle University, research projects can use the Research Data Warehouse (RDW).

The RDW is

  • Backed up
  • Designed for research data
  • Accessible to project members
  • Shared at the project level
  • Suitable for large datasets
  • Connected to institutional computing infrastructure

Unlike personal storage, access is linked to the project rather than an individual, which allows

  • New collaborators to be added
  • Departing collaborators to be removed
  • Data to remain available to the project

For most research data, RDW is the recommended storage location.

Discussion

Accessing Different Storage Locations

Follow along with the instructor who will demonstrate how to navigate to

  • Local storage
  • OneDrive (or equivalent cloud storage)
  • Shared project storage
  • Research Data Warehouse (or institutional research storage)

For each example:

  1. Open File Explorer.
  2. Navigate to the storage location.
  3. Show where files appear.
  4. Discuss who can access the data.
  5. Discuss backup arrangements.
  6. Discuss limitations.

Permissions and Collaboration

Choosing a storage location is only the first step for storing research data.

You also need to decide

  • Who can read data?
  • Who can upload data?
  • Who can edit data?
  • Who can delete data?

For example, you may need permissions such as

Role Permissions
Project members Read and write
External collaborators Read only
Project lead Full control

but the appropriate permissions depend on the project. We recommend creating an access plan so all project members know who can access which data and how to provide access to external or new collaborators.

Callout

Grant the minimum access necessary to reduce the risk of accidentally deleting, leaking, or modifying important data.

Permissions should be regularly reviewed to update access as needed (e.g., to remove access after team members or collaborators leave the project).

Personal and Confidential Data

Some projects require additional controls to protect personal data, such as names and and contact information, as well as confidential data, such as trade secrets or intellectual property.

Special category data, or sensitive data, is personal data that requires additional protection due to its sensitivity - for example, race and ethnic origin, political opinions, religious beliefs, biometric data, and health data.

When working with these types of data, consider

  • Is the storage location approved for personal or confidential data?
  • Who should have access to the data?
  • Does the data need to be encrypted?
  • Are there legal or ethical restrictions for using or distributing the data?

Personal and confidential data should never be collected or stored solely because it is convenient. Always follow institutional policies and governance requirements; for example, Newcastle University requires researchers to complete a Data Protection Impact Assessment if their research project may involve personal data.

Caution

There are strict legal requirements for using and storing certain types of data:

  • People can request access to data held on them, via Subject Access Requests
  • People can request erroneous data about them be corrected
  • People can request data about them be deleted
  • Personal data must be used only for the reasons it was provided
  • Data breaches must be reported to the University in line with its policies
  • and many others

For more information, see

Ensure you understand your obligations for managing your data.

Encryption

One way to protect personal and confidential data is encryption, which “scrambles” data so that only people with the correct key can access and read the data.

Your institutional laptop may already be running whole disk encryption, which protects all the files on the computer. Without encryption, anyone can access the files, even if you have set a password. However, data on your laptop is only fully protected when the laptop is shut down; different modes of standby have different levels of protection. For example, Hibernate provides more protection than Sleep mode in Windows.

External hard-drives (often used for backups or data transfer) need to be explicitly encrypted, even if your laptop is encrypted. External disks can be encrypted using Bitlocker in Windows, Finder in macOS, or Disk Utility in Linux.

Caution

If you forget your password and encryption key you have lost your data! It cannot be recovered.

The different tools on Windows, macOS, and Linux can make an encrypted external disk or USB key unusable on a different operating system. You can often encrypt individual files and folders if required.

Discussion

Where Should You Store Your Data?

Consider a current or future research project.

  • Where are you storing data now?
  • Is that the most appropriate location?
  • How is the data backed up?
  • Could collaborators access it?
  • What would happen if you left the institution?

Would you change your storage solution after today’s discussion?

Changing Data Storage Location

Storage locations may need to change as projects evolve.

When moving project data

  • Inform collaborators
  • Update documentation
  • Update links, references, and shortcuts
  • Update analysis workflows if required

Changing locations can have unintended consequences.

For example:

may break scripts that reference the previous location.

A Brief Note on Version Control

Sometimes researchers use multiple files with descriptive filenames to keep track of changes:

report_v01.docx
report_v02.docx
report_v03.docx

For documents and data, this approach can be useful.

However, software projects often use dedicated version control systems such as Git.

Version control provides

  • History tracking
  • Collaboration tools
  • Change management
  • Recovery of previous versions

Note that some services, such as OneDrive, also retain version histories.

Callout

We will not cover version control and Git in this workshop, but you can learn more about them by attending a Carpentries Git and Version Control workshop.

Sharing Research Outputs

Eventually many research projects produce outputs that should be publicly available, such as

  • Supporting datasets
  • Processed data
  • Code
  • Supplementary materials

Rather than emailing files or placing them on personal websites, researchers often store these outputs in repositories that are publicly available, such as

  • Zenodo
  • Figshare
  • GitHub (particularly for code/software)
  • Institutional repositories
  • Subject-specific repositories

More repositories can be found at re3data.org and in Newcastle University’s Find Data resource. Note that individual funders may have their own requirements on how research outputs are shared.

Repositories can provide

A DOI allows any type of digital object to be cited in publications, making research outputs such as datasets easier to find, cite, and reuse. Journals create DOIs for research articles as well to provide a stable citation and metadata for each article.

Challenge

Finding digital objects using DOIs

DOIs can be mapped to their digital objects using a DOI resolver. The resolver looks up the DOI in the DOI registry to find the corresponding record and send you to the location of the digital object.

Use the DOI Foundation’s DOI resolver to find the digital object that corresponds to each of these DOIs:

  • 10.5281/zenodo.3960218
  • 10.1371/journal.pone.0090081

For each DOI, answer

  1. Where is the object stored?
  2. What type of object is it? (e.g., publication, dataset, software)
  3. What is the name of the object?
  4. Who created/authored the object?
  5. When was the object published?
  6. How would you cite the object?

10.5281/zenodo.3960218

  1. Location: Zenodo (repository)
  2. Type: Software (with dataset)
  3. Name: allisonhorst/palmerpenguins: v0.1.0
  4. Creators: Allison M Horst, Alison Presmanes, Kristen B Gorman
  5. Publication date: July 25, 2020
  6. Citation: Allison M Horst, Alison Presmanes Hill& Kristen B Gorman. (2020). allisonhorst/palmerpenguins: v0.1.0 (Version v0.1.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.3960218

10.1371/journal.pone.0090081

  1. Location: PLOS One (journal)
  2. Type: Research article
  3. Name: Ecological Sexual Dimorphism and Environmental Variability within a Community of Antarctic Penguins (Genus Pygoscelis)
  4. Creators: Kristen B. Gorman, Tony D. Williams, William R. Fraser
  5. Publication date: March 5, 2014
  6. Citation: Gorman KB, Williams TD, Fraser WR (2014) Ecological Sexual Dimorphism and Environmental Variability within a Community of Antarctic Penguins (Genus Pygoscelis). PLoS ONE 9(3): e90081. https://doi.org/10.1371/journal.pone.0090081
Challenge

Matching Data to Storage

For each scenario, identify the most appropriate storage location and explain your reasoning.

  1. Active analysis files that only you are currently editing.
  2. A shared project involving five collaborators.
  3. Large datasets processed on the HPC.
  4. Personal identifiable information.
  5. Published research outputs accompanying a journal article.

Discuss your answers with a partner.

Example answers:

  1. Local working copy with institutional backup.
  2. Shared project storage or Research Data Warehouse.
  3. Research Data Warehouse connected to HPC resources.
  4. Approved secure institutional storage.
  5. Public repository such as Zenodo or an institutional repository.

The key consideration is matching storage decisions to requirements around access, backup, scale, and security.

Looking Ahead

We now know where research data should be stored.

The next challenge is dealing with situations where data is challenging to store or share.

In the next episode we will discuss

  • Storage limits
  • Compression
  • File transfer
  • Storage efficiency
  • Reducing duplication
Key Points
  • Storage decisions should consider accessibility, recoverability, security, and sustainability.
  • Local storage is convenient for active work, but is rarely meets other storage requirements.
  • Personal cloud storage can create access problems when researchers leave projects.
  • Shared project storage is generally preferable to individual-owned storage.
  • Research data should ideally be stored in project-owned, backed-up systems.
  • Permissions should be managed carefully and reviewed regularly.
  • Personal and confidential data may require additional controls and approved storage locations.
  • Updating data locations can affect collaborators and workflows.
  • Repositories such as Zenodo can make research outputs more discoverable and citable.
  • Version control is important for managing changes and is covered in dedicated Carpentries Git workshops.

Content from Storing and Transferring Data Efficiently

Last updated on 2026-07-31 | Edit this page

Overview

Questions

  • Why do some datasets take up more storage than others?
  • How can I measure the size of my data?
  • What is file compression and when should I use it?
  • When is changing file format an appropriate solution?
  • What are the best ways to transfer data between collaborators?
  • How can I reduce storage requirements without losing important information?

Objectives

By the end of this episode, learners will be able to:

  • Determine the size of files and folders using a graphical file browser.
  • Explain what file compression is and how it works.
  • Compress files and folders using common desktop operating systems.
  • Identify situations where compression is useful and where it is not.
  • Recognise that file formats can affect storage requirements.
  • Evaluate different methods for transferring data.
  • Choose an appropriate transfer method based on data size and project requirements.

The Problem: The Data Is Too Big

Over the last few episodes we have:

  • Organised the folder structure
  • Improved file names
  • Added documentation
  • Chosen an appropriate storage location

The project is now much easier to understand.

Unfortunately, a new problem has appeared.

Your supervisor asks:

“Can you send the cleaned dataset to our collaborator?”

The collaborator has an older lab computer that:

  • Has very little free storage space
  • Has a slow internet connection
  • Cannot easily accept very large files by email

You need to make the dataset as small and transferable as possible without damaging the data.

This situation is common in research.

As projects grow, researchers often encounter:

  • Storage quotas
  • Slow file transfers
  • Limited network bandwidth
  • Data archives that are too large to share easily

Before we can solve these problems, we need to understand how data size is measured.

Challenge

How Could We Make The Data Smaller?

Working in pairs:

What options can you think of for reducing the amount of storage used by a dataset?

Consider:

  • Temporary files
  • File formats
  • Compression
  • Duplicate data
  • Images and media

Write down as many ideas as possible.

After a few minutes, discuss responses as a group.

Common answers may include:

  • Compressing files
  • Deleting duplicates
  • Removing temporary files
  • Choosing more efficient file formats
  • Archiving old outputs
  • Resizing images
  • Removing unnecessary intermediate results

Understanding File Size

Before making data smaller, we need to know:

How large is it now?

In data management we often measure storage using units such as:

KB  (Kilobyte)
MB  (Megabyte)
GB  (Gigabyte)
TB  (Terabyte)

Each step represents a large increase in storage requirements.

For example:

1,000 MB ≈ 1 GB
1,000 GB ≈ 1 TB

As datasets grow, even small improvements can translate into large storage savings.

What Does File Size Represent?

Every file occupies space on a storage device.

Files are ultimately stored as binary data on:

  • Hard drives
  • Solid state drives (SSDs)
  • Network storage
  • Cloud storage

When File Explorer reports that a file is:

250 MB

it is telling us approximately how much storage capacity that file occupies.

Larger files:

  • Require more storage
  • Take longer to transfer
  • Take longer to back up
  • May exceed attachment limits

Understanding file sizes helps us make informed decisions.

Challenge

Viewing File Sizes

The instructor will demonstrate, and you can try it on your own system. Which is the largest file in the example data?

It is big_file.data

File Compression

One of the most common ways to reduce storage requirements is compression.

What Is Compression?

Compression reduces the amount of storage required by a file without changing its contents.

It works by storing information more efficiently.

For example:

AAAAAAABBBBBBBBBCCCCCCCC

contains repeated patterns.

A compression tool can represent these patterns more efficiently than storing every character individually.

The result is a smaller file.

Lossless and Lossy Compression

There are two broad forms of compression.

Lossless Compression

Lossless compression preserves all information.

After decompression:

Original File == Restored File

Examples:

  • ZIP
  • GZIP
  • 7Z

These are commonly used for research data.

Lossy Compression

Lossy compression achieves greater size reduction by discarding information.

Examples:

  • JPEG images
  • MP3 audio
  • Some video formats

The resulting file may be smaller, but some original information has been lost.

For research data, lossy compression should be used with care. See LINK

Demonstration: Compressing A Folder

When Should You Compress Data?

Compression is often useful when:

  • Archiving data
  • Transferring data
  • Emailing files
  • Backing up projects

Examples:

project.zip
results.zip
archive.zip

Compression can significantly reduce transfer times.

When Should You Avoid Compression?

Compression is not always beneficial.

For example:

  • Actively edited files
  • Frequently accessed files
  • Files already compressed

Many formats already include compression.

Examples:

jpg
png
mp4
mp3
pdf

Compressing these files often produces very little size reduction.

Callout

If a file format is already compressed, zipping it again may provide almost no additional benefit for size reduction. It can be used to keep files together or reduce the total number of files you have to deal with.

Challenge

Challenge

How Much Space Can We Save?

Determine the size of the cleaned project folder.

Then:

  1. Create a ZIP archive.
  2. Measure the size of the ZIP file.
  3. Calculate the reduction in size.

Discuss:

  • Was the reduction larger or smaller than expected?
  • Why do you think this happened?

Results will vary depending on the data.

Text files and spreadsheets often compress well.

Images, video, and already-compressed formats may show little improvement.

File Formats And Storage

Compression is not the only way to reduce file size.

Sometimes the format itself matters.

Different file formats store information differently.

For example:

report.docx
report.pdf
report.txt

all contain text but may occupy different amounts of storage.

Similarly:

image.bmp
image.png
image.jpg

can have dramatically different sizes.

Switching from a human readable format to a binary (computer readable) format can also significantly reduce the file size and the speed of reading and writing the file.

An Example: Bitmap Images

Suppose we save the same image as:

sample.bmp

A bitmap stores every pixel explicitly.

This often results in a large file.

The same image saved as:

sample.png

may be significantly smaller because PNG uses lossless compression internally.

The image looks the same, but storage requirements differ.

Trade-Offs

Smaller files are not always better.

Changing formats may affect:

  • Data quality
  • Software compatibility
  • Metadata support
  • Long-term preservation

For example:

CSV

is highly portable and easy to share.

However:

Excel

supports multiple sheets and richer formatting.

Neither format is universally superior.

The best format depends on your intended use.

Looking Ahead

We will discuss choosing data formats in much more detail in a later episode.

Discussion

How Have You Shared Data Before?

Think about the last time you shared research data.

How did you do it?

Examples might include:

  • Email attachments
  • OneDrive links
  • SharePoint
  • USB drives
  • Shared network storage
  • Cloud platforms

What limitations did you encounter?

Caution

This is not legal advice. Before transferring files check your project’s data management policy for the type of data you are handling, or contact the relevant department at your institution.

Methods For Transferring Data

Different transfer methods are appropriate for different situations.

Email Attachments

Suitable for:

  • Small files
  • Documents
  • Individual figures
  • Non-private data

Email is generally unsuitable for large datasets and personal data.

Suitable for:

  • Collaborative projects
  • Medium-sized datasets
  • Shared documents

Shared Project Storage

Suitable for:

  • Ongoing collaborations
  • Research teams
  • Institutional projects

Globus

Globus is commonly used in research environments for transferring large datasets.

Suitable for:

  • Large research datasets
  • HPC environments
  • Multi-gigabyte or multi-terabyte transfers

Institutional File Drop-off Service

Some institutions have a file drop off service, such as Newcastle University’s Drop-off

Suitable for:

  • Medium sized data sets
Callout

A Note On Command-Line Tools

Large-scale research computing environments often use command-line transfer tools such as:

rsync
scp
sftp

These tools are extremely powerful and commonly used on HPC systems.

However, they are beyond the scope of this lesson.

Sharing Personal and Confidential Data

As discussed in the previous lesson, additional precautions need to be taken when storing personal or confidential data. Likewise, additional care is needed when sharing these types of data.

One option is to encrypt your data before sending it to colleagues. The password for decrypting the data should always be shared separately.

Newcastle University offers a File Transfer Service, which can encrypt data before transferring it and will also notify you when someone accesses the data. Services such as these offer additional security over email, which is not recommended for moving personal or confidential data. Email passes from you to the destination through a number of intermediate servers. If any link in the chain is exposed, the data can be leaked. This vulnerability is why companies rarely email you your bills, but instead offer secure websites for viewing them 1, 2.

Using OneDrive and Sharepoint to transfer files is an alternative to sending the data as a copy. If you send data as an email or via the File Transfer Service, you give the recipient a copy of your data. Using University OneDrive or Sharepoint only provides access to the original data. If an email is sent in error, you can remove the share link or revoke permissions to make the data inaccessible again (as long as the recipient as not copied it). This approach helps reduce errors such as sending data to the wrong email address or attaching the incorrect document.

Sanitising Your Data

When moving data, you need to ensure you only send the intended data. You need to check not only the contents of your files, but also any file metadata, documentation, file headers etc. For example, software often auto-generates metadata, such as the name of the creator, date created, the name of the last modifier, and date and time of the last modification, that is stored and transferred with the file.

Further Learning

Learners interested in transferring data to servers or HPC systems should consider attending:

  • Carpentries Shell/Bash workshops
  • HPC training courses

These courses cover tools such as rsync in more detail.

Challenge

Challenge

Choosing A Transfer Method

Match the transfer method to the scenario.

  1. A 5 MB document for a colleague.
  2. A 300 MB dataset shared among a research group.
  3. A 2 TB dataset for HPC processing.
  4. A published dataset accompanying a paper.

Discuss your choices with a partner.

Example answers:

  1. Email attachment.
  2. OneDrive, SharePoint, File Drop-Off service, or shared project storage.
  3. Globus or institutional research storage.
  4. Repository such as Zenodo.

The important consideration is matching tools to scale and purpose.

Looking Ahead

We now know how to:

  • Measure storage usage
  • Compress data
  • Reduce transfer sizes
  • Choose transfer methods

The next challenge is deciding what data should actually be kept.

In the next episode we will explore:

  • Data lifecycles
  • Archiving
  • Retention policies
  • Digital waste

and discuss how to make sensible decisions about what data should remain part of a project and what can be removed.

Key Points
  • Storage space and transfer speed are important practical considerations in research.
  • File size can be measured using file browser tools.
  • Compression reduces file size and is particularly useful for archiving and transfer.
  • Lossless compression preserves data and is generally preferred for research data.
  • Some file formats are more storage-efficient than others.
  • Choosing a different file format may involve trade-offs.
  • Different transfer methods are appropriate for different dataset sizes.
  • Large research datasets often require specialist transfer tools.
  • Understanding your data helps you make informed decisions about storage and transfer.

Content from What to Store and When to Remove

Last updated on 2026-07-23 | Edit this page

Overview

Questions

  • How do I decide which research data to keep and which to delete?
  • How long am I required to retain my research data?
  • What are the risks and hidden costs of keeping everything forever?
  • How do I permanently delete data across different systems?

Objectives

By the end of this episode, learners will be able to:

  • Identify key factors (funder policies, data value, provenance) when evaluating data retention.
  • Evaluate a sample dataset to distinguish between high-value raw data and redundant temporary files.
  • Permanently delete files using both local desktop file managers and remote server environments.
  • Explain the psychological barriers to deleting data and the impact of digital waste.

The Problem: Too Much Data, Too Little Clarity

In the last episode, we looked at how to measure file sizes, compress directories, and move data efficiently. But as our projects progress, we encounter a deeper problem:

We are running out of space, and we don’t know what half of these files actually are.

It is incredibly easy to adopt a “keep everything just in case” mentality. We generate temporary versions, intermediate analysis steps, raw outputs, and duplicate backups. Over time, this creates a digital swamp that makes finding relevant data difficult and increases the risk of mistakes.

As researchers, we need a strategy to determine what data is worth keeping, what must be kept due to policy mandates, and what should be safely removed.

Discussion

To Keep or To Delete?

Before looking at specific rules, let’s think about how we evaluate the value of our data.

Working in pairs or small groups:

Imagine you are cleaning out a project directory from two years ago. What factors or criteria would you consider when deciding whether to retain a file or permanently delete it?

Take 5 minutes to list as many considerations as you can.

What to Consider: The Retention Checklist

When deciding the fate of your data, expect to balance several competing factors:

  • Funder and Institutional Mandates: Many research funders (e.g., UKRI, NIH, Horizon Europe) explicitly require research data to be preserved for a set period, often 5 to 10 years after project completion or publication.
  • Reproducibility and Provenance: Can this data be easily re-generated? Raw data from an expensive, one-time physical observation or experiment is irreplaceable. An intermediate clean data file generated by a script can be deleted, provided the script and raw data are saved.
  • Legal, Ethical, and Privacy Constraints: If data contains sensitive personal information, or if consent agreements state data must be destroyed at the end of a study, you are legally obligated to delete it. Data can be requested by the person it is about and they can request the date is delected or ammended, see the Personal data access policy This will be discussed in a later section.
  • Data Integrity and Documentation: If a file has no metadata, no clear naming convention, and no documentation explaining its origin, its scientific value drops significantly. Data without provenance is often just digital noise.

Evaluating the Messy Dataset

Let’s look at a snapshot of our project directory and decide what stays and what goes.

Challenge

Challenge

Look at the following list of files found in a messy project folder. Based on what we’ve discussed, decide which files should be Retained and which should be Deleted. Be prepared to justify your choices!

  1. raw_sequencing_data_2024_03_12.fastq (Original machine output from a £5,000 lab run)
  2. temporary_merged_subset_v2.csv (A temporary file used to test a quick plotting idea)
  3. analysis_script_final.py (The code used to clean and process raw data into the final figures)
  4. draft_paper_comments_John.docx (An old manuscript draft from last year with tracked changes)
  5. unlabelled_test_results.txt (A text file with numbers, no headers, and no date; nobody remembers who made it)

Take 5 minutes to categorize these.

  • 1. Raw Sequencing Data: RETAIN. This is original, expensive raw data with clear provenance. Funders generally mandate keeping this.
  • 2. Temporary Merged Subset: DELETE. This was a temporary scratchpad file. It can be easily recreated if the scripts and raw data are kept.
  • 3. Analysis Script: RETAIN. This contains the methodology and workflow. It takes up almost no space but holds massive reproducibility value.
  • 4. Draft Paper Comments: DELETE or ARCHIVE. Once the paper is published, dozens of older internal draft files add noise. If retained, move them out of the active data directory into a dedicated “administrative archive” folder.
  • 5. Unlabelled Test Results: DELETE. Without context, documentation, or headers, this file is useless and potentially dangerous if mixed up with real data later.

How Deletion Actually Works

When we decide to delete a file, we need to understand how different systems handle that process. It isn’t always as instantaneous or permanent as it looks.

1. Local Deletion (Windows and macOS)

When you delete a file using your graphical file manager (File Explorer or Finder), the system doesn’t immediately scrub the data off your hard drive.

The Safety Net: The Recycle Bin / Trash

  1. When you click Delete or drag a file to the bin, the file system simply moves it to a hidden folder called the Recycle Bin (Windows) or Trash/Bin (macOS). Linux will use something similar.
  2. The Catch: The file still occupies space on your computer or hard drive! If you are deleting files to free up room because your drive is full, your drive will remain full until you take the next step. Data can still be recovered, which might be an issue if the data needs to be securely deleted
  3. Action: To truly free up space, you must right-click the Recycle Bin/Trash icon on your desktop and select Empty Recycle Bin or Empty Trash.
Caution

Secure data deletion

Even then data may not be entirely deleted and remains on disk until it is overwritten.

If you need make absolutely sure data is deleted:

  • Combine file deletion with encryption.
  • For Hard Drives use tools like shred (Linux), cipher (Windows) or rm -P in a Terminal (MacOS) .
  • Solid State Drives have their own secure deletion tools.
  • Securely dispose of the disk using confidential waste services.

Contact NUIT for advice if you have concerns.

Hard Drives and Solid State Drives behave differently when it comes to secure data delection.

2. Remote Server Deletion (Linux Storage / Research Data Warehouse)

Many institutions provide remote servers or shared network storage (like Newcastle University’s Research Data Warehouse) for storing massive datasets. Managing files here is fundamentally different.

The No Safety-Net Environment

  • When you access a shared Linux server or an institutional data warehouse via a shared network or file transfer tool, deleting a file is usually permanent and immediate.
  • There is typically no Recycle Bin or Trash folder on a remote Linux server filesystem.
  • Once you confirm you want to delete a file or directory, the pointers are destroyed, and the space is instantly reclaimed by the server.
  • Nothing stops you making your own Recycle Bin by having a folder to move files into, but it will lack the restore feature.
Caution

Double Check Before Clicking

Because remote servers lack a safety net, always double-check the files you have selected before hitting delete. A moment of carelessness on a shared server can wipe out months of collaborative work! Data archiving can function as an intermediate step before deletion.

Some remote servers have backups. Make sure you know if your server makes backups, how regular they are and how long backups are retained. It is useful to know how to access the backups and who to contact for help.

General Principles: The Psychology and Impact of Data Retention

Why do we find it so difficult to delete files?

The Hoarding Trap

Psychologically, deleting data can feel risky. We often worry: “What if I need this obscure intermediate file in three years?”. It feels that consequences of leaving useless data on disc are less than the consequences of accidentally deleting good data. The thought of having to explain why you accidentally deleted data from a £10,000 experiment is not a pleasent one. This fear causes us to hoard data.

However, keeping everything introduces hidden costs:

  • Mental Overhead: It takes longer to navigate folders, copy data, and find what you actually need.
  • Security Risks: Retaining old, sensitive participant data past its expiration date increases the severity of potential data breaches.
Discussion

What strategies could you use to mitigate worries about deleting data?

Discuss as a group.

Digital Waste and the Environment

Data storage isn’t free, and it isn’t completely green. Every gigabyte of data kept forever lives on a physical server spinning in a data center somewhere in the world.

These data centers require significant amounts of electricity to run and massive cooling infrastructure to keep from overheating. Deleting useless, duplicate, or temporary files directly reduces your project’s digital waste and environmental footprint.

Summary

Key Points
  • Research funders often mandate data retention for specific timeframes (typically 5–10 years).
  • Raw data and code have high preservation value; temporary files and undocumented data should be routinely purged.
  • Deleting files on a local desktop moves them to a Recycle Bin, meaning they still occupy disk space until it is emptied.
  • Deleting files on remote Linux servers or Data Warehouses is usually immediate and permanent with no undo option.
  • Over-retaining data creates mental clutter, increases security risks, and contributes to environmental digital waste.

Content from Best Practices for Tabular Data

Last updated on 2026-07-23 | Edit this page

Overview

Questions

  • What makes a tabular dataset “bad” or difficult to work with?
  • How do inconsistent column names, missing values, and stacked sub-tables cause errors in research?
  • How can I clean and standardise messy spreadsheet data using standard tools like Excel?
  • How can I track changes and apply version control to data files?

Objectives

By the end of this episode, learners will be able to:

  • Identify common pitfalls in tabular datasets (inconsistent naming, stacked tables, implicit missing values, omitted units).
  • Explain the core principles of tidy, machine-readable data.
  • Apply standard spreadsheet tools (including Find and Replace) to clean and bulk-fix data errors.
  • Evaluate strategies for tracking versions of data files and identify appropriate version control tools.

The Problem: Tabular Data Is Everywhere, But Often a Mess

Tabular data—data organised in rows and columns such as CSV files and Excel spreadsheets—is the most common format used across research disciplines. Whether you collect sensor output, clinical trial data, or survey responses, chances are high that your data lives in a table.

Unfortunately, spreadsheets are designed to be flexible for human eyes, which often leads to habits that make the data nearly impossible for computers (and collaborators) to process cleanly.

Consider a typical research scenario: you open a dataset shared by a predecessor or collaborator, and you want to plot trends or run a statistical script. However:

  • Columns have slightly different names for the same measurement.
  • Units are missing or buried in cell comments.
  • Blank rows divide different sub-tables within the same file.
  • Category names change midway down the sheet.

Before writing analysis scripts or generating figures, we must learn how to spot these issues and transform messy tables into structured, standardized data.

Discussion

Spotting the Pitfalls in Bad Data

Let’s start by inspecting a real-world example of a problematic dataset: Incidence-Data(bad).csv.

ID,table,diagnosis Year,Age Specific Incidence Age0_49,age Specific Incidence  Age 0-54,ageSpecificIncidenceAge 55-59,age SpecificIncidence Age60-64,ageSpecificIncidenceAge65_69,age Spes IncidenceAge70-74,Age_Specific_IncidenceAge75_79,Age80_84,Age85_89,Age 90
1,All Oral Cavity,All Years,,,,,,,,,,
2,All Oral Cavity,2016,1.0226257,7.1519032,11.546455,14.977962,15.698924,,18.930473,18.508072,20.708221,20.908237
3,All Oral Cavity,2017,1.1054269,7.7797451,10.903189,13.126522,16.235617,16.384249,14.734702,17.668417,16.770851,18.774832
4,All Oral Cavity,2018,0.97927344,7.1045599,10.339744,13.697983,15.216359,16.599554,17.230083,16.623766,18.70484,22.232193
5,All Oral,,0.8994543,6.5189319,9.9068785,12.557129,18.081358,17.912762,18.943886,19.327847,20.068048,23.00526
6,All Oral,2020,0.80034059,6.8461018,9.9605932,12.894961,15.298739,16.424881,16.835621,17.944742,18.093088,20.342873
...

Working in pairs:

Open Incidence-Data(bad).csv in Excel or your preferred spreadsheet viewer. Look closely at the header row, the values, and the overall layout. List as many data management issues as you can find.

Consider:

  • Header names and formatting
  • Category values in rows
  • Blank cells and missing data
  • Units of measurement
  • Table structure and layout

After 5 minutes, gather responses as a group.

Why These Issues Matter: Human-Readable vs. Machine-Readable

Human eyes are remarkably forgiving. When a human reads a spreadsheet, they can instantly guess that "All Oral" and "All Oral Cavity" mean the same thing, or that a blank cell under the year 2018 probably means 2019.

Computers, however, are strictly literal. To an automated script or statistical package (like R, Python, or SPSS):

  • "All Oral" and "All Oral Cavity" are two completely unrelated experimental groups.
  • age Specific Incidence Age 0-54 (with two spaces) is a different variable name from age Specific Incidence Age 0-54 (with one space).
  • A blank year cell is treated as null or NaN, causing data to be silently dropped from time-series plots.

Here is a breakdown of the issues identified in Incidence-Data(bad).csv:

1. Inconsistent Header Naming Conventions

The column headers mix title case, lowercase, camelCase, snake_case, and arbitrary spaces:

  • diagnosis Year (contains a space and mixed case)
  • age Specific Incidence Age 0-54 (contains double spaces)
  • ageSpecificIncidenceAge 55-59 (camelCase mixed with spaces)
  • age Spes IncidenceAge70-74 (spelling typo: “Spes”)
  • Age80_84 vs Age 90 (abruptly dropped the variable name prefix entirely)

2. Missing Units in Headers

What do the numbers in the data cells actually represent? Is this incidence per 1,000 people? Per 100,000 people? A percentage? Without units explicitly recorded in the header or accompanying metadata, the numbers are ambiguous.

3. Stacked Sub-Tables in a Single File

Rows 10 and 20 are completely blank, serving as visual dividers between sub-tables (All Oral Cavity, Male, Female). Furthermore, summary rows (e.g., All Years) are mixed directly into the same column as specific annual data.

4. Inconsistent Category Values

In the table column, the category is written as "All Oral Cavity" in rows 2–4, but changes to "All Oral" in rows 5–9.

5. Implicit Missing Data

In several rows, values for diagnosis Year or table are left blank because the author assumed the reader would know it “carried over” from the row above.

Core Principles of Tidy Tabular Data

To make tabular data easy to analyze, archive, and share, adhere to three core rules of Tidy Data:

  1. Each variable forms a column: Do not mix multiple variables into a single column, and do not drop variable names midway across headers.
  2. Each observation forms a row: Every row should represent a single timepoint, subject, or experimental run.
  3. Each cell contains a single value: Do not embed units, notes, or multiple values in a single cell.
Good Header Example:
id, category, diagnosis_year, incidence_per_100k_age_0_49, incidence_per_100k_age_50_54

or for an alternative example dataset where this might not be appropriate:

id, location, date,    object, spectral_flux_density
  ,         , YY-MM-DD,      , 10^-23.erg.s^-1.cm^-2.Hz^-1

Here there is a new row for the units because they can become unwieldy and hard to read in the header name. Use your own judgement and the conventions of your discipline.

  • This unit, used in radio astronomy for the spectral energy density, can also be written as the none SI/CGS unit, Jy (Jansky), but is included here as an example using the largely superceded CGS unit system for dramatic illustration (!). See: “IAU Style Manual” by G.A. Wilkins, Comm. 5, in IAU Transactions XXB (1989)”
Callout

Golden Rules for Clean Headers

  • Use lowercase letters.
  • Use underscores (_) instead of spaces.
  • Avoid special characters (?, $, %, -, /, #).
  • Keep names descriptive but concise.
  • Include units directly in the header (e.g., weight_kg, temp_celsius, incidence_per_100k). If this is not reasonable place units in the next row or column to the column name.
Discussion

Demonstration: Cleaning Data in Excel

Follow along with the various practical steps to fix these issues in Excel (or LibreOffice Calc / Google Sheets) without altering the underlying raw data.

Step 1: Standardising Column Headers

  1. Open Incidence-Data(bad).csv in Excel.
  2. Select Row 1 (the header row).
  3. Systematically rename headers to follow a single, consistent naming convention (e.g., lowercase with underscores):
Old Header New Clean Header Reason for Change
diagnosis Year diagnosis_year Replaced space with underscore, lowercase
age Specific Incidence Age 0-54 incidence_per_100k_age_0_54 Fixed double spaces, added unit (per_100k), standardized naming
age Spes IncidenceAge70-74 incidence_per_100k_age_70_74 Fixed typo (Spes), standardized
Age80_84 incidence_per_100k_age_80_84 Restored missing variable prefix

Step 2: Bulk Fixes Using Find and Replace

Manually retyping mismatched category names across thousands of rows is slow and error-prone. We can use Excel’s Find and Replace tool to perform bulk corrections.

  1. Highlight the column containing category names (Column B: table).
  2. Press Ctrl + H (Windows) or Cmd + H (macOS) to open the Find and Replace dialog.
  3. In Find what, enter: All Oral
  4. In Replace with, enter: All Oral Cavity
  5. Click Match entire cell contents (if available) to avoid accidental partial matches.
  6. Click Replace All.

Excel will report how many replacements were made, ensuring all rows now use a single category label.

Step 3: Handling Missing Values and Stacked Tables

To convert stacked tables into a single, clean table:

  1. Fill in implicit missing values: If rows 5 and 9 are missing the diagnosis_year value, consult the source documentation and enter the explicit year (e.g., 2019, 2023). Never leave cells blank if the value is known.
  2. Remove empty separator rows: Delete completely blank rows (such as rows 10 and 20) that were used purely for visual padding.
  3. Separate summary rows if necessary: Filter or remove summary rows like All Years if you plan to aggregate data programmatically, or move distinct sub-tables into separate CSV files if they represent fundamentally different experimental structures.
Challenge

Cleaning the Incidence Dataset

Working with your copy of Incidence-Data(bad).csv:

  1. Use Find and Replace to standardise any inconsistent category names in the table column.
  2. Standardize the headers for age groups Age85_89 and Age 90 so they match the format of the other age columns (incidence_per_100k_age_85_89 and incidence_per_100k_age_90_plus).
  3. Fill in missing values in the diagnosis_year column where values were implicitly omitted.
  4. Delete the blank row dividers.
  5. Save the modified file as Incidence-Data_clean.csv.

Compare your cleaned file with a partner. Are all header names identical?

A clean version of the dataset should have:

  • Uniform headers across all columns without spaces or special characters. * Units (per_100k) included in the header metadata or documented clearly.
  • No blank separator rows.
  • Explicit values in every row (no implied carry-overs).
  • Consistent category labels (All Oral Cavity throughout).

Version Control for Data Files

Once you have spent time cleaning a dataset, how do you manage changes as the project evolves?

The Problem with File-Naming “Version Control”

It is common to see project folders filled with files like:

Incidence-Data_clean.csv
Incidence-Data_clean_v2.csv
Incidence-Data_clean_v2_final.csv
Incidence-Data_clean_v2_final_FINAL_corrected.csv

This manual approach quickly leads to confusion over which file is the true “current” version, and makes it impossible to track what changed, why it changed, and who changed it.

Version Control Strategies for Research Data

Depending on your data size and computational setup, several strategies exist for tracking changes:

1. Separation of Raw and Processed Data

Never modify your original raw dataset. Maintain a strict directory structure:

  • data/raw/: Read-only original files (never modified).
  • data/processed/: Cleaned, standardized outputs generated by documented steps or scripts.

2. Cloud and Storage Revision Histories

Platforms like OneDrive, SharePoint, and institutional research data repositories automatically record file version histories. If you accidentally overwrite a spreadsheet, you can restore previous versions through the cloud provider’s web interface.

3. Formal Version Control with Git

For researchers working with text-based tabular files (such as CSVs, TSVs) and processing scripts (Python, R, MATLAB), formal version control systems like Git offer the gold standard for tracking history.

Git records snapshots of your files over time, allowing you to:

  • View exact line-by-line differences between file versions.
  • Revert to any previous state if an error is introduced.
  • Collaborate safely without overwriting a colleague’s work.

Further Learning: Version Control with Git

Demonstrating Git is beyond the scope of this introductory lesson, but learning Git is highly recommended for any researcher handling data and code. If you want to learn how to track changes, collaborate effectively, and manage versions using Git, consider signing up for our companion workshop:

  • Software Carpentries: Version Control with Git
Key Points
  • Humans and computers read tables differently; layout spreadsheet data for machine readability.
  • Tidy data requires one variable per column, one observation per row, and one value per cell.
  • Column headers should be consistent, lowercase, free of spaces, and explicitly state units of measurement.
  • Tools like Find and Replace in spreadsheet software enable quick, bulk standardisation of inconsistent values.
  • Never modify raw data directly; maintain separate raw and processed data folders.
  • Formal version control tools like Git, alongside institutional cloud history, provide robust methods for tracking data changes over time.

Content from Choosing a Data Storage Format

Last updated on 2026-07-28 | Edit this page

Overview

Questions

  • How does a file extension determine how data is stored and read?
  • What factors should I consider when choosing a data format for my research?
  • What are the differences between discipline-specific and domain-agnostic formats?
  • What are the risks of converting files between different formats?

Objectives

By the end of this episode, learners will be able to:

  • Explain how filename extensions correspond to underlying data structures.
  • Evaluate the trade-offs (portability, metadata support, file size) of different data formats.
  • Identify common domain-specific (e.g., .nii, .fastq) and multi-discipline cross-cutting formats (e.g., HDF5).
  • Demonstrate the impact of format conversion on file size and data fidelity using image files.

The Problem: The Wrong Format Can Break Your Data

Over the last few episodes, we have cleaned up our files, established rules for deleting digital waste, and structured our tabular data. However, a major decision remains before we wrap up our data management workflow:

What format should we save our data in?

Often, we default to whatever software we have open—saving tables as Excel workbooks (.xlsx) or text as Word documents (.docx). But the format you choose dictates who can open your data, how large the file will be on disk, whether metadata can be embedded safely, and if your data will still be readable in 10 or 20 years.

Choosing the wrong format can cause unreadable files, missing metadata, or a permanent loss of data fidelity.

Understanding File Formats and Extensions

A file format is a standardized way that information is encoded for storage in a computer file.

The suffix at the end of a filename (e.g., .csv, .xlsx, .txt), known as the file extension, tells the operating system—and you—which application is expected to read that specific layout of bits.

Files can be plain text (human readable) or binary (readable only by the computer). Plain text files have the advantage that s user can look inside them and see the data, while binary files are more compact and faster for the computer to read and write.

Common Formats and Trade-Offs

Some examples of the many types of file formats are

  • CSV (Comma-Separated Values - .csv): A plain-text format where data fields are separated by commas. It is highly portable, open-source, and readable by almost any programming language or spreadsheet tool. However, it cannot store multiple sheets, cell formatting, or formula logic.
  • Excel Workbook (.xlsx): A proprietary, compressed binary format owned by Microsoft. It is excellent for multi-sheet organization, complex formulas, and built-in plotting. The trade-off is that it requires specific software to parse cleanly, can introduce automated corruption (such as converting gene names into dates), and is less suitable for long-term data archiving.
  • FWF (Fixed-Width Format, - usually .txt): A plain-text format where each column has a fixed width. Because it is portable and open source it is easily readable, like .csv files. However, it can waste space and be hard to edit if values vary in length. If the size of a given column can vary significantly (e.g. have value 12 or 124535643245667 then shorter entries need considerable padding or risk truncating longer entries).
  • JSON (Javascript Object Notation - .json): A plain-text format that organises data into sets of key-value pairs, such as {“first_name”: “John”, “last_name”: “Smith”}, where “first_name” and “last_name” are the keys and “John” and “Smith” are the corresponding values. This flexible format allows JSON to store very complex data structures, but it can be verbose and difficult to follow. Many programming languages have libraries for reading and manipulating JSON. Unlike the other formats listed above JSON files are not limited to tables and not every field needs to be the same for every entry, making it highly flexible.

CSV, Excel Workbook, and FWF files are for storing tables. They are good for data which have a certain number of items (rows) each with a number of values, or noted missing values, for a consistent set of variables (columns).

Discussion

What to Consider When Choosing a File Format?

Before looking at complex formats, let’s establish a set of requirements for evaluating how we store data.

Working in pairs or small groups:

Imagine you are starting a new 3-year research project that will generate thousands of files. What factors should you consider when deciding what file format to use?

Take 5 minutes to think about compatibility, long-term access, and data complexity.

Domain-Specific vs. Cross-Discipline Formats

Different types of research demand different types of file architectures. Some formats are highly specialized, while others cross scientific boundaries.

1. Discipline-Specific Formats

Many fields have developed unique file formats tailored to their specific instrumentation and data scale requirements:

  • Neuroscience / Medical Imaging: .nii (NIfTI - Neuroimaging Informatics Technology Initiative). Used to store 3D and 4D brain scans from MRI machines, embedding spatial orientation metadata directly into the file header.
  • Genomics / Bioinformatics: .fastq (FASTQ format). A text-based format for storing both a biological sequence (like DNA/RNA reads) and its corresponding quality scores, tightly packed for fast programmatic querying.
  • Geophysics / Seismology: .asdf (ASDF - Adaptable Seismic Data Format). Used to store seismic data.
  • Astronomy: .fits (FITS - Flexible Image Transport System). A binary format for astronomical images and tables. Stores flexible header information.
  • Machine learning: .pmml. A file format for storing some types of machine learning models.

2. Language specific formats (e.g. Pickle)

Some formats are designed to store data for specific programming languages.

  • Python .pkl (Pickle format)
  • R .rda (RData format)
  • Matlab .mat (Mat format)
  • Julia .jld2 (Julia Data Format 2)

These formats allow the given programming language to save data and even their current state to disc. This storage approach can be convenient, but introduces security risks. For example, Python pickle files can contain instructions that execute arbitrary Python code when loaded. They are not recommended for data exchange or long-term storage.

3. Cross-Discipline Formats (e.g., HDF5)

Some formats are designed to solve the problem of multi-dimensional data across multiple fields with different types of data.

  • HDF5 (Hierarchical Data Format v5 - .h5 or .hdf5): This open format behaves like a “file system within a single file.” It allows you to store large, multi-dimensional arrays of numbers alongside rich, custom metadata attributes. If is fast, flexible and compact. It is not human readable.
  • Parquet: This open format preserves metadata, data types, and file headers. It is fast for reading but slower for writing. Excellent for Big Data which will be written once and read many times. Like HDF5 it is not human readable.

Who uses these file formats?

Common users of HDF5 include climate scientists for satellite weather models, physicists for electron microscopy data, machine learning researchers for deep neural network models, and neuroscientists for brain activity (e.g., EEG) data.

4. Other

Some software has their own files formats for designed for specific needs - for example, Microsoft PowerPoint uses .pptx files and Adobe Photoshop stores image data in .psd files. Other formats may be based on common formats, such as binary or HDF5, but require a certain structure for the header or metadata. These formats will have software tools for reading them and/or provide descriptions of the file format to allow writing bespoke software to read it.

The Format Evaluation Checklist

When choosing a format, expect to balance these core pillars:

  • Open vs. Proprietary: Open file formats are data structures that are publicly documented, free for anyone to use, and maintained by an independent, non-profit standards organisation. Meanwhile, proprietary file formats are owned by a private company or individual that imposes legal restrictions on their use. While some proprietary formats, such as .mat files (created by MATLAB), have open documentation and can be read using free software, the free readers may not support all of the format’s features. Additionally, the owner retains control over its use and definition, potentially leading to restrictions or changed formats in future versions. As such, open formats are generally preferred for FAIR data sharing.
  • Self-Documentation and Metadata: Does the format allow you to embed metadata (data about the data) directly within the file? For example, can you include system headers, timestamps, and sensor configurations at the top of the file?
  • Data Complexity and Structure: Is your data flat and two-dimensional (a simple table), or is it hierarchical and multi-dimensional (such as time-series mapping across a 3D spatial grid)?
  • Human Readable or Binary: Can your file format be opened in a text editor and read (e.g., .csv, .json), or is data stored in a format which requires software to view (e.g., a binary file)? Human-readable files can be easier to manually review, but many machine-readable file formats are smaller and can be read/written much faster than human-readable formats.
Discussion

Formats in Your Research Field - Shared Experiences

Take 3 minutes to talk with the person next to you:

  1. What are the primary file formats you generate or interact with in your specific research domain?
  2. Are these formats open and easy to share, or do they require specialised, proprietary instruments or software to read?
  3. Why are these formats used? What needs do they address?

The Dangers of Format Conversion (Data Fidelity vs. Size)

Changing a data format can be a powerful way to reduce file size or improve software compatibility, but it can also be incredibly dangerous if you do not understand how the format handles information.

This is easiest to demonstrate using image formats, which are widely used across sciences to capture figures, microscopy, or satellite imagery.

Lossy vs. Lossless Formats

  • BMP (.bmp): A raw bitmap format. It saves the exact color data of every single pixel explicitly. It preserves 100% fidelity but results in large file sizes.
  • PNG (.png): A compressed, lossless format. It reduces the file size significantly using smart patterns, but when you open it, the image is mathematically identical to the original pixel-by-pixel.
  • JPEG (.jpg / .jpeg): A compressed, lossy format. It achieves tiny file sizes by permanently throwing away visual information that the human eye is less likely to notice.
  • SVG (.svg): A vector graphics format which scales freely and defines graphics using vertices and lines. They are easily edited. However, they are not suitable for pixel-based images.
Challenge

Challenge

Let’s look at what happens when we convert a high-resolution microscopy image (cell_sample.bmp) into a space-saving JPEG format.

## Challenge: Comparing Image Conversion and Fidelity - Inspecting the Artifacts

  1. Open your operating system’s File Explorer or Finder and look at the file sizes of the following three identical-looking images:
  • cell_sample.bmp (Size: 12.5 MB)
  • cell_sample.png (Size: 2.1 MB)
  • cell_sample.jpg (Size: 350 KB)
  1. Open cell_sample.jpg and zoom in closely (400% or higher) on high-contrast areas or fine edges. Compare it side-by-side with cell_sample.bmp.

Questions for Discussion: * What happens to the file size when the BMP file is converted to JPEG format? * What visual defects (compression artifacts) do you see around the fine details in the JPEG version? * Why is saving raw experimental images as JPEGs potentially dangerous for automated scientific analysis?

  • File Size: The JPEG version is a fraction of the size of the original Bitmap, saving a large amount of storage space.
  • Fidelity Loss: Zooming in reveals blocky distortions and blurry halos (“artifacts”) around fine structures.
  • Scientific Danger: Automated image processing scripts (e.g., counting cells, measuring pixel intensities) will read these compression artifacts as actual data or fail to detect faint structures entirely. Never use lossy compression formats (like JPEG) for raw analytical images.

See Image Processing in Matlab and Section 6.8 of Cromey D. W. (2013). Digital images are data: and should be treated as such. Methods in molecular biology (Clifton, N.J.), 931, 1–27. https://doi.org/10.1007/978-1-62703-056-4_1

Data storage facilities – Recommendations

Many data facilities have suggestions for storing data. For example

  • the UK Data Service has recommendations for various types of data, including tables, image, video, audio, and text, as well as their associated metadata.
  • the Library of Congress has similar recommendations

but funders and other data archives will have their own preferences.

The Library of Congress also provides an overview of various file formats.

Key Points
  • Filename extensions guide the operating system on how to parse binary data.
  • Prefer open data formats over proprietary ones to guarantee long-term accessibility and transparency.
  • Formats like NIfTI and FASTQ serve specialized domain requirements, while HDF5 provides cross-discipline support for hierarchical data.
  • Metadata should ideally be integrated directly within the data file format headers.
  • Converting between formats can lead to permanent data corruption or loss of fidelity, especially when shifting to lossy formats.