Documenting Your Data

Last updated on 2026-07-28 | Edit this page

Estimated time: 45 minutes

Overview

Questions

  • How can I remember what my data is and where it came from?
  • What information should I document about a project?
  • Where should documentation be stored?
  • What is a README file?
  • How can I create project documentation using a graphical file browser?

Objectives

By the end of this episode, learners will be able to:

  • Explain the purpose of research data documentation.
  • Identify information that should be captured about a dataset.
  • Create a README file using a graphical file browser.
  • Document project structure, provenance, and data sources.
  • Identify situations where additional documentation is needed.
  • Improve the reusability of a dataset through documentation.
  • Use file headers to store key metadata.

How Do I Remember and Share Where My Data Is Stored?

In the previous two episodes, we improved our folder structure and renamed our files.

The project now looks considerably better than it did when we inherited it.

However, several important questions remain unanswered:

  • What does this dataset contain?
  • Who collected it?
  • When was it collected?
  • How was it collected?
  • What do the variables mean?
  • What units were used?
  • Which files are raw data?
  • Which files were generated during analysis?
  • What changes do different versions capture?
  • Which version should be used?
  • Which files relate to which other files?
  • When was the data downloaded?
  • Where was the data downloaded from?
  • What software and versions of software were used to analyse the data?

Imagine returning to this project six months from now. Would you still remember all of these details? What about three years from now? What if somebody else joins or takes over the project?

Good folder structures and filenames help people navigate data. Documentation helps people understand data.

Challenge

What Makes Data Easy to Understand?

Think about a dataset, project, software package, or repository that you found particularly easy to use.

  • What made it easy to understand?
  • What information was available?
  • How was that information presented?
  • Was there a README file?
  • Was documentation easy to find?

Conversely:

  • Have you ever received poorly documented data?
  • What information was missing?

Discuss responses with your neighbour.

Helpful documentation practices include

  • Clear explanations
  • Consistent organisation
  • Readily available information
  • Definitions of terms and variables
  • Information about data collection
  • Contact details

Many learners will have experienced poorly documented datasets.

Encourage them to connect their frustrations as data users with their responsibilities as data creators.

Documentation Is For Humans

When discussing data management, people often focus on technology:

  • Storage
  • File formats
  • Software
  • Analysis tools

However, one of the biggest barriers to reusing data is simply understanding it. Good documentation reduces the number of assumptions that somebody must make when using a dataset.

Callout

The most important collaborator you will ever have is often future you.

Documentation helps future you understand decisions that seem obvious today, but may be forgotten later.

What Is Metadata?

Callout

Metadata is data about data.

Metadata provides context that helps us understand a dataset.

For example, consider the file:

incidence_data.csv

The filename tells us very little.

Useful metadata might include:

  • What the dataset contains.
  • Who collected it.
  • When it was collected.
  • How it was collected.
  • Units of measurement.
  • Data quality or collection issues.
  • Licensing information.
  • Contact details.
  • Link to the original source of the data if using an external source of data.

Metadata helps transform a collection of numbers into something meaningful.

Some software autogenerates some metadata for you.

For example, Microsoft Word produces

  • Author
  • Company
  • Template
  • Date Created
  • Date Modified
  • Last Saved By
  • Revision number
  • Total editing time
  • etc…

by default. Other software will store different metadata.

Caution

Automated metadata may add information you do not wish to share to the file. Ensure you do not inadvertently share sensitive or private metadata when distributing data.

Microsoft Office provides tools for removing automated metadata if required.

What Should We Document?

Given the huge variety of research and data, there is no universal documentation standard suitable for every project. However, most project-level documentation should include

  • Project description
  • Dataset descriptions
  • Data provenance
  • Data use and analysis
  • Contact details
  • README file(s)

Some projects may also include

  • Data dictionaries
  • Standard operating procedures
  • Project logs
  • Parameter files

These elements will be expanded on below.

Callout

Research should be reproducible. Given the information in the documentation and an accompanying paper or thesis, would someone else be able to repeat your research?

Would future you be able to repeat your research?

Project Description

Provide a short description that summarises the project’s goal; for example,

This project investigates disease incidence in marine mammals
along the Scottish west coast between 2020 and 2025.

A new collaborator should understand the purpose of the project within a few seconds.

It may be tempting to skip this description and instead rely on a long project folder name such as

marine_mammals_disease_scotland_westcoast_2020_to_2025

but this name is long and still ambiguous. Additionally, a project’s scope and goals change over time; what if your focus narrows to investigating seals or dolphins? It is easier to change the documentation than update folder names and references.

Data descriptions

Describe the available datasets and folders.

For example:

raw_data/
Contains original incidence records obtained from collaborators.

processed_data/
Contains cleaned datasets used in analysis.

analysis/
Contains scripts and outputs generated during analysis.

This documentation helps people understand how to navigate your files and the relationships between them.

Data provenance

Data provenance - the full record of a dataset’s origins and modifications - becomes increasingly important as projects grow, and certain projects may have strict rules for tracking providence. Some data sources include provenance information in file headers, while others store provenance in other locations.

If data sources are updated, for example to correct collection errors or accidental changes, it is important to keep track of the new and old files.

Challenge

Documenting data provenance

What things do you need to document to track data provenance for one of your projects?

Think about details you may need to - report in a paper or other publication - reproduce the work in future - assess or fix a data error that is discovered during the project

Possible information to document:

  • Was the data collected by your team? When? Where? Include links to other notes, such as lab manuals.
  • Was it obtained from a collaborator? When?
  • Was it downloaded from a repository? When?
  • Was it generated by software? Which version?
  • Were any corrections or other modifications made? Why, when, and by whom?

Data use and analysis

Many projects involve extensive data analysis. This analysis needs to be well-documented to ensure anyone with the same data can reproduce your results.

If you share your data, you also need to include any access conditions, such as restrictions for publishing or storing the data.

Challenge

Documenting your data analysis

Consider what information you should document on how you used and analysed the data during one of your project.

Useful documentation might include

  • Required software and their versions
  • Processing steps and settings
  • Important assumptions
  • Known limitations of your analysis
  • Access conditions - i.e., how to access the data and any restrictions on using it

These details can save future users significant amounts of time.

Contact details

Include contact information whenever possible, including for a primary contact, so that collaborators and other researchers can ask questions or raise concerns.

For example:

Project Lead (Primary Contact):
Dr Jane Smith

Email:
j.smith@example.ac.uk

Data Collection Lead:
Dr John Doe

johndoe@biguniversity.ac.uk

README File(s)

One of the simplest and most effective forms of documentation is a README file.

A README is usually placed at the top level of a project:

project/
│
├── README.txt
├── data/
├── analysis/
└── results/

It explains the project and helps people navigate it.

A good README provides enough information for someone to understand the project at a high level without opening any other files.

Callout

A README does not need to be perfect. A simple README containing a few useful paragraphs is significantly better than no README at all!

Try to keep README files up to date.

Keep files together with their READMEs if you exchange data with colleagues.

Example README Structure

Example template:

Project Name

Project Description

Project Structure

Data Sources

File Naming Convention

Software Requirements

Contact Information

Last Updated

For example:

Project Name:
    Marine Mammal Incidence Study

Project Description:
    Investigation of marine mammal health records collected
    between 2020 and 2025.

Project Structure:
    data/raw - original data
    data/processed - cleaned datasets
    analysis - analysis scripts
    results - generated outputs

Software Requirements:
    Data was analysed using Python 3.13
    See analysis/pyproject.toml file for Python package dependencies and versions

Contact:
    bob.badgerton@example.ac.uk

Last Updated:
    2026-07-01

Another example can be found here.

Challenge

Creating A README File

Follow along with the demo to make a README file for the inherited project.

1) Navigate To The Top-Level Folder

Open File Explorer.

Navigate to:

legacy_dataset/

2) Create A New Text File

Right-click in an empty area of the window and select

New → Text Document

to create a new file.

3) Rename The File

Rename the file README.txt.

4) Open The File

Double-click the file to open it in Notepad (or a similar text editor).

5) Add Basic Structure

Add headings such as

Project Name

Description

Folder Structure

Data Sources

Contact Information

Last Updated

Save the file. We’ll fill out the fields later in the lesson!

Demonstrate each step live.

Some learners may never have manually created a text file before.

Pause between steps to allow learners to follow along.

Documentation Beyond The Top Level

Project-level documentation is important, but subfolders sometimes require their own documentation. For example, a results folder may contain dozens of generated outputs and sub-folders. Several months later, someone may reasonably ask

  • Which software generated these files?
  • When were they created?
  • Which parameters were used?
  • Which version of the data was analysed?
  • What is the data access and retention policy of these data?

A README placed inside the results folder can answer these questions.

For example:

results/
│
├── README.txt
├── figures/
└── tables/

The README could document

Generated:
2026-07-01

Analysis Software:
R version 4.6

Input Data:
processed_data_v03.csv

Parameters:
Significance threshold = 0.05

to preserve important contextual information.

Caution

Any READMEs written by hand may also need to be manually updated as your project progresses, which risks the documentation becoming out-of-date. Where possible, automate your documentation (for example, by exporting settings from analysis tools or scripts).

Discussion

Consider one of your projects. What additional README files may you need beyond the top-level project README, and what information should they contain?

Other Useful Documentation

Data Dictionaries

These explain variables in tabular datasets.

For example

Variable Description
incidence_rate Number of cases per 100,000
region Geographic study area
year Observation year

We will revisit these dictionaries in the tabular data lesson.

Standard Operating Procedures

Standard operating procedures (SOPs) describe how data is collected or processed.

Project Logs

Project logs provide a record of major project decisions; they help record not only what you did, but why. This logic can be help justify methods in publications as well as warn future researchers about inefficient or unsuitable approaches that were ruled out in earlier project stages.

Parameter files

Many analyses are highly adjustable, with various different parameters, or settings, that can be changed depending on your needs. Parameter files record these settings to ensure that your data processing and analysis steps can be reproduced.

Challenge

Write A README

Fill out the README.txt file for the inherited project.

Include

  • Project name
  • Project description
  • Folder structure
  • Data sources
  • Contact information
  • Last updated date

Make reasonable assumptions; if you inherit a real, poorly-documented project, you’d probably need to read through old project emails/messages, find the original project/grant proposals, and talk to current lab members to get all the README information.

The goal is to make the project easier for a future collaborator to understand.

There is no single correct answer, but it should allow somebody unfamiliar with the project to begin understanding its contents.

Additional resources:

File-level documentation

Individual files often require additional documentation; for example, you may have many data files that were collected under different circumstances. Where possible, this data should be stored directly in the file in the file header or metadata. We will discuss approaches for storing headers later in the data formats lesson.

The advantage of storing metadata in a file is that it cannot be separated from the corresponding data. File names and folder names do not have this property since files can be easily renamed or moved to different folders. As such, avoid storing important metadata in file or folder names.

Discussion

Reflection

Imagine you are leaving your current project tomorrow. What information would a replacement researcher need to continue your work? Does your project’s current documentation include that information?

Caution

You may not always have much warning before temporarily or permanently leaving a project; plans can unexpectedly change due to new opportunities, emergencies, or illness. Similarly, new people may join a project without much notice and need to get up-to-speed quickly. Keeping your documentation up-to-date can make unplanned transition periods less stressful.

Looking Ahead

Our project now has

  • a clearer folder structure
  • more information file names
  • initial documentation

The next challenge is deciding where data should be stored so it is accessible, secure, shareable, and recoverable.

In the next episode we will explore how to choose appropriate storage locations for different types of research data.

Key Points

Key Points

  • Documentation makes data understandable and reusable.
  • Documentation is primarily written for people, including future-you.
  • Metadata provides context about a dataset.
  • README files are a simple and effective way to document projects.
  • A README should explain what the project is, how it is organised, and where the data came from.
  • Documentation should be stored close to the data it describes.
  • Additional README files can be useful within subfolders such as results or processed data.
  • Good documentation reduces confusion and improves collaboration.