Key Points

Introduction to Data Management

Data management is an essential part of research practice.
Many common research frustrations arise from poor organisation and documentation.
Computers organise information using files and folders within a file system.
File explorers provide a graphical way to navigate stored data.
Understanding how files are organised is a foundation for good data management.
A well-organised project should be understandable to collaborators and to your future self.

Folder structures should help people find and understand data.
Use meaningful, consistent folder names.
Separate different types of project content into dedicated folders.
Avoid excessive nesting.
File Explorer allows folders to be moved, renamed, and deleted without using the command line.
Consistency is usually more important than any specific organisational scheme.
A good folder structure should be understandable by collaborators and by your future self.

Storage decisions should consider accessibility, recoverability, security, and sustainability.
Local storage is convenient for active work, but is rarely meets other storage requirements.
Personal cloud storage can create access problems when researchers leave projects.
Shared project storage is generally preferable to individual-owned storage.
Research data should ideally be stored in project-owned, backed-up systems.
Permissions should be managed carefully and reviewed regularly.
Personal and confidential data may require additional controls and approved storage locations.
Updating data locations can affect collaborators and workflows.
Repositories such as Zenodo can make research outputs more discoverable and citable.
Version control is important for managing changes and is covered in dedicated Carpentries Git workshops.

Storage space and transfer speed are important practical considerations in research.
File size can be measured using file browser tools.
Compression reduces file size and is particularly useful for archiving and transfer.
Lossless compression preserves data and is generally preferred for research data.
Some file formats are more storage-efficient than others.
Choosing a different file format may involve trade-offs.
Different transfer methods are appropriate for different dataset sizes.
Large research datasets often require specialist transfer tools.
Understanding your data helps you make informed decisions about storage and transfer.

Research funders often mandate data retention for specific timeframes (typically 5–10 years).
Raw data and code have high preservation value; temporary files and undocumented data should be routinely purged.
Deleting files on a local desktop moves them to a Recycle Bin, meaning they still occupy disk space until it is emptied.
Deleting files on remote Linux servers or Data Warehouses is usually immediate and permanent with no undo option.
Over-retaining data creates mental clutter, increases security risks, and contributes to environmental digital waste.

Humans and computers read tables differently; layout spreadsheet data for machine readability.
Tidy data requires one variable per column, one observation per row, and one value per cell.
Column headers should be consistent, lowercase, free of spaces, and explicitly state units of measurement.
Tools like Find and Replace in spreadsheet software enable quick, bulk standardisation of inconsistent values.
Never modify raw data directly; maintain separate raw and processed data folders.
Formal version control tools like Git, alongside institutional cloud history, provide robust methods for tracking data changes over time.

Filename extensions guide the operating system on how to parse binary data.
Prefer open data formats over proprietary ones to guarantee long-term accessibility and transparency.
Formats like NIfTI and FASTQ serve specialized domain requirements, while HDF5 provides cross-discipline support for hierarchical data.
Metadata should ideally be integrated directly within the data file format headers.
Converting between formats can lead to permanent data corruption or loss of fidelity, especially when shifting to lossy formats.