Academic programmes generate data long before anyone calls it a dataset. Applications arrive through forms, course choices are recorded in tables, attendance changes over time, and documents must eventually reflect decisions made at several earlier stages. Each individual action may appear simple. The difficulty lies in keeping them connected.
My work on data and document workflows for an international educational programme made this especially visible. The challenge was not simply to collect information or automate a repetitive task. It was to create a structure in which records could move through several stages without losing their identity, context, or status.
This note describes the principles behind that work. It contains no personal records, operational credentials, or internal datasets.
A workflow is also a data model
A form is often treated as the beginning of a process and a spreadsheet as its destination. In practice, the important question is what happens between them—and what must remain stable afterwards.
An application may be reviewed, corrected, assigned to a branch, connected to a course, and later used in document production. If those stages rely only on names, email addresses, or manually copied rows, small inconsistencies accumulate quickly. A spelling variation, an empty field, or a duplicated submission can become a problem much later, when the original context is harder to reconstruct.
Stable identifiers provide a spine for the workflow. They allow one record to be recognised across forms, tables, scripts, and documents even when other values change. But an identifier is useful only when the surrounding model is clear: which entity it identifies, when it is created, which system is authoritative, and how relationships between records are represented.
The technical design therefore begins with the process itself. What are the stages? Which decisions change a record’s status? Where can corrections be made? Which values come from users, and which are generated by the system? Data modelling is not a separate layer added after administrative work. It is a way of making that work legible.
Validation is more than finding empty cells
Missing values are the most visible errors, but they are rarely the only ones that matter. A field can be filled and still be inconsistent with another part of the workflow.
Useful validation operates at several levels:
- Structural checks confirm that required fields, identifiers, and expected columns exist.
- Value checks normalise spelling, formatting, and controlled categories.
- Relational checks compare connected sources and reveal mismatches between selections, assignments, and statuses.
- Duplicate checks look beyond identical rows and identify records that may represent the same submission or person.
- Process checks flag combinations that are technically possible but do not make sense at a particular stage.
These checks should not silently overwrite uncertain cases. Automation is most reliable when it separates corrections that can be made safely from exceptions that require human judgement. A good validation report does not merely say that something is wrong; it shows where the conflict occurs and gives a coordinator enough context to resolve it.
Automation should preserve provenance
The appeal of automation is speed, but speed is not its most important contribution. A well-designed script also makes repetitive transformations consistent and reviewable.
In my workflow, automation supports tasks such as generating identifiers, normalising values, detecting missing or conflicting information, comparing records across sources, and preparing structured inputs for documents and reporting. The tools can vary—Google Apps Script, JavaScript, Python, pandas, or JSON—but the underlying questions remain the same.
Can we tell where a value came from? Can we distinguish an original response from a later correction? Can the operation be repeated? Can someone inspect the result without reading the entire codebase?
Preserving provenance means avoiding a single opaque table in which raw data, corrected data, decisions, and outputs become indistinguishable. Even a modest workflow benefits from a clear separation between source records, normalised values, operational status, and public or generated outputs.
Data minimisation is part of the architecture
Educational workflows inevitably involve personal information. This makes data minimisation a design requirement rather than an optional policy statement.
Not every task needs access to every field. Validation can often work with identifiers, categories, and status values rather than complete personal records. Public demonstrations should use synthetic or abstracted data. Access to operational sources should be limited according to responsibility, and exported files should contain only what their purpose requires.
This principle also improves the system technically. Smaller, clearly scoped datasets are easier to validate, document, and maintain. Privacy and data quality are not competing goals; careful boundaries support both.
Infrastructure is the part that makes work dependable
Data infrastructure is often invisible when it functions well. Forms open and close when expected, records retain their identity, inconsistencies appear before documents are produced, and coordinators can understand what needs attention.
That reliability does not come from one large application. It emerges from many deliberate choices: stable identifiers, documented statuses, controlled values, explicit validation rules, reproducible scripts, and room for human review.
For me, this is where technical and academic work meet. The purpose of the system is not automation for its own sake. It is to make complex collaborative work more accurate, traceable, and sustainable.