---
title: Day 24 — The Data That Trained It
type: newsletter
date: 2026-07-31
source: linkedin
summary: "Day 23 opened the model's environmental receipt: every model in production carries the footprint of the training run that produced it, spread across every inference it serves. Today's issue turns to what came before the training run. Every AI model remembers…"
newsletter: Green AI Bytes
draft: false
---

Day 23 opened the model's environmental receipt: every model in production carries the footprint of the training run that produced it, spread across every inference it serves. Today's issue turns to what came before the training run.

Every AI model remembers the data that trained it. Few teams remember the footprint of producing that data. The most expensive part of a training project can happen before training ever starts, in the compute and storage of the pipeline that assembled, labeled, augmented, versioned, and staged the dataset the run will consume. That pipeline has a receipt of its own. This issue introduces it, names it, and gives it a place on the ledger. Call it the **Dataset Receipt**.

Day 18 covered the data that AI systems leave behind at production time: chat histories, tool traces, orphan indexes, evaluation outputs. This issue is about the data AI systems consumed before production began. Different phase, different set of artifacts, and usually a different owner. The Dataset Receipt is real, largely fixed, and rarely disclosed. Community datasets amortize their Dataset Receipt across every downstream user, the same way open-weights models amortize their training receipt across every adopter. Self-produced training data does not. If a team collected, labeled, and augmented its own dataset for a fine-tune, the full Dataset Receipt sits on that team's ledger. The training run gets a line on the bill. The Dataset Receipt that produced its inputs often does not, and that gap is the topic of this issue.

A machine learning team at a healthcare analytics company is preparing a fine-tune of an open-weights base model for a specialized clinical task. The base model was chosen carefully. The training run itself is right-sized, using parameter-efficient methods to keep the compute bounded. The team has done the Day 23 discipline well.

The sustainability review asks a question the team has not costed. What did it take to produce the dataset the run will consume, and where does that footprint live on the ledger?

The audit surfaces three patterns that show up across almost every serious ML pipeline.

**The Label at Scale.** The clinical dataset required expert annotation. Roughly two hundred thousand examples were labeled over eighteen months, through a mix of human annotators using web-based tools and LLM-assisted pre-labeling that was then verified. The web tools ran in the cloud, drawing compute the whole time. The LLM-assisted pre-labeling made inference calls for every candidate example, some of which were then discarded during quality review. Millions of inference tokens can be consumed simply preparing the dataset before a single training epoch runs. None of it had been on the training footprint estimate.

**The Synthetic Fill.** For rare clinical categories, the team used a larger LLM to generate synthetic training examples. Synthetic data generation is inference, priced and metered like any other inference, drawn from the same grid. Every synthetic sample that ended up in the training set was an inference call that never served a user. Every synthetic sample that was generated and then rejected during filtering was an inference call that produced nothing at all. The synthesis budget across the project was, in aggregate, a training-scale compute expense in itself.

**The Dataset Sprawl.** The dataset lived in nine places. The primary versioned store. Snapshots from three ablation experiments. Regional staging copies for the training runs. A pre-processed cache used by the evaluation harness. A researcher's notebook environment. The compute pipeline that assembled the training splits kept its own working copy. Sprawl is not a passive storage problem. Every copy triggers duplicated preprocessing, duplicated indexing, duplicated transfers between regions, and duplicated backup jobs. Each duplicate carries its own operational compute cost every time it is refreshed, and its own share of the Dataset Receipt every time it is regenerated. The team knew the primary store existed. The eight others were discovered during the audit.

> The training run is a bill you can see. The data pipeline that fed it is a bill that arrived before it.

The obvious objection is that data-pipeline work has always been part of ML, is often larger than the training run in effort, and treating it as a separate emissions category feels like scope creep. That is a real concern. The counter is that pipeline effort has always been carried somewhere on the ledger, usually invisibly, and naming it changes which decisions get made. If a team knows that labeling two hundred thousand examples has a real carbon cost, the decision to label two hundred thousand versus twenty thousand starts to be a decision that carries a footprint alongside a budget.

### The Data's Origin

Training data comes to a team in three shapes, and each shape carries a different amortization story for the Dataset Receipt.

**Inherited.** Community datasets, open corpora with wide adoption, and licensed corpora used by many customers. The pipeline that produced them was incurred once by the originator, and the footprint is amortized across every downstream user. A team fine-tuning on the Pile, on Common Crawl, on a public medical imaging benchmark, is inheriting a dataset whose Dataset Receipt has been amortized to a very small per-user share. Data amortization works the same way as model amortization.

**Purchased.** Licensed datasets from commercial providers. The provider incurred the pipeline, the team pays for the license, and the emissions share sits somewhere on the provider's ledger. Adopters can and should ask for a per-license emissions estimate, but few providers publish one today.

**Self-produced.** Data collected, labeled, augmented, or generated by the team itself. The full Dataset Receipt sits on the team's ledger. Self-produced data is where practitioners have the most control, and where the largest embodied numbers usually live.

The three shapes are not mutually exclusive. Most serious training runs use a mix: an inherited base corpus, a purchased specialty dataset, and a self-produced fine-tuning set. The effective footprint is a weighted combination of all three. The self-produced share is the one that carries the highest cost per team and the highest lever.

Four rungs move the Dataset Receipt from an off-book cost into a governed one.

**Source.** Before producing any new training data, ask whether an inherited or purchased dataset would serve. Community datasets amortize their Dataset Receipt across the entire adoption base, and adopting one is often the largest single embodied action available to a project before writing a line of pipeline code. Self-produced data should be reserved for the cases where inherited data genuinely does not fit the task, the domain, or the compliance envelope. The greenest training dataset is often the one someone else already assembled.

**Scope.** When self-production is genuinely required, size the dataset to the task. Labeling more examples than the task needs draws annotation compute against no measurable capability gain. Augmenting more aggressively than the task needs multiplies pipeline compute and storage. Every additional example in the training set also extends the training run itself, cascading Dataset Receipt into training receipt. Right-sizing the dataset is the direct data-pipeline analogue of right-sizing the training run from Day 23.

**Synthesize with Care.** LLM-generated synthetic data is inference in disguise. Every synthetic sample carries the same operational carbon per token as any user-facing inference, drawn from the same grid. Synthesis is a legitimate technique for rare categories, privacy-sensitive domains, and rapid iteration, but it is not free. Use it where the trade against real-data collection genuinely wins, and measure the synthesis budget the same way training compute is measured. A synthesis budget that quietly exceeds the training-run budget is a signal that the pipeline is doing something the training run was supposed to.

**Archive and Retire.** Datasets that no longer serve their original purpose keep drawing storage cost and keep carrying an unamortized Dataset Receipt. Not every dataset that has finished its immediate use should be deleted outright: some ablation snapshots, some experiment corpora, and some regulated data have legitimate long-tail value and belong in cold archival storage where the cost drops but the artifact remains available. Others have no such claim and should be removed entirely. The decision requires versioning discipline as the prerequisite: named datasets, lineage records, ownership assignments, and retention rules. Once lineage exists, archival and retirement become possible. Experimental branches, ablation snapshots, and researcher working caches all have a natural end-of-life, whether that end-of-life is a cold archive or a delete. A dataset registry without an archival-and-retirement policy is an inventory that only grows.

The Dataset Receipt crosses every seat at the table, and it has usually been claimed by the ML team alone.

**Engineering.** Data-pipeline decisions become design-review decisions. How much to collect, whether to synthesize, how to version, and whether to source from a community dataset are all questions with a carbon dimension the engineering seat is well-placed to answer at build time.

**Architecture and CTO.** Dataset reuse joins the enterprise architecture conversation. A stack that shares one training corpus across three fine-tunes has one Dataset Receipt. A stack that produces three separate corpora has three. Cross-team dataset registries, shared feature stores, and shared labeling infrastructure turn dataset reuse into an architectural default rather than a happy accident.

**Platform and Infrastructure.** Data-pipeline compute and dataset storage become metered categories, sitting alongside training runs on the same dashboards. Labeling tools, synthesis pipelines, and versioned stores are provisioned components, and provisioned components deserve utilization targets. Archival tiers become a first-class storage layer, not an afterthought.

**Sustainability and ESG.** The reporting boundary widens again. Emissions disclosures that already covered operational inference, hardware embodied carbon, and training-run embodied carbon now extend to the Dataset Receipt behind each training run. The number is more complete. The disclosure is more defensible.

**Business and Product.** The decision to build a proprietary training corpus is a decision to carry the Dataset Receipt alongside the strategic value the corpus is meant to create. Some proprietary corpora genuinely earn their receipt, in defensibility or accuracy. Others do not. Making the footprint visible makes that decision honest.

**Five seats, one pipeline.**

The work begins with a dataset inventory and ends with a lineage policy.

List every training dataset behind every model in the production stack. For each, record five fields:

* Origin (inherited, purchased, or self-produced)
* Pipeline footprint (Dataset Receipt estimate: published for community, license disclosure for purchased, internal accounting for self-produced)
* Copies (count across primary stores, regional stages, experimental snapshots, and researcher environments)
* Current status (active production, active development, dormant, or unused)
* Retention decision (active tier, cold archive, or retirement candidate)

Order the list by the largest self-produced Dataset Receipt. Pick the top entry and ask three questions. Could an inherited or purchased dataset have served in place of, or alongside, the self-produced one? Is the number of copies justified by an active use case, or is it sprawl the team never revisited? And for each copy, is the right destination an active tier, a cold archive, or retirement?

Publish the dataset inventory, the origin tagging, the copy count, and the archival-and-retirement decisions in the AI data governance guidelines. Where a copy has no active use, move it to cold archival if it has residual value, retire it if it does not. Where a self-produced dataset could be partially replaced by an inherited one, plan the transition.

**One inventory, one lineage, one policy. That is the work.**

> Before the model was trained, the data was.

Day 25 closes the embodied arc. Four categories will have been named, each with its own receipt and its own lever. The final issue in this arc consolidates the four into a single discipline that carries through to the governance arc that follows.

---