Day 18: The Forever Data
Day 17 examined what every call carries on the way in — the system prompt, the tool catalog, the examples, the payload that pays a bill on every request whether it needs to or not. Today’s issue steps back from the call itself.
Every AI feature leaves something behind. Chat logs. Tool traces. Embeddings. Evaluation datasets. Prototype vector stores. Feedback records. Call it the tail — the data that keeps existing after the feature stops changing. Long after the feature stops changing, the data keeps growing.
Every AI system carries two footprints — the work it performs, and the data it leaves behind. Days 3 through 17 have been about the first. Today’s issue is about the second. The persistence footprint usually becomes visible only three times: during a storage bill review, during a model upgrade that triggers a full re-embedding, or during a compliance audit. Between those moments, it grows quietly, in every account, at every scale. Today’s principle is that the data lifecycle around AI is itself an emissions and cost surface, and treating it as one changes the numbers materially.
A B2B SaaS platform ran a retrieval assistant across roughly forty million customer documents. Over eighteen months, the embedding model was upgraded three times — each upgrade a routine drop-in as far as the application code was concerned. Each upgrade also triggered a full re-embedding of the entire corpus, because embeddings are specific to the model that produced them. Nobody added the re-embedding to the upgrade review. An infrastructure review, prompted by an unrelated storage question, discovered that the three re-embeddings together became one of the largest non-production AI infrastructure expenses on the account that year.
The same review surfaced three patterns that show up across almost every AI system at scale.
The Keep-Everything Log. Chat histories, tool traces, and evaluation outputs are retained by default because deletion feels risky, storage feels cheap, and no one has been given the authority to say expire this. Two years later, the store is the size of the corpus itself, and none of it is being read.
The Re-Embed-on-Upgrade. Every embedding model version produces a different vector space. When the model upgrades, the entire corpus is re-embedded — sometimes on schedule, sometimes automatically. The compute required to re-embed a very large corpus can rival months of normal inference traffic, and it lands in the model provider’s inference bill under a line item nobody thinks of as an upgrade cost.
The Orphan Index. Deprecated features, abandoned prototypes, and experimental products leave live vector stores, feature indexes, and log streams behind them. Ownership dissolves when the team moves on. The infrastructure continues to charge, to draw power, and to hold data that no product still reads.
The product ships, the feature retires, the data stays. That is the default, and the default has a bill.
The objection is real. Some data has to be kept. Regulated industries have retention requirements measured in years. Some evaluation datasets have long-term scientific value. Some logs are indispensable for debugging. The counter is that retention should be a purposeful decision per class of data, not a system-wide default. Compliance-mandated data is a small fraction of what most teams keep. The rest is inertia — data that stays because nothing has ever told it to go.
Data-lifecycle discipline for AI systems responds to a sequence, and each rung reduces the load on the next.
Inventory. Inventory every AI data store once. Vector databases, embedding caches, logs, evaluation datasets, prototype indexes. Record its size, owner, purpose, and growth rate. Most teams discover orphaned stores before the inventory is complete.
Tag. Once the inventory exists, each store gets a lifecycle role. Production. Evaluation. Compliance. Historical archive. Orphan. The role determines the retention. Production data lives on the hot tier and gets queried. Compliance data lives on the coldest tier and gets read on request. Historical archives are reduced to only what has to persist. Orphans are marked for retirement. The tag makes the lifecycle explicit rather than implicit.
Retire. Set time-to-live rules at the class level. Thirty days for debug logs, one year for evaluation runs, whatever the compliance regime requires for regulated data, nothing at all for feature indexes whose feature no longer ships. Actually delete what has been tagged for retirement — deletion, not archival. Retention as a default is the pattern to break. Retention as a decision is the pattern to establish.
Reuse. Treat re-embedding as an intentional cost, not an automatic step. Three principles govern the upgrade decision. Don’t re-embed automatically when a new model version lands. Re-embed incrementally — one query at a time, or only the corpus that will actually be read. Justify the upgrade with a real quality delta weighed against the compute burn. The decision becomes a decision, with a cost weighed against a benefit.
Few AI engineering problems span as many teams as data lifecycle, and the sequence only shifts when all five seats are looking at the same tail.
Engineering. Every vector store, log stream, and embedding cache your code writes to becomes someone’s cost forever unless you set the retention explicitly. The default is keep. The discipline is decide.
Platform/Infrastructure. AI storage belongs in capacity planning the same way compute does. Track growth rate per store, tag ownership at provisioning time, and alert on stores that have not been read in a defined window.
Architecture/CTO. Data lifecycle is an architectural concern, not a housekeeping task. Retention policy, storage tiering, and re-embedding strategy belong in the same design reviews as model selection and routing.
Sustainability/ESG. Storage growth and re-embedding events both contribute to the operational footprint of AI systems. The upgrade that triggers a full re-embedding is a discrete event with a measurable footprint, and it belongs in the reporting alongside training and inference.
Business/Product. Every feature you ship has an operational tail. Retiring a feature means retiring its data, its indexes, and its logs. The tail is part of the total cost of ownership, and treating it as one changes what sunset actually means.
Five seats, one tail.
The work fits inside a normal quarter of engineering time, and the first week returns most of the value.
Inventory every AI-related data store on one account — vector databases, embedding caches, chat and tool trace logs, evaluation datasets, feature indexes. Size each one, note the growth rate, and identify the owner. Assign a lifecycle role to each store: production, evaluation, compliance, archive, orphan. Retire the orphans this week — deleted, not archived. Set a retention policy on one class of retained data next week, starting with the largest. Add re-embedding cost to the model upgrade review checklist, so no future upgrade goes through without an explicit decision on what to re-embed, when, and why.
One inventory, one tag, one retirement, one policy. That is the work.
If one line carries forward from this issue, it is this.
Storage feels free. AI-scale storage, kept forever, is not.
Day 19 continues the arc on the system around the model, moving from the data that persists between calls to the capacity that stays warm between calls — the standing infrastructure of production AI, and what it costs to keep the lights on when nothing is being asked.