The Shape of Data I: The Ceiling. The shape you give data decides which questions you can ever ask of it.

The question that had no answer [summarize]

Two numbers, both real, describing different worlds

A client asked me which campaign made the money. I couldn't tell him. The analysis wasn't hard, and the tooling wasn't missing. The answer didn't exist.

His dashboard reported record performance. His bank statement reported a slow bleed. Both numbers were correct. The dashboard counted conversions the ad platform claimed, and the platform claims a conversion whenever somebody who saw an ad later bought something inside a lookback window, whether or not the ad had anything to do with it. The bank counted deposits. Nothing reconciled the two, because nothing in the system had ever recorded which order came from which click. That link was never written down. Once you have not written something down, no amount of analysis puts it back.

This pattern shows up in most accounts I audit, and the recoverable waste usually lands somewhere between 10 and 30 percent of spend that can be reallocated or cut without hurting revenue. The waste is the smaller problem. The owner had been paying an agency, a BI contractor, and a SaaS analytics vendor, all of them doing competent work on top of data that could not answer his question. Three vendors, none of them lying, none of them able to help.

The failure lives in the schema

Here is what his orders table looked like, and it looks like this almost everywhere:

An order row carried an order id, a customer email, a total, a timestamp, and a status. Clean, normalized, well indexed, backed up nightly. A good table. It had no column for the click that preceded it, no session id, no campaign id, no anything that pointed back at the marketing that produced the sale. Meanwhile the ad platform had click records with campaign ids and no order data, and the two systems shared exactly one field that might match: an email address the customer sometimes typed differently in each place.

So the question "which campaign produced this revenue" required a join between two tables that had no key in common. You can approximate it. You can match on email and accept whatever error rate that produces, or you can build a model that assigns credit probabilistically and call the output attribution. What you cannot do is answer the question, because the fact that would answer it was never recorded. The shape of that orders table set a ceiling on what the business could ever know about its own marketing, and the ceiling was set years earlier by whoever wrote the first version of the checkout.

Nobody made a bad decision. Somebody built a checkout that stored what a checkout needs, which is what to ship and where and for how much. The campaign id wasn't relevant to shipping an order, so it wasn't in the model. The decision that foreclosed attribution forever was made by a competent engineer solving a different problem correctly, and it cost nothing at the time.

Why this class of failure stays invisible until somebody asks

A missing index announces itself. Queries get slow, somebody profiles, the fix is obvious and cheap. A missing column announces nothing. The system runs fine, the dashboards populate, the reports go out, and everything looks healthy right up until a question arrives that the schema can't serve. Then the failure surfaces as a vague sense that the numbers are untrustworthy, which gets diagnosed as a data quality problem, which triggers a cleanup project, which does not help, because the data is clean. It is clean and it is the wrong shape.

The people best positioned to catch this are the ones least likely to be in the room. The engineer who designed the table knew exactly what it did and did not carry. Three years later that engineer is gone, the table has been through two migrations, and the CFO is asking a question nobody thought to ask when the shape got decided. Shape decisions are made early, by people optimizing for something else, and they are the least reversible decisions in the entire stack.

That is what this series is about. Three parts. This one establishes that the ceiling exists, where it comes from, and how to find yours before you hit it. The second walks the five shapes data actually takes and what each one is structurally bad at. The third covers the three properties almost every system defers and cannot: when a fact was true, where it came from, and what contradicts it.

Shape is the logical model [summarize]

Shape means the logical model. What one row stands for. Which things in the business exist as first-class objects with their own identity. Which relationships between those objects can be written down at all. Which keys get captured at the moment an event happens, as opposed to reconstructed later by somebody matching strings. Everything underneath that is storage: file format, index, partition scheme, engine, region, replication topology, whether the bytes live on spinning disks in Virginia or in object storage priced by the request.

Storage gets almost all of the attention and almost none of the consequence. Every storage decision is reversible at a known price. Every logical decision sets a boundary on the question space and most of them are permanent.

What Codd made free, and what he left expensive

This split is a founding guarantee of the relational model rather than a stylistic preference, and it's the reason the industry works the way it does. E. F. Codd's A Relational Model of Data for Large Shared Data Banks (CACM 13(6), pp. 377-387, 1970) introduced physical data independence: storage layout, indexes, and access paths can change without changing the logical schema or the queries written against it. A database administrator adds an index on a Tuesday afternoon and not one line of application code changes. That's a deliberate engineering achievement, and it's the reason storage feels cheap to change.

Physical data independence protects your queries from your storage. Nothing in the relational model, or in anything that succeeded it, protects your queries from your logical model.

That guarantee runs in exactly one direction, and the direction it doesn't run is the one nobody budgets for. Change the logical model and you change the set of expressible questions, which is the definition of what a query language can be asked. That asymmetry is what the rest of this series runs on, and it's been sitting in the literature since 1970 where anyone could read it.

Sorting real decisions into two buckets makes the difference concrete. The top three rows below are things teams agonize over in architecture reviews. The bottom three are things that get decided in a pull request nobody reviews carefully.

The migration that changes nothing

A pattern I see repeatedly: a company moves its warehouse from Postgres to Snowflake, or from Redshift to BigQuery, budgets serious money for it, and expects the reporting to get better. Reporting gets faster. Concurrency improves. Somebody writes a satisfying internal post about cost per query. The set of answerable questions is byte-for-byte identical to what it was before, because a migration's success criterion is fidelity. The tables came over. The grain came over. The missing foreign keys came over, still missing.

A migration preserves the logical model on purpose, so it preserves the ceiling on purpose.

Nobody hides this. It's written into the acceptance criteria as row-count parity and checksum matching, which is exactly correct engineering for a migration and exactly why the ceiling survives untouched. What breaks is the expectation attached to the migration, which usually got set in a vendor conversation where "modern data platform" did a lot of work that "faster copy of your current ceiling" would have done more accurately.

Run the comparison the other way and the economics invert. Adding one column to a checkout write, a session identifier carried from the ad click through to the order row, is an afternoon of engineering and a deploy. It moves the ceiling further than a platform migration does at a tiny fraction of the cost, because it makes a class of question expressible that was previously unaskable. The cheapest work in the stack is usually the work that moves the ceiling, and the most expensive work usually doesn't touch it.

One question tells you which kind you're looking at

Any proposed change to a data system can be sorted with a single test. After this change lands, can I express a question I couldn't express before, or does a question I used to be able to express become inexpressible? No to both means it's storage, and you judge it on cost, speed, and reliability like any other infrastructure work. Yes to either means it's shape, and you judge it on which questions.

The test is easy to run and it catches the thing reviews normally miss, because the two directions get wildly different scrutiny. Changes that add expressible questions arrive labelled as features and get discussed. Changes that remove them arrive labelled as cleanup, and all four of these lower the ceiling while reading in a diff as tidying up:

• Dropping a column nobody queries. Query traffic measures which questions have been asked so far, which is a different thing from which questions matter.
• Collapsing two tables that "always join anyway." They always join today. The separation was recording that two things are distinct, and merging them asserts they never will be.
• Deduplicating records that are "obviously the same customer." Obviously is doing enormous work in that sentence, and the merge is usually irreversible.
• Rolling up a table nobody reads at the row level. The rollup is fine. Deleting the rows underneath it is the part that sets a permanent floor on resolution.

Deduplication is the version of this that costs the most and looks the most responsible. A team notices two customer records sharing an email address, merges them, and updates the orders to point at the survivor. Sometimes those really were one person. Sometimes they were a couple sharing a household address, or one person's work and personal accounts with genuinely different buying behavior, or a small reseller who bought under two names for two of their own clients. The merge was a judgment call made in bulk by a matching rule, and if it rewrote the source rows in place, the evidence needed to review the judgment went with them. Everything downstream now inherits a decision nobody can inspect. Part III of this series covers why the fix is to record the merge as a claim with its own provenance and leave the originals alone, which costs almost nothing at write time and is impossible to retrofit.

A column with no reads costs you a fraction of a cent. Losing it costs you a question class forever. The default should be keep, and the burden of proof should sit with deletion.

The most dangerous change in a data system is deleting a key nobody queries. Nobody queries it because no report was ever built on it, and no report was built because nobody asked the question, and nobody asked because the person who would ask hasn't been hired yet or hasn't hit the problem yet. Current query traffic is the weakest available evidence about whether a key will matter, and it's the evidence almost every cleanup decision runs on. A column with zero reads costs you storage measured in fractions of a cent. Losing it costs you a question class forever. That trade is lopsided enough that a deletion should have to argue for itself.

Which leaves the harder problem. Knowing that shape sets the ceiling doesn't tell you where your ceiling actually is, and the ways a question dies are specific enough to be worth naming individually.

Three ways a question dies [summarize]

Questions die three ways. The distinction matters because the three have different symptoms, different diagnostics, and wildly different repair costs, and teams routinely apply the fix for one to a case of another. A grain problem gets treated as a data quality problem. A key problem gets treated as a modeling problem. A structure problem gets treated as a tooling problem and somebody buys a graph database for a business whose actual issue is that nobody wrote down the campaign identifier.

Here's the taxonomy the rest of this series runs on. The third column is the one worth memorizing, because it's what decides whether you're looking at a project or a permanent condition.

Death one: the grain was too coarse

Ralph Kimball and Margy Ross put grain declaration first in The Data Warehouse Toolkit, 3rd edition (Wiley, 2013), ahead of dimensions, ahead of keys, ahead of everything. The grain of a fact table is what one row means, and it sets the maximum resolution of every analytic question anyone will ever ask about that fact. One row per order line, one row per order, one row per customer per day, one row per channel per month. Each of those is a different ceiling, and each one is chosen in about four minutes by whoever is writing the transform.

You can produce a thumbnail from a full-resolution photograph any time you like. You can never produce the photograph from the thumbnail. A rollup is a thumbnail generator pointed at an event stream.

Aggregation is non-invertible. From a sum you can't recover the addends or their distribution, and no amount of later cleverness changes that, because a many-to-one map has no inverse. Image compression is the same operation with the same consequence, which is why the intuition transfers without any explanation. Running a rollup is safe as long as the full-resolution rows stay on disk. It stops being safe the moment a retention policy deletes the detail and keeps the summary, which is the single most common way a company destroys its own future analytics while cutting its storage bill.

A restaurant group I looked at stored daily totals per location, which is a perfectly reasonable shape for a P&L. Their live operating question was which hours were unprofitable to stay open, and that question needs one row per hour per location with labor cost attached. The data to answer it had existed, briefly, inside the point-of-sale system before the nightly job rolled it up and the vendor's export window closed behind it. The company was making a staffing decision worth real money on intuition, while sitting on a warehouse that recorded the exact events they needed at a resolution one notch too coarse to see them.

The diagnostic is mechanical. For any fact you care about, find the finest grain that any table at any layer carries, including staging tables, raw landing zones, and vendor exports still sitting in a bucket. That grain is your resolution limit for that fact, forever. If nothing anywhere carries the finer grain, the finer question is foreclosed. Fixing it going forward is usually cheap and often takes a day. Fixing it backward is a wish.

Death two: the key was never captured

The second death is the one from the opening of this article, and it has a precise name in two literatures. At the relational level it's a loss of referential integrity: two tables both hold real records, and nothing connects a row in one to a row in the other. Across systems it's a failure of entity resolution, also called record linkage, and that field has been rigorous about its own limits since Ivan P. Fellegi and Alan B. Sunter published A Theory for Record Linkage (JASA 64(328), pp. 1183-1210, 1969).

What Fellegi and Sunter established is worth stating carefully, because it gets cited as though it solves the problem. Their framework gives an optimal decision rule for matching records without a shared key, and the optimal rule still produces a region of uncertain pairs that must be declared possible matches carrying explicit error rates. That's the mathematically best case, achieved with full information and correct parameters, and it still hands you an error rate rather than an answer. A deterministic key, once not written down, can't be recovered from the content of the records, because the content was never sufficient to determine it. That's the entire reason keys exist.

Three flavors of this show up constantly, and they look different enough that teams don't recognize them as the same failure:

• The acquisition. Two companies merge, both have a customer base, neither issued the other's identifiers. Every subsequent "unified customer view" is a matching exercise, and the merged file's error rate silently becomes the denominator of every cross-sell metric leadership looks at.
• The support desk. Tickets are keyed by whatever address the customer wrote from, and accounts are keyed by whatever address the buyer signed up with. Those differ often enough that ticket volume per account is an estimate, which means "do unhappy customers churn" is an estimate about an estimate.
• The click and the order. Covered above. Worth noting that the platforms know this, which is why their reporting is built on their own lookback windows rather than on your order table.

The propagation is what makes this expensive rather than merely annoying. A matched identifier gets written into a warehouse table once, and from that write forward it looks exactly like a recorded fact. Downstream consumers join on it, dashboards aggregate it, a forecast gets built on it, and the error bar that lived in the matching job's documentation never travels with the column. Uncertainty stops being visible at the first join, and every layer above that treats a guess as a measurement. The diagnostic question is short: for each join your reporting depends on, was the key written by the system that observed the event, at the moment it observed it? If it was derived later by comparing strings, you own a permanent error rate and you should know its size.

Death three: the structure was flattened

The third death is the hardest to see, because there's no missing column to point at. The relationship itself had more participants than the schema had slots, so the write silently kept the participants that fit and dropped the rest. Nothing failed. No error was logged. The row looks complete.

A single manager_id column on an employees table is the clean example. It models a tree, one parent per node, and it works perfectly for a company that is a tree. A person with a functional manager and a project lead doesn't fit, so somebody picks one, usually the one payroll needs. The org chart branches along several dimensions at once, and a column that permits one parent can only ever record one of them. Every question about the dimension that lost the coin flip is now unanswerable, and the table looks fully populated.

The general form is an n-ary fact squeezed into a binary slot. Write the fact as a plain English sentence and count its participants. "Alice approved this invoice on behalf of Bob, under a named exception policy, during the quarter-end freeze." That sentence has five participants. An approved_by column holds one. The four missing participants are also precisely what an auditor asks about, which is why audit findings so often arrive as a discovery that the data cannot answer the question rather than as a discovery of wrongdoing.

Count the participants in the sentence. Count the slots in the schema. The difference is exactly what the system threw away, and it threw it away without logging anything.

Document stores hit this from the opposite direction and land in the same place. Martin Kleppmann frames the trade-off in Designing Data-Intensive Applications, Chapter 2 (O'Reilly, 2017) as locality against relationship expressiveness. A document wins when related data gets read together, and many-to-many relationships aren't naturally representable inside one document, so modelling them requires references across documents. Those references are joins without the enforcement, which is how a team ends up hand-rolling referential integrity in application code and discovering its gaps in production.

A reference across documents is a join without the enforcement, which is how a team ends up hand-rolling referential integrity in application code and finding its gaps in production.

Relational theory is precise about when a split is safe, and the precision is useful here mostly for what it doesn't cover. A decomposition preserves information when it's lossless-join and dependency-preserving, and Ronald Fagin's Normal Forms and Relational Database Operators (ACM SIGMOD, pp. 153-160, 1979) is the source for the project-join case. Denormalization destroys answerability in three ways: it aggregates to a coarser grain without keeping detail, it drops keys needed to reconstruct joins, and it co-locates independent facts so no copy is authoritative. Every one of those maps to a death in the table above. What no normal form addresses is a relation that was never represented at all, because normalization operates on relations you wrote down.

All three deaths share one property, and it's the reason this is worth a Tuesday afternoon rather than a roadmap item. Each is nearly free to prevent before the data starts flowing and impossible to repair after. Which raises the question people reasonably ask next: if a question is unanswerable, how is that different from a question that's merely expensive?

Unanswerable is different from hard [summarize]

Teams use "we can't answer that" for two situations that have nothing in common. One is a budget problem. The other is a physics problem. Sorting them takes about ten minutes and it changes what you should do with the next quarter's money, so it's worth having precise vocabulary rather than a shared shrug.

Hard means the answer costs money

A hard question has a correct answer sitting in the data, and reaching it costs compute, engineering hours, vendor spend, or all three. A full scan across billions of rows. A graph traversal to a depth where the intermediate result set stops fitting in memory. A scheduling or routing problem where the exact solution is computationally intractable and the practical answer is a good approximation with a stated bound. Every one of those is genuinely difficult and every one of them responds to money. The defining property of a hard question is that more budget buys a better answer, and the relationship holds even when the curve is brutally steep.

This is the class the entire data industry is built around, and it serves it well. Query engines, columnar formats, distributed execution, caching layers, approximate algorithms with error guarantees: all of that machinery exists to move hard questions from unaffordable to affordable. When a vendor demo makes something look easy that used to look impossible, this is almost always the category they're operating in, and the demo is accurate.

Unanswerable means the fact was never observed

An unanswerable question has no correct answer available at any budget, because the fact that would determine it was never written down anywhere. Adding compute changes nothing. Adding engineers changes nothing. Hiring a better analyst changes nothing, and hiring a worse one changes nothing either, which is a useful tell: when the answer's quality is independent of who's working on it, you're looking at a structural condition rather than a skill gap.

A third class sits between them and gets misclassified in both directions. Sometimes the determining fact does exist, just not in the warehouse: it's in a raw application log with a short retention window, an email archive, a vendor's system you can still export from, a backup nobody has mounted since the migration, a spreadsheet on somebody's laptop. That's archaeology, and archaeology is expensive rather than impossible. Teams call things unanswerable that are merely buried, and they call things hard that are actually gone. Both errors cost real money in opposite directions.

Sorting takes minutes once you know the categories. Three questions from one subscription business, all of which had been sitting in the same "the data team is looking into it" bucket for most of a year. Which plan tier has the worst gross margin? Hard, and only barely: the cost data lived in a finance system nobody had joined to the product database, which is a bounded piece of integration work and then it's answered permanently. How many trial users hit the rate limit before they cancelled? Buried, because rate-limit events were emitted to an application log with a retention window that had rolled over for older cohorts, so recent cohorts were answerable and historical ones were gone. Which onboarding email changed the decision to convert? Unanswerable, because email sends were recorded, conversions were recorded, and nothing ever wrote the link between them. Three questions, three categories, three completely different correct responses, and every one of them had been receiving the same response the entire time.

Proving a question is answerable takes one query. The other direction is the hard one.

There's an asymmetry underneath this that explains why ceilings stay invisible for years inside competent organizations. Confirming that a question is answerable is cheap and it terminates. Write one query. Get a row back. Done, one witness, no further work required. Establishing that a question is permanently unanswerable runs the other way, and the direct version of that check never finishes. You'd have to rule out every query anyone could write, every join path through every table, every tool not yet purchased, every vendor not yet called, every model not yet trained, and every analyst sharper than the ones you have. That check is unbounded, so nobody runs it, so the question stays open forever with "we're still working on it" as its permanent status.

The positive is cheap and the negative is priceless, which is exactly backwards from how attention gets allocated. First-level thinking hunts for a confirming query and reports what it finds. Second-level thinking asks what would have to be true for this question to be answerable at all, notices that checking it directly is unbounded, and goes looking for a shortcut that makes the unbounded check finite.

Nobody ever finishes proving a question is unanswerable, so the question stays open forever and "we're still working on it" becomes its permanent status.

A shortcut exists, and it's the entire practical payoff of this article. Instead of searching the space of possible queries, you compute one property of the system: name the event that would have to have been observed for the question to have an answer, then check whether any record of that event exists at the grain the question needs. If no such record exists anywhere, no query can return it, no join path can reconstruct it, no vendor can sell it, and no model can manufacture it. One finite check settles an infinite question, and it runs against the schema rather than against the tooling. The section below turns that into a procedure you can run on a Tuesday morning with a whiteboard.

The move itself is old and it isn't ours. Mathematics spent roughly a century on precisely this problem in a purer form: showing that two shapes are the same takes one witness, and showing they're different would require ruling out infinitely many ways of deforming one into the other. The resolution was to stop examining deformations and instead compute a fingerprint by a fixed recipe, engineered so that no legal deformation can change it. Different fingerprints, different shapes, and you never checked a single deformation. Part II of this series takes that machinery seriously, because there's a branch of mathematics that measures the shape of data literally rather than by analogy. The transferable part here is the structure of the move. When the direct check is unbounded, stop checking and find the property that settles it in one pass. Structure is where the value hides.

What a plausible number costs you

Misclassification has a specific commercial signature, and once you've seen it you can spot it in any quarterly review.

Money spent on a hard question buys an answer. Money spent on an unanswerable question buys a plausible number, and a plausible number is worse than no number at all.

No number leaves a decision explicitly under uncertainty, which keeps the decision-maker appropriately careful and keeps the question alive. A plausible number closes the decision. Somebody moves budget between channels on it, cancels a product line on it, or renews a contract on it, and there's nothing in the delivery format that distinguishes a modeled estimate from a recorded fact. Both arrive as a figure in a cell. The error bar exists in the methodology document and it stops travelling with the number at the first copy-paste.

Then there's the recurring bill. Unanswerable questions generate permanent vendor spend, because no deliverable ever settles them. The engagement ends, the number gets questioned, somebody notices it disagrees with the bank statement, and next quarter a different firm is hired to answer the same question with a different methodology and a different logo on the deck. Nobody in that cycle is doing bad work. The question genuinely doesn't have an answer, so it keeps coming back, and it will keep coming back until somebody fixes the shape that foreclosed it.

If you've never watched the same question come back three quarters running with three different firms' logos on the answer, this section is describing somebody else's business and you can skip ahead.

Which brings up the objection that arrives at this exact point in every conversation I have about this, usually phrased as a solution rather than a question. Can't a model figure it out?

No model on top recovers what the shape discarded [summarize]

The objection deserves a serious answer, because the intuition behind it is trained on real successes. Models read handwriting nobody can read. They pull structured entities out of a decade of unstructured support tickets. They match customer records that defeated every rule a team wrote by hand. Somebody who has watched that happen has correct evidence that models recover things which looked unrecoverable, and it's reasonable to extend the pattern.

The extension breaks on one distinction, and the distinction is sharp enough to state in a sentence. A model can extract a fact that's present but unstructured. A model cannot manufacture a fact that was never recorded. Those are different operations that get sold under the same word.

A model trained on collapsed data inherits the collapse

A model fits a function from inputs to outputs using the data it was shown. Training pressure comes entirely from that data. If the determining variable was never present in it, there's no gradient anywhere in the optimization that points toward the missing fact, because the loss function has no way to notice its absence. What happens instead is more expensive than an error would be.

The model routes the missing variable's influence into whatever correlates with it. When campaign identity was never recorded, a model asked to attribute revenue will lean on the variables that survived: time of day, device, geography, landing page, order value. Those correlate with campaign, because campaigns target them. So the model produces a number, the number is internally consistent, and it's systematically wrong in the direction of whichever proxy happened to correlate hardest in the training window. When the market shifts and the correlation between proxy and principal changes, the model's output changes for reasons that have nothing to do with what actually happened, and nothing in the output signals that.

A model asked to explain something it was never shown will explain it with whatever it was shown. The output looks like an answer because it has the shape of one.

This is the same failure I shipped in 2018 on an automated bidding system for a programmatic display client spending roughly $50,000 per day, and it's covered in full in the Intelligence Engineering essay. The math was correct. The optimization did exactly what the equations said. The system was optimizing a proxy that had decoupled from the principal, and every additional dollar of compute made it better at the wrong thing. A model on top of a collapsed schema is that failure with a friendlier interface.

Probabilistic linkage is rigorous about its own limits

Record linkage is the version of this work that publishes its own error rates, and it's worth understanding because it's the best-case bound on what any matching approach can achieve. Fellegi and Sunter's 1969 framework gives an optimal decision rule for matching records that share no key, and that optimality is proved rather than asserted. The rule sorts candidate pairs into links, non-links, and a middle region of possible matches that carry explicit error rates and are supposed to go to a human.

Two error types live in that middle region and they trade against each other directly. False links merge two different people into one customer. Missed links leave one person counted as two. Tighten the threshold and false links fall while missed links rise. Loosen it and the reverse happens. You can push either error toward zero by pushing the other up, and you can never push both down, because the information that would separate the cases isn't in the records. That's what it means for a key to be gone.

Here's the part that makes this a business problem rather than a statistics problem. That threshold gets set once, usually by whoever wrote the matching job, and it silently decides which channel wins the budget. A loose threshold merges aggressively, which inflates the credit assigned to channels that touch a lot of people, because broad-reach channels produce more near-matches. A tight threshold deflates exactly those channels. A parameter chosen for tidiness in a data pipeline is picking the winner of your media budget, and it appears in no meeting where budget is discussed. I've never seen that number reviewed by anyone with commercial authority, and I've never seen a vendor volunteer it.

Similarity carries no direction

Vector search is the current default answer to "find the relevant thing," so it earns a note here even though Part II handles it properly. An embedding maps text into a fixed-dimensional space, and the map is many-to-one, so it's neither injective nor invertible. Distinct inputs can land on the same vector, and whatever distinguished them is gone at that point. Retrieval built on that map returns neighbors.

Vector search returns neighbors. Neighbors are useful, and they're a structurally different object from answers. Most disappointment with retrieval systems lives in the gap between those two words.

Cosine similarity, the standard scoring function, is a single symmetric scalar. Symmetric means "A caused B" and "B caused A" score identically against each other, because there is nowhere in one number for direction to live. Negation has the same problem, and it's been measured: Nora Kassner and Hinrich Schutze report in Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (ACL, 2020) that "we find that PLMs do not distinguish between negated (“Birds cannot [MASK]”) and non-negated (“Birds can [MASK]”) cloze questions." The knowledge-graph embedding literature exists largely because asymmetric relations need structure that plain similarity doesn't provide, which is a strong signal from the field itself about where the boundary sits.

Three properties of a fact have nowhere to live inside one symmetric number, and each of them is load-bearing in ordinary business questions.

Absence has the same problem one level up, and it's worth flagging because it surprises people who assume better retrieval fixes it. "Which policies were never reviewed" is a set difference, and computing a set difference requires an authoritative list of the full domain. Nearest-neighbor retrieval returns neighbors among things that exist, so there's no passage anywhere for it to find. Negative space has no document, which means no amount of retrieval quality reaches it and the answer has to come from a manifest somebody deliberately maintained.

One overclaim worth disarming, because it gets deployed to end this conversation. The Johnson-Lindenstrauss lemma, from William B. Johnson and Joram Lindenstrauss's Extensions of Lipschitz mappings into a Hilbert space (Contemporary Mathematics, Vol. 26, pp. 189-206, 1984), guarantees that n points can be embedded into a number of dimensions logarithmic in n while preserving pairwise Euclidean distances to within a small relative factor. That's a real and powerful result about distances between points. It says nothing about semantic content, logical structure, or entailment, and citing it to argue that learned text embeddings are near-lossless is an overclaim about a theorem that was never making that claim.

What models genuinely fix, and it's a long list

Disarming that overclaim leaves the tool fully intact, and the useful applications are large enough that reading this section as anti-model would get it backwards. Four categories where a model earns its cost outright:

• Extraction from unstructured text. The fact is present in a free-text field, a PDF, a call transcript, or a log line, and a model turns it into a column. This is recovery in the full sense, and it routinely unlocks questions a team assumed were dead. Run this before concluding anything is unanswerable.
• Candidate generation for linkage. A model proposes match candidates that rule-based blocking would never surface, and then a rule or a person adjudicates. Recall goes up, the error rate stays visible, and nobody pretends the middle region vanished.
• Imputation with a stated mechanism. Filling gaps where the missingness pattern is understood and the uncertainty is reported alongside the estimate. Legitimate statistics with a variance attached.
• Ranking and retrieval over content that exists. Finding the right document out of five million is exactly what the tooling is for, and the alternative is a person with a search box and a bad afternoon.

The rule that separates those from the failure case is short. Models are excellent at reading what's there and structurally incapable of restoring what isn't. Buy the first job enthusiastically. Stop paying for the second.

The commercial trouble is that a buyer can't tell the two jobs apart from the output, because a manufactured number and an extracted number render identically in a cell. Which means the diagnostic has to happen before the work is commissioned, at the point where somebody can still ask which event would have had to be observed. That question turns out to sit at a specific rung of a longer chain, and most teams skip straight past it.

The chain that puts shape last [summarize]

Shape is the last thing you derive. Almost every team treats it as the first thing they decide, and the gap between those two sentences is where most permanent ceilings get installed.

The chain I run has eight rungs and it moves from the operator's pain down to the data shapes that carry it. Nothing on the chain is exotic. The discipline is entirely in refusing to skip, and the line I keep coming back to is that you can't skip from problem to entities any more than you can skip from requirements to code.

Eight rungs, and entities are the last one

Each rung takes constraint from the rung above and produces the raw material for the rung below. Read the middle column as what the rung physically produces, because a rung that produces no artifact was skipped regardless of how much it got discussed.

The chain runs top-down for constraint and bottom-up for evidence. Rung 3 tells rung 8 which things have to exist. Rung 8 tells rung 3 whether the world model was actually observable, which is a real check and it fails often: plenty of world models describe things nobody has any way to see happen. An entity that can't name the rung above that required it is decoration, and decoration in a schema is the cheapest thing to add and the most expensive thing to remove.

The documents table that answered the wrong problem

The cleanest example I have of this is one of my own systems, at two versions. ContentFactory version one had a documents table. That's a defensible choice by every ordinary standard. Developers think in documents. The customer hands you documents. The filesystem stores documents. The word appears in the sales conversation and in the requirements doc and on the whiteboard, so it lands in the schema without anyone deciding to put it there.

The user's actual problem was that the chatbot gave wrong answers. Run that down the chain and the shape it demands looks nothing like a container.

• Story. A support rep asks the assistant a policy question, gets back a confident answer assembled from two unrelated policy sections, and repeats it to a customer.
• World model. Knowledge in that domain lives in claims, each with a source, a scope, and a period during which it holds. A document is a container carrying several of them with no marking of where one ends.
• Features. An answer that cites the specific passage it came from, so the rep can check it in two seconds instead of trusting it.
• Systems. Chunking, embedding, retrieval, synthesis, citation.
• Entities, finally. A chunk model, an entity model, and an entity-relationship model, because those are the grains the systems above actually operate on.

A documents table can't answer "which claim produced this sentence," because a document isn't the grain of a claim. That's death one and death three from the taxonomy above, arriving together: the grain is too coarse to isolate a claim, and the relationship between a claim and the answer that used it has no slot to live in. The failure surfaced as the assistant being unreliable, which reads as a model quality problem and gets escalated to whoever owns model selection. The model was fine. The retrievable unit was a whole document, so the model was reasoning over containers and being blamed for it. The RAG Knowledge Engines entry covers what the fixed version does.

The version two catalog carries 40+ entities across 7 problem domains, a 9-component library, and 15+ systems. A count presented as an achievement is usually a warning, and the derivation is what makes that catalog worth anything. Every one of those entities exists because a specific rung above it required it, and the derivation is what makes the schema defensible when somebody proposes changing it two years from now.

Why the shortcut feels correct in the moment

Nobody skips the chain out of laziness. The jump from problem to schema is rewarded at every step in the short run, and understanding why it's rewarded is the only reliable defense against doing it again next quarter.

• The schema is the first executable artifact. Rungs one through seven produce documents and diagrams. Rung eight produces something that runs, and a running thing reads as progress in a way a world model never will, especially to whoever is funding the work.
• The customer hands you their nouns. Documents, orders, tickets, campaigns, accounts. Those nouns arrive in the customer's own voice, which makes them feel validated rather than assumed, and every one of them looks like a table.
• Migration feels like an undo button. Teams accept a shape they're unsure about because they believe they can change it later, and they can change the storage later. The questions foreclosed in the meantime never come back.

The middle one causes the most damage, so it's worth stating on its own. The customer's nouns name containers, because containers are what people physically handle, and the questions a business needs answered are almost never about containers. They're about events and about relationships between things inside the containers. Which campaign produced the sale. Which claim produced the answer. Which policy exception approved the invoice. Which hour was unprofitable. Every one of those questions lives at a grain finer than the noun somebody said out loud in the kickoff meeting, and a schema built from the kickoff nouns forecloses all of them before a line of application code exists.

There's a fast tell for this. Read your entity list next to the transcript of your first customer call. If the list is the call's nouns with _id suffixed, six rungs got skipped and the skip is now load-bearing.

The catalog is the first build artifact

Where the logical model physically lives determines whether anyone ever reviews it. My rule is that everything starts as a typed data catalog, and that catalog is the first build artifact of any system: storage backends, API layers, frontends, and agents all derive from it, and none of them get to redefine what the data is. Pydantic models are what I use for it, and the specific library matters far less than the property it buys.

That property is a single reviewable location for shape. When the logical model lives implicitly across a pile of table definitions, an ORM's classes, a set of frontend types, and a JSON schema pasted in an API doc, nobody reviews the shape, because shape has no address. Four representations drift independently, each one is locally correct, and the question of what the system believes a customer is has four answers.

A shape decision nobody can locate is a shape decision nobody reviews, and unreviewed shape decisions are how ceilings get installed by accident.

The composition model underneath matters for the same reason. Entities are rows, components are columns, and systems are queries: an entity is an identity with a set of components attached rather than a position in a class hierarchy. Inheritance forces a single axis of specialization, and real domains branch along several axes at once. A customer who's also a supplier, a document that's also a contract, an employee who's also a channel partner: each of those breaks a tree and each of them is ordinary. Composition lets an entity carry both sets of components without anyone having to pick which parent it belongs to, which is the same argument as death three, made one layer up. The Data Pipelines entry works through what each stage does to the shape as data moves, and the Intelligence Engineering essay carries the five-layer stack from events up to decisions that this chain terminates in.

Which leaves the practical question. If shape sets the ceiling, and the ceiling is invisible until somebody hits it, how do you find yours deliberately before it costs you a decision?

Finding your own ceiling [summarize]

This runs on a whiteboard in one morning and it needs no tooling, no vendor, and no access to production. It's the finite check from earlier in this article, written out as a procedure: rather than searching the unbounded space of queries somebody might one day write, you compute one property of the system and it settles the question.

Bring whoever owns a budget. The data team can execute this and the data team can't source the inputs, because the inputs are decisions somebody is currently making badly and only that somebody knows which ones those are.

Step one: write five questions as decisions

Most teams write their questions as metrics. "What's our customer acquisition cost by channel." A metric is something you display. A decision is something you do differently depending on the answer, and the difference matters here because only decisions have a determining event behind them.

The format that forces it: if the answer is X I will do A, and if the answer is Y I will do B. Write both branches. A question where both branches produce the same action is decoration and it comes off the list immediately, which usually removes about half of what people write down first.

Real ones from real rooms sound like this, and note how little they sound like analytics:

• Do I renew this channel's contract when it comes up? Branch A: renew and increase. Branch B: cut it and move the budget to the next-best channel.
• Do I hire a second onboarding person? Branch A: hire, because onboarding load is what's driving early churn. Branch B: don't, because churn is coming from a product gap and a second hire just absorbs the complaints faster.
• Do I keep the discount program? Branch A: keep it, because it converts people who otherwise wouldn't buy. Branch B: kill it, because it's discounting people who were going to buy anyway.
• Which hours do I staff? Branch A: current schedule holds. Branch B: cut the two worst hours per location and redeploy the labor.

Five is the working number. Teams that write twenty have written a wish list, and a wish list doesn't force the prioritization that makes the rest of this exercise land.

Step two: name the event that would have to have been observed

For each question, write one sentence describing what happened in the physical world that, had somebody written it down at the time, would settle the question. Two rules on that sentence, and both are strict. It names a real-world occurrence with participants and a moment. It mentions no table, no tool, no report, and no vendor. If you can't write the sentence without naming a system, you haven't found the event yet, and you're describing your software instead of your business.

Work the discount question through and something useful happens. The event is: a specific customer encountered a specific discount at a specific moment and then bought or didn't, while a comparable customer who didn't encounter it also bought or didn't. That's a demanding event. It requires knowing who saw the offer, which is often unrecorded, and it requires a comparison group, which requires that somebody deliberately withheld the offer from part of the audience. Discovering that the event is demanding is the finding. Most discount programs are evaluated by comparing revenue before and after, which measures the discount plus the season plus the product mix plus whatever else moved, and reports the total as the discount's effect.

Step two takes fifteen minutes of arguing per question and produces no artifact anyone can demo, which is exactly why it gets skipped and exactly where the value is.

Step three: trace the event down and watch where it breaks

Now run each event through four checks in order. Stop at the first failure, because the first failure is the ceiling and everything below it is moot.

1. Is the event captured at all? Anywhere. Application logs, raw landing zones, a vendor's system you can still export from, a queue nobody drains. Check before concluding anything, because this check fails less often than teams assume.
2. Is it captured at the grain the question needs? Per customer or per segment. Per hour or per day. Per order line or per order. Compare the grain the question requires against the finest grain anything carries.
3. Is it captured with the keys that connect it to the other things in the question? An event recorded in isolation answers nothing if the question relates it to something else, which almost every real question does.
4. Does the record survive? Retention windows, overwrites, and in-place updates all destroy correctly captured events after the fact, which is the cruelest failure of the four because the system did its job and then a maintenance policy undid it.

The check number that fails tells you the repair. Failing check one means nothing exists and the only move is instrumentation, effective from the day it ships. Failing check two is death one, too coarse, and the repair is capturing finer while somebody digs through raw layers for a surviving copy. Failing check three is death two, the missing key, and the repair is writing the key at capture while you measure and publish the error rate of whatever matching you're doing in the meantime. Failing check four is Part III's entire subject, and it's the one where "we have backups" gets offered as a defense and fails.

Step four: the output is a ceiling register

What comes out is a table, and the table is the deliverable. Here's the shape, with the attribution question from the opening of this article filled in alongside two others.

That artifact does three jobs at once, which is why it's worth the morning. It converts an argument about tooling into a list of specific missing facts, and specific missing facts are cheap to argue about and expensive to hand-wave. It lets each lift be priced separately, so the cheap ones ship immediately rather than waiting behind a platform decision. And it replaces "the data team is looking into it" with a named missing fact, an owner, and a price.

The clock starts the day you fix it

One property of shape repairs governs how you should sequence them, and it's the reason this belongs on this quarter's list rather than next year's.

Fixing shape answers nothing retroactively. It starts the accumulation of the history that will answer the question later.

A business that carries the session identifier through checkout starting this month can answer the campaign question a few cycles from now, on real recorded joins, with no matching error at all. A business that waits for the platform migration first answers it that much later, and every cycle in between is a cycle of decisions made on a plausible number. The waiting cost never appears on a budget line, because unrecorded history generates no invoice, which is precisely why it loses every prioritization meeting it enters.

This also settles the most common objection to doing any of it now, which is that the team would rather do it properly during the big rebuild. The migration doesn't need the column. Adding a field at capture is independent of every platform decision downstream, it survives the migration by definition because migrations preserve the logical model, and it starts the clock today. Ship the column now and argue about the platform later.

All of which comes with a bill, and the bill is the part worth arguing about. Shape work is expensive, and a substantial share of the businesses reading this should do none of it.

What this costs and who should ignore it [summarize]

The real bill

Instrumentation competes with feature work that has a visible customer attached, and it loses that competition most quarters for reasons that are locally rational every single time. That's the obvious cost. Three less obvious ones do more damage.

The first is the dual-write period. While the new shape is being adopted, two systems are both running and both look authoritative, and every number in the business has two defensible values. That's the worst state a data system can occupy, worse than the old ceiling, and it lasts exactly as long as the migration does. Teams routinely underestimate this and then discover they've spent a quarter with a company that can't agree on its own revenue.

The second is political and it surprises people. Fixing grain changes historical numbers. Recompute last year at a finer resolution with the double-counting removed and last year's figures move, which means somebody has to walk into a room and explain that the number everyone has been quoting was wrong. A better number is an argument, and arguments lose to incumbency more often than they win. Budget for that conversation the way you'd budget for the engineering.

Being more correct buys you very little credit in the room where last year's number just moved. The person who reported the old number has an interest in the old number.

The third is that shape is a maintained property rather than a project. A key captured at checkout stops being captured the first time somebody stands up a second checkout path and doesn't know the field is load-bearing, which is the exact mechanism that installed the original ceiling, running again on a system that had been fixed. Shape decays silently, and it decays through people doing reasonable work who were never told which columns are holding up the building. The defense is a written record of which fields exist to answer which question, kept where an engineer will actually encounter it, plus a check that fails loudly when a required key arrives null.

A precise model is harder to onboard into than a loose one. Every entity you add is something a new engineer has to understand before they can safely change anything, and that tax gets paid on every hire, forever.

Who should stay on a flat table

Plenty of businesses should read this article, run the diagnostic once, find no breaks, and go do something else. That outcome is a success and it's more common than the consulting industry's output would suggest. The profile:

• One product, one channel, one path to an order. With a single acquisition channel, attribution is arithmetic and every question about it is answerable from the invoice.
• An owner who holds the whole business in their head. Below a certain size the fastest retrieval system in the company is a person, and it beats anything you'd build.
• Decisions on a monthly cadence where being roughly wrong doesn't switch the branch. Precision is worth paying for when it changes what you do. When both branches survive a wide error bar, precision is a hobby.
• Five questions that all pass all four checks. If the trace runs clean end to end on every question that matters, your ceiling is above your head. There's nothing here to buy.

Stated directly: if you've never had a decision blocked by a number nobody could get you, I'm not your guy and this isn't your problem. Bookmark it for the version of your business that has three channels and a second product line, and go spend the money on something with a customer at the end of it.

There's a failure mode on the other side that's worth naming with the same bluntness, because I've watched it burn more money than any missing key. Shape work done ahead of the question is a hobby with a budget line. Building a metagraph for a single-product, single-channel business produces a beautiful model of a world nobody is asking questions about, and the team that built it will defend it for years because it's genuinely good work. The question comes first, and a shape with no question behind it is decoration at enterprise prices. Everything in this article runs backward from a decision somebody is currently making badly, and with no such decision the whole procedure returns empty, correctly.

Three conditions that flip it

Three signals mean the arithmetic has changed and the work now pays. Any one of them is enough.

A question comes back repeatedly and blocks a real decision each time. Once is a curiosity. The third time the same question returns with a different vendor's methodology attached, you're paying rent on an unanswerable question and the instrumentation would have been cheaper the first time.

Somebody reconciles two systems by hand on a recurring schedule. This one is usually visible from the org chart if you know what to look for, and it survives reorganizations because the reconciliation is genuinely load-bearing and everyone knows it. The Intelligence Engineering essay covers the pattern at length as heroic failure, where a competent person is quietly compensating for a system that failed structurally and the heroism is the diagnosis.

That person is functioning as a foreign key. The job belongs to the schema, and a human doing it by hand is the clearest structural signal you will get.

You're about to commit serious money to a decision whose determining event isn't captured. This is the cheapest moment that will ever exist to instrument, because the spend justifies the engineering on its own and the history starts accumulating before the money goes out rather than after. A channel test instrumented the week before it launches produces an answer. The same test instrumented the week after produces an argument.

The instrumentation you skip this quarter is the answer you can't buy in two years, from anyone, at any price. That's the whole trade, and it never appears on a budget line.

That's the ceiling, where it comes from, and how to find yours. The shape you give data decides which questions you can ever ask of it, and the decision gets made early by somebody solving a different problem correctly, which is why it's invisible and why it's expensive. Part II walks the five shapes data actually takes, one at a time, and names what each one is structurally bad at, because choosing a shape is choosing which question class you're willing to lose.