Data Contracts — the schema is a shared interface, not a private detail

The release that broke Monday's number

A column is renamed in an ordinary Friday release. Every test the application team owns passes, because nothing in that repository knows the column is read anywhere else. On Monday the pipeline that fed the finance dashboard either fails loudly, or — worse — succeeds and publishes a number that is quietly wrong. Someone in finance notices the figure looks off; a data engineer spends the morning tracing it; the application team is genuinely surprised, because nothing they own told them anyone depended on that field.

Most data incidents that reach a dashboard start like this, and they come from two failures that feed each other.

The private schema. The application team — the producer in what follows, whether it publishes a table, an event stream or an API payload — treats its schema as internal state, changeable at will. That is a reasonable default: a schema is an implementation detail of a service, and nothing in that team's build says otherwise.

The absorbed change. The data team — the consumer, along with every pipeline, model and dashboard behind it — treats whatever arrives as a given and works around it downstream: a coercion here, a rename in the transformation layer there, schema evolution switched on, drift alerts routed to its own channel. That is also reasonable locally, because the dashboards stay green and nobody upstream has to be argued with.

The usual explanations — poor communication, missing documentation, the wrong ingestion tool — are real and are downstream of those two. Documentation does not fail a build.

Why this costs more than it used to

A column used to feed one report, read by people who would notice if it turned strange. It now feeds features, embeddings and AI use cases, and those fail quietly rather than loudly: a model does not throw an exception on a renamed column, it scores on a default and returns a plausible number. A model whose inputs changed meaning without anyone noticing is a model nobody can evidence — which is where this stops being a data engineering inconvenience and starts being a governance problem.

It also undermines the records built on top. The Data Governance pillar argues that governance artefacts should be generated from what systems actually do rather than maintained beside them. A record of processing generated from lineage describes what the pipeline read. If the application changed what a field means, that record is confidently, automatically wrong.

Regulation reaches this in specific sectors rather than everywhere, and the exact claim is short: no general law requires data contracts. What does exist is narrower, and it is named precisely further down.

Each failure feeds the other

The private schema and the absorbed change are not independent failures. Every change the consumer absorbs teaches the producer that changing a schema costs nothing, because the cost lands somewhere the producer never sees: another team's pipeline, in a repository whose build was already green. So the schema stays private, and the next change arrives with as little warning as the last one and has to be absorbed in turn.

The producer keeps changing the schema because its own build never fails, and the consumer keeps absorbing the change because nothing it owns can make that build fail.

The market sells the halves separately. Contract specification formats and registries address the producer's side of the paperwork; observability and drift detection address the consumer's side of the damage. Neither changes what happens at the moment a release ships, which is the only moment where the cost is still small.

What a data contract is, and what it is not

A data contract is an agreement between a producer and named consumers about the shape and meaning of data, and about how either may change — enforced where the change is made.

There is good work to build on here, and it should be credited precisely rather than waved at:

• The Open Data Contract Standard (v3.2.0, stewarded by Bitol, which graduated in the Linux Foundation's AI & Data Foundation in July 2026) defines what a contract document contains — schema, quality, service levels, roles, support — across fifteen sections. It deliberately says nothing about where the contract is enforced.
• The Data Contract Specification is deprecated in favour of that standard, and should not be cited as current. The Data Contract CLI, maintained by Entropy Data, runs on ODCS and ships test, ci and breaking commands meant for pull requests.
• Andrew Jones' Driving Data Quality with Data Contracts (Packt, 2023) makes the organisational case and argues that producers own the contract. The book also discusses continuous integration checks, at four points, including "Continuous Integration (CI) checks, giving both the generators and consumers confidence that their code is using the data models correctly as they develop their services".
• dbt model contracts enforce shape at build time for a modeller's own models, and detect breaking changes by comparing against a previous project state — the pattern that runs in CI. For unversioned models dbt warns rather than fails, which is worth knowing before relying on it.

Enforcement in the producing application's build is therefore not a new idea, and this pillar does not claim it. Adrian Kreuziger put it plainly in An Engineer's Guide to Data Contracts, written for Chad Sanderson's newsletter in October 2022: "Data contracts must be enforced at the producer level. If there's nothing enforcing a contract on the producer side, you don't have a contract."

Three things a contract is commonly mistaken for:

• Not a specification file. A document is the contract's text. Without a check where the change is made, it is a wish list.
• Not a schema registry. A registry records the versions a table has had. A contract also records who reads them — that fraud_raw.transactions at version 7 is read by a fraud pipeline and a finance one, so removing a column from it has two teams' names attached before anyone ships. That table and those two consumers are the example the rest of this pillar runs on — first as a descriptor, then as the output of the check itself.
• Not a service-level agreement. Freshness and availability promises matter, and they describe the output side — a different contract from the shape-and-meaning agreement this pillar is about.

Both sides, in one page

The producer side is four practices rather than a programme, and each one exists to make a particular failure impossible.

Every published shape carries a version, so that a consumer can say which shape it was written against instead of hoping the current one still matches. A breaking change produces a new version rather than an edit to the existing one — breaking meaning a column removed, a column retyped, or a column that may now arrive null where the old version promised it would not, which are the three ways a consumer's query stops meaning what it meant. Additive changes need no new version, because a consumer reading the old columns is unaffected by a column it never selected.

A deprecated version stays readable until a stated date. The date is what makes the deprecation real: without one, the old version either disappears on the day the producer stops thinking about it, or survives for years because nobody can say when it should end. With one, moving the pins becomes a scheduled piece of work for the consuming teams rather than an interruption.

The release runs a check against the consumers before it ships, and fails the build when one of them would break. This is the one practice of the four that sounds like a proposal, so it is worth saying that it is not: the Data Contract CLI ships a breaking command written for pull requests, dbt detects breaking changes to a contracted model by comparing against a previous project state, and both were cited above. What differs between them and what follows here is not the check but where it gets its list of consumers — and the section on what the runtime enforces shows ours running, on a table with two of them.

Where the application's own database is the source, the interface is an outbox table or a published event rather than the internal tables, so the producer keeps the freedom to restructure what sits behind it. The point of a contract is to have an interface, not to freeze the implementation.

The consumer side is two practices. Each consumer declares the version it reads, which is what puts it on the list the producer's check consults. And it stops on a mismatch rather than absorbing one — the failure is the whole value, because a pipeline that silently accommodates a changed column is how a wrong number reaches a dashboard. Drift detection stays in place behind both, as the safety net for sources that cannot be put under contract at all.

Both sides are technology-neutral. Outbox and change data capture are patterns here, not products.

The consumer list is declared, not maintained

Every contract has two parties, and the awkward one is the consumer. A contract document can describe a table's shape precisely and still be wrong about who reads it, because the list of consumers is typed by hand: it goes stale the first time a pipeline is added by someone who did not know the document existed. What decides whether a producer-side check is worth running is where its list of consumers comes from.

The alternative is to stop maintaining the list at all. Every pipeline here already declares the version of each table it reads, because the executor refuses to start otherwise — part 1 § 4's pin, moved under a name that says which side it binds.

The shape below differs from the descriptors part 1 § 2 prints, deliberately. Those are cordata.tech/v1, where one top-level contract: block held the freshness SLA and the LF-tags, and the pin on the input sat loose in source: as schema_version. This is v2, which gives each side a contract of its own — and the published v1 descriptors stay exactly as they are and still run, because the executor picks the model to parse against from apiVersion, so v2 lands beside v1 rather than replacing it.

apiVersion: cordata.tech/v2
 
source:
  database: fraud_raw
  table: transactions
  contract:                          # what this pipeline depends on upstream
    schema_version: 7
 
target:
  database: finance_curated
  table: merchant_settlement_daily
  contract:                          # what it promises downstream
    freshness_sla: PT6H

Two blocks of one real descriptor, trimmed to the contracts: the metadata, steps, expectations and processing around them are omitted, as are the storage details inside them — kind, location, the partition key, and the LF-tags the target contract also carries.

That pin is not documentation. It is what the pipeline runs on, and changing it changes the run. So the set of descriptors pinning fraud_raw.transactions is its consumer list, and nobody has to remember to update it: a pipeline joins the list the moment its descriptor does, and leaves when the descriptor does.

The naming is this pillar's argument in miniature. Version 1 had one contract: block at the top of the file, and everything in it described what the pipeline promised downstream; what the pipeline depended on upstream was a bare schema_version inside source:, with no word for it at all. Everyone specifies what they publish and nobody specifies what they consume, down to the vocabulary.

The Data Governance pillar's argument — that a governance artefact should be generated from what systems actually do rather than maintained beside them — applies here one step earlier in a pipeline's life. There the processing record is generated from the lineage events a pipeline emits once it has run; here the consumer list is generated from what its descriptor declares before it runs at all.

There are two limits, and the first is large. Declaration-first sees only the consumers that declare, and ad-hoc SQL, notebooks and dashboards read tables without pinning anything, so no descriptor will find them — often the readers a rename breaks first.

The fallback is to recover the undeclared readers from whatever logs the reads, and what that is depends on how the table is read. With a query engine in front of it there is a view to query — ACCOUNT_USAGE.ACCESS_HISTORY in Snowflake, INFORMATION_SCHEMA.JOBS in BigQuery, or Lake Formation's data-access events in CloudTrail wherever it brokers the credentials. With nothing in front of it, which is the case for an Iceberg table on S3 read by a Spark job holding its own role, the only record is at the bucket: CloudTrail data events or server access logs on the table's prefix, neither enabled by default, and both naming an IAM principal that still has to be traced back to a team. The operation is the same in either case, filtering for the table and subtracting the descriptors that already pin it, and what remains is a set of teams to contact rather than a second list to keep current, because those logs have retention windows and report who read the table inside one rather than who depends on it.

The second limit is the one Jones names. A version check compares shape, not meaning. A column that keeps its name and its type while its definition moves underneath — an amount that becomes tax-exclusive, a timestamp that changes timezone — passes everything above, and the pipeline keeps running against data that no longer means what it did. What this pillar offers is that meaning is declared too: the descriptor already carries purpose and legal_basis for the processing record, and units and definitions belong beside them. That is not built, which is worth saying plainly rather than sketching. And declared meaning is only as good as whoever declares it, which is where a contract stops being a technical mechanism and becomes two teams agreeing what a number is.

What each side owes

Diagram read-out. Each lane holds the steps that happen on one side, and the two edges crossing between them carry the design. A lane says where a step runs rather than who maintains it, which is why the platform's other obligation, the command itself, is named inside the producer's lane: the platform ships it, the application's build runs it. The check runs in the producing application's own pull request, in the producer's lane, and it arrives at the list of consumers by scanning rather than by lookup: the command takes the version the application is about to publish, plus one or more paths to the repositories holding the descriptors, parses every descriptor under those paths and keeps the ones whose source names the table. The list is the pins themselves, so no consumer is registered anywhere and a pipeline added tomorrow is on it the moment its descriptor is. Only three changes fail that build, which the edge states: a column removed, a column retyped, or a column that may now arrive null. A compatible change instead publishes a new table version in the catalog, recording who published it and from which release, and that version is what a pipeline run resolves its pin against later, so anything that got past the check still fails at the consumer with the column and the release named. The dotted edge running from the descriptors to the run, the same pin, at run time, is one declaration doing both jobs: a single pinned version, read once by the producer's build and once by the consumer's own executor.

One obligation per lane, and each is small enough to hold a team to:

• The producer owes a versioned interface, a release reference on every version, and the check running in its own build. The team that can change the data is the team whose build fails.
• The consumer owes a declaration — the version it reads, pinned in the descriptor — and a loud failure when reality and the pin disagree.
• The platform owes the command that turns declarations into a consumer list and a verdict, runnable in any repository's CI, and a catalog that records which release published each version.

The producer's obligation carries a prerequisite worth stating, because it is the part that takes setting up. The command belongs to the platform team, who ship it as a package and keep it working, and it runs inside the producing application's build like any other build step. It discovers nothing: it scans the paths it is given, so the application's build needs the domain repositories beside its own — already true in a mono-repo, and a read-only clone step ahead of the check where domains keep their own repositories.

That leaves one list maintained by hand, which is worth naming rather than letting the phrase nothing to register cover it: the set of repositories to scan. It is a much slower-moving list than a consumer registry, holding one entry per domain repository rather than one per pipeline and changing when a domain is created rather than when a pipeline is added, and it lives in the producing application's CI configuration where that team can see it. The failure mode is worth knowing too, because it is quiet: a repository nobody added is simply not scanned, and its pipelines are absent from the verdict rather than reported as unknown. Where the domains sit in one organisation, generating that list from the organisation's repositories is the way to keep it from going stale. That the domains sit in one organisation is the assumption underneath all of this, since the check reads descriptors the producing build can open and the run resolves against a catalog someone operates for both sides; the section on when this is the wrong answer says which half survives when data is published past that boundary.

Break the build, not the dashboard

Cordata's position is one sentence, and it is the sentence to take into the next argument about release speed:

A breaking schema change should fail the build of the application that made it, reach the consumer's run only when nothing in that build could have caught it, and reach a dashboard only when both of those have failed.

That position is shared with Kreuziger, Sanderson and Jones; what this pillar adds is where the check's list of consumers comes from, and running code for both halves.

Where a breaking change surfaces decides who pays for it. Caught in the producing build, it costs a red pipeline and a conversation with the team named in the output, before the change exists anywhere, and the people reading that failure are the people who can amend it. One step later, at the consumer's next run, the change has already shipped, so a pipeline stops and somebody spends a morning establishing why — late data rather than wrong data, which is the good version of a bad day. The dashboard is where it becomes expensive, because nothing failed there at all: the number is wrong, somebody doubts it, and what has to be repaired afterwards is confidence in the figure rather than a column.

It is not an argument for freezing schemas. A versioned change with a deprecation window ships as fast as any other release, and the check names exactly which teams to talk to. The friction moves to the moment the change is cheapest to discuss, which is before it exists in production.

The consumer's check stays in place even when the producer's build is green, and not as insurance against the first check failing. An added column cannot fail a producer's build, because no consumer can pin a version that does not exist yet, so a pipeline that needs the new shape still learns about it at its own next run; and readers that declare nothing are invisible to the scan by construction. The two checks answer different questions, which is why the design keeps both rather than treating the second as a fallback for the first.

What Cordata refuses to sell

Cordata refuses to sell (a) data contract programmes that the data team writes and the application teams never see, or (b) downstream drift handling sold as the fix for upstream schema changes, rather than as the safety net behind one. Both refusals come from experience.

A contract only one party signed is a wish list. The data team writes down, in good faith, the columns and types it expects from each source, publishes the specs in a registry, and wires a check into its own pipelines. None of that runs where the change is made. An application team renames a column in an ordinary release, its CI passes because nothing in its CI knows the column is consumed, and the first system to learn about the contract is the pipeline that just broke against it — the situation the contract was bought to prevent.

A pipeline that absorbs upstream changes teaches the upstream team that its changes cost nothing — which, from inside that team's repository, is true. Automatic schema evolution, type coercion and drift alerts routed to the data team's channel all keep the dashboards green, and each of them moves the cost of a breaking change from the team that made it to the team that did not. Over a year the source drifts further because nothing pushes back, and the downstream workarounds become the undocumented logic the numbers actually depend on.

What Cordata sells instead: contracts enforced in the producing application's release process, so a change that would break a registered consumer fails the application's build rather than the pipeline's next run, and schema versions that carry the release that published them, so when something does get through, the failure names the team that can fix it. That puts the application teams in the engagement from the first week rather than informing them at the end. We are willing to lose the deal when a client wants contracts without them in the room — a contract the producer never agreed to is enforced only against the party that cannot change the data.

Where regulation actually reaches

No general law requires data contracts. Four things come close enough to be worth naming, and each reaches a specific audience:

Two notes for anyone checking these. Austria has no national instrument of its own: the ECB guide's own annex lists what each member state has, and Austria appears with two generic risk-management provisions where Belgium, Spain and Portugal have circulars written for this purpose. And the same annex still credits Germany with a MaRisk module that the June 2026 amendment deleted — a reminder that a regulatory citation has a shelf life.

Where AI belongs in contract work

AI helps with the text around a contract and not with the verdict. It can draft a first contract from an existing schema and the descriptors that already read it, summarise for a pull request which consumers a proposed change would affect and what each would have to do about it, and propose a deprecation path for a field that has to go. Each of those produces a draft the producing team signs off in the pull request, which is the point: the work AI does here is the writing, and the accountability stays with the team that can change the data.

Where it does not belong

Compatibility is a comparison with one right answer rather than a judgement: the check asks whether a column the consumer reads is still there, still has the same type, and still promises not to arrive null. Putting a model in that path trades a reproducible verdict for a plausible one. No model decides whether a change is breaking, approves one, or rewrites consumer pipelines to absorb it — that last one is the second refusal above, automated.

This section and Where AI belongs in contract work are about AI as a tool inside contract work. AI systems as consumers of the data a contract protects are a different question, and the subject of the AI Readiness pillar.

What the runtime enforces, and what you can run yourself

Both halves are running code rather than an argument. pipeline-runtime is public, clones without a cloud account, and runs on two commands. Every output below is a transcript captured from a real run and committed in that repository's docs/evidence/, each naming the commit that produced it.

The consumer side catches a change that already shipped. Every descriptor pins the schema version it reads, and the executor resolves that pin against the catalog before a single row is read. When an application publishes a new version of a table, the pipeline that pinned the old one stops, and the failure names the column that changed and the release that changed it:

fraud_raw.transactions is at v8, descriptor pins v7. v8 was published by
card-ledger release v4.12.0. Diff: + merchant_category_code (VARCHAR,
nullable). Bump the pin to v8 to accept.

One line in the program's output, wrapped here to fit the column.

The release is in that message because every version in the catalog carries the producer and the release reference that published it. On AWS this is metadata a publishing job sets in a Glue table's Parameters, which we measured rather than assumed: UpdateTable writes the whole Parameters map rather than merging into it, so every key the call omits is dropped and a publishing job has to send producer and release on each schema change or the attribution silently disappears. All three ways a Glue table changes were measured, and two of them cost nothing: a crawler leaves those keys alone and keeps the version it replaced, and so does an ordinary update. The exception is Iceberg, which keeps the keys but by default tells Glue not to keep the version it replaced, so on an Iceberg table a pinned version can stop existing unless the engine is configured to archive.

The producer side catches the same change before it ships. The command takes two arguments — the version the application is about to publish, and the paths to scan for descriptors — and reports which pipelines would break, exiting non-zero:

$ python -m pipeline_runtime.consumers example/proposals/card-ledger-v5.0.0.yml example/domains
card-ledger release v5.0.0 proposes fraud_raw.transactions v8; the catalog is at v7
2 descriptors under example/domains read fraud_raw.transactions

BREAKS      merchant-settlement-daily (finance)  pins v7  owner [email protected]
            example/domains/finance/pipelines/merchant_settlement.yml
            + merchant_ref (VARCHAR, not null)
            - merchant_id (VARCHAR, not null)   <- breaks

…

2 of 2 consumers fail this check

The second verdict is elided at the …; the full block is in the transcript.

Nobody registered either consumer anywhere: they are the descriptors that pin that table, they sit in two different domains, and each arrives with the address of the team to talk to.

An added column does not fail that build, and the distinction is worth stating. A consumer cannot pin a version that does not exist yet, so failing the producer's build on every schema change would block every schema change. The check fails on a removed column, a retyped one, or a not-null column that becomes nullable, and reports the rest as pins to move, with their owners.

That rule is forward compatibility under another name — the property a schema registry tests when it asks whether data written under the new version can still be read by a consumer sitting on the old one — and it differs from a registry's version of the test in two ways. The comparison is against the version each consumer actually pinned rather than against the latest published schema, so a pipeline three versions behind is checked against what it really reads. And it is stricter about deletions, because a registry can allow an optional field to disappear where a reader falls back to its default, while a SQL query naming a dropped column fails whether or not that column was nullable.

A run that succeeds carries the same attribution outward, which is the case that matters most, because a failure is loud and a quiet pass is what a contract is bought to protect:

"descriptor_git_commit_signed": true,
…
"source_published_by": "card-ledger",
"source_published_release": "v4.11.0",
"source_schema_version": 7

Four fields of the facet; six more are omitted at the ….

The first line reaches the reviewed commit that authorised the pipeline; the two source_published_* fields reach the application release that gave its input that shape. A processing record generated from this lineage can therefore name the release behind the number it describes.

One limit stated here rather than left to be found: a pipeline's own output carries no producer yet, so the chain covers the boundary between an application and a pipeline, not between two pipelines. The Data Contract CLI has shipped a producer-side compatibility check for some time, and it checks a proposed change against a contract document; what differs here is where the list of consumers comes from.

When this is the wrong answer

Producer-enforced contracts are the wrong engagement for at least four shapes, and pretending otherwise spends a client's money and their teams' time on a mechanism that will not pay for itself.

One team owns both sides. The application and the pipeline live in one repository, under one team, released together. An integration test in that repository is the contract; a separate mechanism is overhead.

Sources you cannot put in CI. A SaaS system or a vendor database whose releases you do not control has no build of yours to fail. Consumer-side detection is then the primary defence rather than a workaround, and that is where the second refusal above stops: it refuses drift handling sold as the fix for upstream change, which assumes a producer-side check that could have existed instead, and here none could.

Readers outside the organisation. Both halves of the design assume one organisation: the check reads the descriptors that pin the table, and the run resolves against a catalog the producing application also writes to, so somebody operates both for both sides. Data published across that boundary — sold to customers, exchanged with a partner, or opened to the public — leaves the producer half with nothing to stand on, because those readers declare nothing the producing build can parse and some of them will never be known to it at all. The consumer half crosses intact, and it is worth saying which half that is: a team reading someone else's data can still pin the version it was written against and stop on a mismatch, which is the only protection available to anyone who does not control the release. What replaces the producer half is what public interfaces have always used, a published version with a deprecation window and a date in it, announced rather than derived, with the access log reaching whichever readers it reaches.

Schemas that churn by design. Exploratory and sandbox layers, where breaking change is the point. Contracts belong where data is consumed by something that cannot tolerate surprise.

How Cordata engages

Engagements start from one real incident rather than from a contract catalogue. Take a breaking change from the last quarter's incident history, trace it to the release that caused it, and wire the check for that one source end to end — with the application team in the room from the first week, as the refusal above requires.

The shape is deliberately narrow: declarations in the consuming descriptors, release references on the versions the producer publishes, the check running in the producer's CI, and one schema change shipped through the new path as proof. Then the next source, if it is worth it. Deliverables are organisational as much as technical, because a check nobody agreed to is a check somebody will switch off.

Handover-first, as in both published pillars: the measure of the engagement is whether the client's teams extend this to the next source without us.

If you have put a schema under contract and had the producing team actually agree to it — or watched one become a wish list in a registry nobody upstream opened — I would like to hear which part held and which part gave way. The door is open at cordata.tech/contact.

Companion + further reading: cordata-tech/pipeline-runtime is the runnable version of everything quoted above — it clones without a cloud account, and the transcripts live in its docs/evidence/. The pipeline half, part 1 introduces the descriptor and the schema pin this pillar builds on; part 2 covers the provenance the successful run carries outward. The AI Readiness pillar is about the consumer a contract ends up protecting most: a model reading the data downstream.