Snowflake external lineage keeps addressing, not meaning

On 3 September 2026 Snowflake moved external lineage to general availability. Send it OpenLineage events over REST and it will place objects that live outside Snowflake — an S3 prefix, a Postgres table, whatever your pipeline actually read from — into the lineage graph beside your own tables. dbt and Airflow already emit OpenLineage, so for a lot of teams this is a configuration change rather than a project.

That is a genuinely good thing, and the pipeline half § 1 argued for exactly this outcome six weeks ago: pick the open standard, because proprietary lineage makes both the portability and the coverage of your graph contingent on a commercial relationship. A warehouse adopting OpenLineage as an ingestion API is that bet paying off.

So this post is not a complaint. It is the more useful question that comes after the good news: once the events are in, what does Snowflake actually keep?

I spent a morning finding out, against a free trial, and the answer is sharper than I expected.

Snowflake keeps the addressing — where the data came from, where it went, which columns, which run, when. It discards the meaning — why the processing happened, under what legal basis, from which commit, and even what the job was called.

That distinction decides whether you can build anything on top of it.

Two of the three behaviours below are in the documentation, and I confirmed them rather than discovered them — which is worth doing, because one is stated only as an example and the other only in outline. The third is documented nowhere, and it is the one that decides what you can build.

What was tested, and how you can repeat it

Everything below came from a 30-day Snowflake trial with no payment method attached. External lineage needs Enterprise Edition, which you choose at signup and cannot change afterwards, and the two privileges the feature requires — INGEST LINEAGE and DELETE LINEAGE — grant to a role on a trial account without complaint, after which the REST endpoint answers. Nothing here required a paid relationship with Snowflake, so nothing here has to be taken on trust.

One trap worth naming before it costs you a day: the documented list of trial limitations includes "external network access", which is Snowflake reaching outward to something else. That is a different feature from external lineage, which is an inbound POST to Snowflake's own API. The names collide and the wrong conclusion is easy to reach.

The probe kit is one script that signs a JWT and one that runs SQL through the REST API, both using nothing beyond the Python standard library and openssl. Authentication is key-pair only — username and password will not open this door.

The payload was a single COMPLETE event modelled on what our own pipeline runtime emits: an S3 prefix as input, a Snowflake table as output, and four facets attached. Two of those facets are the ones the pipeline half § 1 puts on every event:

"job": {
  "namespace": "cordata.fraud",
  "name": "transactions-scored-daily",
  "facets": {
    "processing": {
      "purpose": "fraud-detection",
      "legal_basis": "legitimate-interest"
    }
  }
}

Those two fields are not decoration. The governance companion § 2 generates a DSGVO Art. 30 record by projecting them out of the lineage stream. If they survive the trip, the warehouse can hold that evidence. If they do not, something else has to.

Confirmed: only successes are admitted

The documentation says the endpoint takes COMPLETE events and gives START and FAIL as examples of what it rejects. It is worth testing the whole set rather than the two that are named, because the answer turns out to be absolute:

Every rejection carries the same body:

{"code": "394902",
 "message": "Unsupported event type, only COMPLETE event type can be ingested"}

So the graph holds successful runs and nothing else. No failures, no in-flight runs, no aborts.

Read as a design decision about a lineage graph, this is defensible and probably right. Lineage records which datasets derive from which, and a run that failed produced no dataset to draw an edge to. Nobody wants a canvas cluttered with arrows to tables that were never written.

Read as an evidence store, it is fatal, and for a specific reason. Part 2 § 3 makes assertion results a data-quality facet on the run, and § 4 states the provenance rule this blog operates under: a run whose provenance points at an unsigned commit is itself a finding. Both of those are claims about runs that went wrong. A sink that returns 400 for FAIL cannot hold the one record an auditor asks for first.

The failed run is not an edge case in governance, it is the central one.

Half documented: which facets survive, and which do not

The event went in and returned 200. Here is what came back out.

The external S3 object appears as a first-class node beside Snowflake tables. SCORED_COPY on the right is native lineage from a CREATE TABLE … AS SELECT, included as a control.

The column-level facet survived intact:

tx_id and amount exist nowhere in the payload except the columnLineage facet. The mapping is stored rather than guessed — amount was deliberately mapped to FRAUD_SCORE, which is named differently on each side, and it came back correctly.

That is a real capability and worth saying plainly: column-level lineage across a system boundary, from a JSON document, is not nothing.

Then select the edge itself:

The entire retained record of an external lineage edge. The panel ends where the screenshot ends.

Type, Run on, Run ID, where it came from, where it went. The external node's own panel is shorter still — source, type, namespace, and that is the end of it. The type is real information rather than a placeholder: send the datasetType facet and the node reports FILE instead of the generic External Node, which is the second of the two facets Snowflake documents.

There is no purpose and no legal_basis, and no sourceCodeLocation either, which is how a lineage event points at the commit that produced the run. The job's own name, transactions-scored-daily, is absent — the graph knows a run happened and does not record what the run was.

The full accounting of what went in against what came back:

The rule turns out to be exact, and Snowflake states it plainly if you read for it: it keeps addressing, plus the two facets the documentation names. columnLineage and datasetType both work precisely as described. Everything else in the OpenLineage spec is discarded.

That table is worth grading rather than reading flat, because it mixes three very different claims.

processing.purpose and processing.legal_basis are ours. A custom facet, not part of the OpenLineage specification, and Snowflake never undertook to keep it. Its being dropped is unremarkable and I would not build a post on it.

sql, parent, ownership and sourceCodeLocation belong to OpenLineage itself. These are facets the specification defines, and they are what the integrations Snowflake recommends actually emit. That is a heavier claim.

job.name, job.namespace and producer are not facets at all. They are required properties of a RunEvent. The documentation's "the payload can contain additional properties, but Snowflake ignores them" does not cover them, because they are not additional — they are the event.

What dbt and Airflow lose in their own metadata

The obvious objection to everything above is that I wrote these events by hand, and that dbt or Airflow — the two integrations Snowflake names — might get a richer deal from their own pipeline. They do not, and the documentation says so first: "Configuring dbt to emit OpenLineage events isn't unique to Snowflake; the only thing specific to Snowflake is the endpoint and base URL of external lineage." The same sentence appears for Airflow. It is JSON over REST either way, and the endpoint has no way to tell my curl from anyone's emitter.

I tested it anyway, with producer set to the dbt integration's own URL and a matching User-Agent, carrying eleven standard facets. The panel came back identical.

Which means the loss is not confined to our governance facets. parent is how Airflow's integration records which DAG and which task produced a run, and it is discarded, so an Airflow user looking at an edge in Snowsight cannot tell which pipeline wrote it. The sql facet, which dbt emits as a matter of course, is discarded too — so the query that built the table is not there either.

The graph knows that a run happened without recording what ran.

The finding: you can look at the graph, and you cannot query it

This is the one I did not see coming, and I got it wrong twice before getting it right.

Snowflake exposes lineage to SQL through SNOWFLAKE.CORE.GET_LINEAGE. Point it at the table and walk downstream, and the native edge is there. Point it at the same table, at the same moment, and walk upstream toward the external object that the console is drawing on screen:

SELECT SOURCE_OBJECT_NAME, TARGET_OBJECT_NAME, DISTANCE
FROM TABLE(SNOWFLAKE.CORE.GET_LINEAGE(
  'LINEAGE_TEST.PUBLIC.TRANSACTIONS_SCORED', 'TABLE', 'UPSTREAM', 3));

0 rows

Downstream it returns one row, and upstream it returns nothing — while the screenshots above were on screen.

There is no argument that turns it on. The deployed signature is GET_LINEAGE(object_name, object_domain, direction, max_distance, object_version); there is no namespace parameter and no include-external flag. 'EXTERNAL' as a domain returns "Unknown domain." And SNOWFLAKE.ACCOUNT_USAGE contains no lineage view of any kind, so there is no second door.

The external half of the graph renders in the console and cannot be reached from SQL.

For an analyst tracing a table back to its source, that is fine. For a platform team that wants to do something with the graph, that is the whole problem. You cannot generate a report from a picture. You cannot diff it between environments, assert on it in CI, project it into an Art. 30 record, or feed it to a control plane.

Metadata that cannot be queried is documentation with better rendering — and documentation is exactly what a descriptor is supposed to stop being.

A 200 does not mean it was stored

One more thing, and it is the one that would worry me most in production.

I sent the same event five times with five spellings of the Snowflake namespace, to find out which form the endpoint wanted. All five returned 200 with an empty body, including one that was not a valid OpenLineage namespace by any reading, having no scheme at all. Only one of the five became the edge in the graph.

There is no acknowledgement, no identifier, no count of edges written, and no error. And because external edges cannot be read back from SQL, there is no way to check afterwards either. A producer emitting into this endpoint on a schedule has no signal available to it that distinguishes stored and linked from accepted and discarded.

If you wire this up, the verification step has to be a data engineer opening the console.

What I got wrong, twice

Two corrections, because a post about whether a system reports honestly what it did should be honest about how long its author took to find out.

I concluded the events were being dropped. GET_LINEAGE returned nothing, repeatedly, over twenty-five minutes. I ran a control — a CREATE TABLE … AS SELECT inside Snowflake, queried through the same function at the same moment — and the native edge appeared in about ten seconds. Read path fine, latency not the explanation, so the ingest must have failed. That reasoning was clean and the conclusion was wrong: the console showed the edge sitting there the whole time. The failure was in the read path after all, just not in the way a working control could reveal.

The documentation described a function that does not exist. The published guidance for querying external objects gives a namespace argument and an object_domain of 'EXTERNAL'. Neither is in the deployed function. SHOW FUNCTIONS LIKE 'GET_LINEAGE' IN SCHEMA SNOWFLAKE.CORE returned the real signature in one line and settled in seconds what the docs had cost me half an hour. On a six-week-old GA feature, introspect the account rather than trusting the manual.

What this means if you are wiring it up

Snowflake's external lineage is a lineage viewer, and a good one. Column-level fidelity across a system boundary, external objects as first-class nodes, no custom integration to write. If the job is an analyst opens a table and asks where this came from, it does that job well and costs a configuration change.

It is not an evidence store, and the distinction is not pedantic:

• It admits only successes, so it cannot hold the failed run.
• It keeps addressing and drops meaning, so it cannot answer why or under what basis.
• It cannot be queried for the external half, so nothing can be automated on top of it.
• It confirms nothing on write, so you cannot verify what it holds.

None of those is a defect in a lineage viewer. All four are disqualifying in an audit trail. The mistake — and it is an easy one, because the events are literally the same events — is treating "we emit OpenLineage to Snowflake" as having discharged the obligation. The wire format is shared, and the retention policy is not.

One audience is served well, another not at all

The Fabric + Mesh reference § 4 sorts these questions by who is asking, and that is the right way to read this release too.

Data engineers, analysts and domain engineers get a good deal. Their question is where did this table come from, and which column fed which. It gets answered in the console, at column granularity, across a system boundary, for the cost of pointing an existing dbt or Airflow integration at a different URL. For that audience this release is straightforwardly good news and the rest of this post does not apply to them.

The platform team gets half a deal. Anything they would want to automate — asserting in CI that no undeclared source feeds a published table, diffing the graph between staging and production, alerting when an edge disappears — needs the graph in SQL, and the external half is not there. They can look at it and cannot build on it.

The DPO, the governance council and the auditor get nothing they can use. Not because Snowflake did this badly, but because of what Art. 30 actually asks. The record of processing activities is an obligation of the controller — the organisation — not of a system. It has to cover every processing activity, including the ones that never touch the warehouse: the Lambda that scores a file in place, the SaaS tool a domain team bought, the export that leaves for a processor. A per-tool lineage graph cannot satisfy a controller-scoped obligation even when it is perfect, and this one additionally drops the purpose, the legal basis and every failed run.

So the second sink is not redundancy or belt-and-braces. It is the only place a cross-system question can be answered, and Art. 30 questions are all cross-system. The warehouse graph answers where did this table come from. The holistic record answers what does this organisation do with personal data, on what basis, and can you show me — and no single tool's console is ever going to be the place that lives.

Which sharpens something the pipeline half § 1 got half right. That post argued for OpenLineage on the grounds that a proprietary catalog makes portability and coverage contingent on a vendor. Snowflake adopting the standard delivers interoperability — the same emitter now feeds another sink, with no adapter to write, and that is real. It does not deliver portability, because portability is a property of where events land, how long they live and whether you can get them back out. None of those is decided by the format they travelled in.

So: emit once, land twice. Send the events to Snowflake, because for the engineers and analysts who open that console the graph is genuinely useful. Land the same events somewhere you control as well, because that is the copy the DPO's question gets answered from — the one that still has purpose and legal_basis on it, still contains the run that failed, and still holds the processing that happened nowhere near the warehouse.

The probes, the payloads and the raw responses are in cordata-tech/snowflake-external-lineage-probe, including the hypothesis that turned out to be wrong. Everything in it runs against a free trial.

If you have wired OpenLineage into Snowflake and found the retention different from this — or think a lineage graph was never supposed to carry the governance facets and I am asking the wrong thing of it — I would like to hear it. The door is open at cordata.tech/contact. Related reading: A pipeline is a descriptor, not a program and the pipeline half are the two posts this one tests, and Behaviour-first governance in practice walks the Art. 30 projection that these facets feed.