Topic
Metadata
4 entries filed under this topic
Snowflake external lineage keeps addressing, not meaning
Snowflake ingests OpenLineage now. It keeps where the data came from and discards why it was processed — which decides what you can build on it.
The catalog is the API — a governed mesh over MCP
The Fabric + Mesh reference ended with a promise: an MCP-exposed catalog tool that answers which sensitivity level applies to a column holding an EU IBAN, from the live LF-tag policy rather than a stale wiki. This is that design. Tools versus Resources as the decision the whole thing hinges on, output schemas derived from the descriptor model already published, why a read-only envelope has to be enforced rather than annotated, and why authorization — not the tool list — is the governance plane. Verified against MCP revision 2026-07-28.
The pipeline half, part 2 — OpenLineage, Great Expectations, and the evidence a pipeline emits
Part 1 covered what a pipeline reads. This part covers what it emits, and why that is the whole governance story: OpenLineage as the vendor-neutral lineage backbone, the AWS-native adoption playbook via DataZone's PostLineageEvent API, assertion results as a data-quality facet a grant can be suspended on, and the provenance facet that answers the auditor's 'which code produced this number'. Plus why the scheduler stays central while compute decentralises, and the one hop this architecture cannot see. Real payloads and real emitter config throughout.
The pipeline half, part 1 — a pipeline is a descriptor, not a program
Metadata-first is usually read as an emission story — lineage and quality events coming out of a pipeline. That is half of it. This post walks the other half: the schemas, descriptors, contracts, expectation suites, and policy metadata a pipeline reads on the way in, and the one generic executor that interprets them instead of a DAG per pipeline. Real descriptor YAML, a complete Python executor, the six rules for defining a descriptor schema, and honest limits on where the abstraction stops working.