Topic
Openlineage
6 entries filed under this topic
Data Contracts — the schema is a shared interface, not a private detail
Most data incidents that reach a dashboard begin as an ordinary application release. Contracts enforced in the producing application's build, with the consumer list generated from what pipelines already declare rather than maintained by hand — and running code for both halves.
AI Readiness — a data governance outcome, not a separate programme
AI initiatives most often stall for one of two reasons, and each tends to produce the other: the model is chosen before anyone can say what data it may lawfully read, and the answer is then manufactured by a parallel AI governance programme whose register drifts from the processing record it was meant to extend. Readiness is a property of the data record, and the AI obligations are projections of it.
From claim to evidence, hands-on — Cordata's OSS projects
Our four open-source tools, one IBAN column: the catalog knows every table it was given, and the lineage finds the copy that was never registered. Every step runs from pip install, quoted as the released versions print it.
Snowflake external lineage keeps addressing, not meaning
Snowflake ingests OpenLineage now. It keeps where the data came from and discards why it was processed — which decides what you can build on it.
The pipeline half, part 2 — OpenLineage, Great Expectations, and the evidence a pipeline emits
Part 1 covered what a pipeline reads. This part covers what it emits, and why that is the whole governance story: OpenLineage as the vendor-neutral lineage backbone, the AWS-native adoption playbook via DataZone's PostLineageEvent API, assertion results as a data-quality facet a grant can be suspended on, and the provenance facet that answers the auditor's 'which code produced this number'. Plus why the scheduler stays central while compute decentralises, and the one hop this architecture cannot see. Real payloads and real emitter config throughout.
Behaviour-first governance in practice — generating artefacts from platform events
The Data Governance pillar established the concept: artefacts are byproducts of behaviour, not deliverables in themselves. This post walks the four mechanisms that make it real — RoPA from lineage, RACI from catalog ownership, audit report from subscription log, and a change-authorisation workflow where the commit is the evidence. Real Terraform, SQL, and YAML throughout.