Behaviour-first governance in practice — generating artefacts from platform events

Companion to the Data Governance pillar. The pillar makes the argument for behaviour-first governance at the level of why the artefacts should be generated, not typed. This post picks up where it left off, at how to actually generate them on the AWS-native stack from Fabric + Mesh on AWS.

What the pillar left on the table

The pillar's central claim, worth restating:

Governance is what the platform enforces, what the domains do, and what the evidence shows — in that order. The artefacts (policies, RACIs, RoPAs) are byproducts of that behaviour, not deliverables in themselves.

Believing it is one thing. Building the pipes that make it true is another. This post walks four mechanisms — the ones that turn "we should generate the RoPA from lineage" into a real Terraform module and a real SQL query — on the AWS-native stack the Fabric + Mesh on AWS reference already establishes: LakeFormation + DataZone + Glue, plus OpenLineage as the vendor-neutral lineage backbone.

The four mechanisms:

1. RoPA from lineage — DSGVO Art. 30 records projected from OpenLineage events + LF-tag ontology + DataZone subscription log.
2. RACI from catalog ownership — the RACI matrix derived from DataZone project ownership + LF-tag metadata + subscription approvals.
3. Audit report from subscription log — the "who read what in the last 90 days" question answered from CloudTrail + DataZone events, in the format supervisors accept.
4. Change-authorisation workflow — GitOps for policy-as-code where the signed commit is the audit-evidenced change record, with works-council notifications auto-generated on RBAC-scope changes.

None of these is Cordata-specific invention. The primitives exist. What most programmes miss is the wiring — the specific projections, the field mappings, the where-to-put-the-human-review question. That is what this post covers.

§ 1 — Where the events come from

Before the four mechanisms, the event sources they all read from. All four projections draw on the same three, and getting them landed is a prerequisite for any of the projections to be honest:

• OpenLineage events — every job (dbt model run, Glue ETL job, Airflow task, EMR Spark job) emits a RunEvent with input datasets, output datasets, job metadata, and run status. Landed in an OpenLineage-compatible backend. Marquez is the reference OSS one, and Amazon DataZone accepts OpenLineage events directly via its PostLineageEvent API (announced 2024, matured since — check the current AWS docs for GA status and the specific facet coverage). On the AWS-native stack this means the OpenLineage emitters — the Spark plugin for Glue jobs, openlineage-airflow for MWAA, dbt's built-in OpenLineage adapter — can post straight to DataZone with a thin adapter to translate their payloads into DataZone's flavour. The projections in § 2-4 don't care which backend actually holds the events; the event sink is abstracted from the emitters. We treat this pipeline as the what data moved through what job source of truth.
• DataZone subscription log — every asset publish, every subscription request, every approval and revocation is a DataZone event. Available via the DataZone API and mirrored to CloudTrail. This is the who is allowed to see what data product source of truth.
• CloudTrail data events — for the tables under LakeFormation management, CloudTrail data events capture the per-query access log (GetTable, GetPartitions, Athena StartQueryExecution, Glue GetPartitions, etc.). Enable data events explicitly — they are not on by default and are what turns the audit answer from guess to proof.

# Data-plane CloudTrail on the governance account, capturing LakeFormation
# access events across all producer accounts (cross-account CloudTrail).
resource "aws_cloudtrail" "governance_data_events" {
  name                          = "cordata-gov-data-events"
  s3_bucket_name                = aws_s3_bucket.audit.id
  is_organization_trail         = true
  is_multi_region_trail         = true
  include_global_service_events = true
 
  event_selector {
    read_write_type           = "All"
    include_management_events = true
 
    data_resource {
      type = "AWS::S3::Object"
      # Registered lake buckets — the ones LakeFormation manages
      values = ["arn:aws:s3:::cordata-lake-*/*"]
    }
 
    data_resource {
      type   = "AWS::Glue::Table"
      values = ["arn:aws:glue:*:*:table/*"]
    }
  }
}

With those three sources landed, everything downstream is a projection — a query pattern that turns raw events into the governance artefact.

§ 2 — RoPA from lineage

Record of Processing Activities (RoPA, per DSGVO Art. 30) is the artefact most governance programmes lie about. Typed once, maintained by hand, wrong within a quarter. Generating it from the platform's own event stream removes the lying.

Quick primer for readers not steeped in Art. 30. The RoPA is a register every controller and processor of personal data has to maintain — Art. 30(1) for controllers, Art. 30(2) for processors. It documents, per processing activity: the purpose of the processing, the categories of data and data subjects involved, the recipient categories, any cross-border transfers, retention periods, and the technical / organisational security measures in place. A supervisory authority (BfDI in Germany, the cantonal DPOs in Switzerland, the DSB in Austria) can demand it on short notice. A wrong RoPA is a documented breach of Art. 30; a missing one is an aggravating factor at any subsequent audit under Art. 83 (fines up to €10M or 2% of global turnover, whichever is higher). That is why "we type it once and maintain it by hand" is a shape governance programmes lie about — the maintenance cost is real, the freshness gap grows quietly, and the risk when it comes is asymmetric.

The DSGVO Art. 30 field set that a controller-side record has to carry:

Before the SQL runs, the events have to live somewhere queryable together. The four sources — OpenLineage events, catalog metadata (Glue tables + LF-tag bindings), DataZone subscriptions, and CloudTrail data events — are not in the same store by default. Two paths get them there:

• Materialise centrally — a nightly ETL ships each source into a governance-account warehouse: Redshift Serverless, Aurora PostgreSQL, or DuckDB-over-S3 accessed via Athena. Simple operationally (one place to secure, audit, and back up), stale by the ETL cadence.
• Federate with Trino — Trino connectors query the sources in place, without moving data. A single query joins OpenLineage's backend, the Glue catalog, the DataZone subscription log, and the CloudTrail export directly. Fresher (current-to-the-minute), more moving parts to maintain (connector configs, credentials per source, query-time cost attribution).

Cordata's default recommendation is materialise centrally for the first year of a governance programme — one warehouse, one team, one grant surface, one place a supervisor can look. Trino federation is worth the complexity when the RoPA needs to be current-to-the-minute (rare) or when source-event volume makes ETL impractical.

The event-collection plumbing itself — how the OpenLineage sink lands, how CloudTrail exports to Athena, the Glue crawler schedule — is out of scope for this post. That is concrete territory for the pipeline-half companion covered in the Governance pillar's follow-ups.

With the events landed in one place, the projection is a single dbt model that reads OpenLineage plus the LF-tag ontology plus the DataZone subscription log:

-- models/governance/ropa_current.sql
-- Materialised as a view, refreshed every 6 hours by an EventBridge
-- schedule. Human review + signature happens on the delta since the
-- last signed snapshot (see the HITL step at the end of this section).
 
WITH lineage_jobs AS (
  SELECT
    ol.job_namespace,
    ol.job_name,
    ol.job_facets->'processing'->>'purpose' AS purpose,
    ol.job_facets->'processing'->>'legal_basis' AS legal_basis,
    ol.inputs,
    ol.outputs,
    ol.run_id,
    ol.event_time
  FROM openlineage_events ol
  WHERE ol.event_type = 'COMPLETE'
    AND ol.event_time > NOW() - INTERVAL '30 days'
),
input_facets AS (
  SELECT
    j.run_id,
    lft.sensitivity,
    lft.residency,
    lft.subject_type,
    d.name AS dataset_name,
    d.schema_fields AS fields
  FROM lineage_jobs j
  CROSS JOIN LATERAL jsonb_array_elements(j.inputs) AS input
  JOIN datasets d ON d.namespace = input->>'namespace'
                 AND d.name      = input->>'name'
  JOIN lf_tag_bindings lft ON lft.dataset_name = d.name
),
output_facets AS (
  SELECT
    j.run_id,
    d.name AS output_name,
    lft.residency AS output_residency
  FROM lineage_jobs j
  CROSS JOIN LATERAL jsonb_array_elements(j.outputs) AS output
  JOIN datasets d ON d.namespace = output->>'namespace'
                 AND d.name      = output->>'name'
  JOIN lf_tag_bindings lft ON lft.dataset_name = d.name
),
recipients AS (
  -- Downstream consumers via DataZone subscription log
  SELECT
    of.output_name,
    array_agg(DISTINCT dz.subscriber_project) AS consuming_domains
  FROM output_facets of
  JOIN datazone_subscriptions dz ON dz.asset_name = of.output_name
                                 AND dz.status = 'APPROVED'
  GROUP BY of.output_name
)
SELECT
  j.purpose                                    AS processing_purpose,
  j.legal_basis                                AS legal_basis,
  array_agg(DISTINCT i.subject_type)           AS data_subject_categories,
  array_agg(DISTINCT i.sensitivity)            AS data_categories,
  array_agg(DISTINCT i.residency)              AS source_residencies,
  array_agg(DISTINCT o.output_residency)       AS output_residencies,
  (SELECT array_agg(consuming_domains)
     FROM recipients r
     JOIN output_facets o2 ON r.output_name = o2.output_name
     WHERE o2.run_id = j.run_id)               AS recipient_categories,
  j.job_namespace || '.' || j.job_name         AS processing_activity_id,
  MAX(j.event_time)                            AS last_seen
FROM lineage_jobs j
JOIN input_facets  i ON i.run_id = j.run_id
JOIN output_facets o ON o.run_id = j.run_id
GROUP BY j.job_namespace, j.job_name, j.purpose, j.legal_basis, j.run_id;

Sample output — three rows from ropa_current:

Cross-border transfer detection is worth calling out. When source_residencies and output_residencies differ — e.g. {eu} in, {eu, us} out — that row is a candidate cross-border transfer under DSGVO Art. 44-49 and needs a documented safeguard (SCCs, adequacy decision, or explicit consent). The third row above is exactly that pattern ({eu} → {eu, ch}). The projection surfaces it; the governance council decides whether the transfer is authorised.

Where human responsibility sits. The projection produces the state of the RoPA at any moment; DSGVO Art. 30 requires a signed record with a named controller-side authority. That signature is the Data Protection Officer (or the designated data-protection lead named in the governance RACI) — nobody else in the org is authorised to sign, and that identity is the accountability chain DSGVO Art. 30 requires. The HITL cadence: weekly on immaterial changes, next business day on material ones (data category added, recipient added, cross-border transfer flag flipped), so the RoPA is never stale by more than seven days.

The full deployable framework — EventBridge schedule, ropa_snapshots warehouse table, diff-and-post Lambda, GPG-signed audit repo, freshness triggers, real freshness metrics after operation — is territory for the follow-up: How we built the RoPA-generation framework (planned). This post's job is the pattern; that post's job is the deployable proof.

The shape is what a DSGVO auditor accepts: not a wiki page maintained by hand, but a generated record with a named signer, a signed provenance chain, and a review cadence a supervisor can inspect.

§ 3 — RACI from catalog ownership

The RACI matrix — Responsible, Accountable, Consulted, Informed — is a governance artefact that dates within a quarter of being written. The reason is banal: the org changes, the paper doesn't. The behaviour-first move: derive the RACI from the catalog's own ownership metadata, which the domains keep current because their own subscription workflow depends on it.

The mapping on the AWS-native stack:

A single query surfaces the full matrix for any data product:

-- models/governance/raci_derived.sql
-- Executes against the catalog + subscription log. Runs on-demand
-- (governance dashboard) or nightly (reconciliation against paper-RACI).
 
SELECT
  ap.asset_name,
  ap.data_product,
 
  -- Accountable: the DataZone project owner
  ap.owning_project_owner_email AS accountable,
 
  -- Responsible: domain members with write-grants
  (SELECT array_agg(DISTINCT lfp.principal_arn)
     FROM lf_permissions lfp
     WHERE lfp.resource_expression @> ap.lf_tag_expression
       AND lfp.permissions && ARRAY['SELECT', 'INSERT', 'UPDATE']
       AND lfp.principal_arn LIKE '%' || ap.owning_domain || '%'
  ) AS responsible,
 
  -- Consulted: approved downstream subscribers
  (SELECT array_agg(DISTINCT dz.subscriber_project)
     FROM datazone_subscriptions dz
     WHERE dz.asset_name = ap.asset_name
       AND dz.status = 'APPROVED'
  ) AS consulted,
 
  -- Informed: governance council + audit subscribers (static config)
  ARRAY['governance-council@cordata', 'audit-log@cordata'] AS informed
 
FROM asset_publications ap;

Sample output — two rows from raci_derived:

The interesting output is not the projection itself — it is the reconciliation. Every week, a job compares the derived RACI against the paper-RACI (typically a spreadsheet Legal maintains) and surfaces divergences:

-- models/governance/raci_divergence.sql
SELECT
  derived.asset_name,
  derived.accountable AS derived_accountable,
  paper.accountable   AS paper_accountable,
  CASE
    WHEN derived.accountable != paper.accountable THEN 'accountable_mismatch'
    WHEN derived.responsible != paper.responsible THEN 'responsible_drift'
    WHEN cardinality(derived.consulted) != cardinality(paper.consulted) THEN 'subscriber_drift'
    ELSE 'aligned'
  END AS divergence_type
FROM raci_derived derived
LEFT JOIN raci_paper paper ON paper.asset_name = derived.asset_name
WHERE derived.accountable != paper.accountable
   OR derived.responsible != paper.responsible
   OR cardinality(derived.consulted) != cardinality(paper.consulted);

Sample output — a governance-council review would land on rows like these:

Divergence-as-signal. When the derived RACI diverges from the paper-RACI, that is a data-quality event in its own right. Two possibilities:

• The paper is stale — the derived version is truth, and the paper needs to be updated (or, better, retired in favour of the derived view).
• The catalog metadata is wrong — a domain has not updated its ownership; the platform enforces stale grants against stale metadata. Either way, the divergence has a specific owner and a specific fix.

The governance council reviews divergences monthly. The paper-RACI stops being the source of truth over time — teams stop maintaining it once they trust the derived view. That is not the abandonment of governance; it is the maturity of it.

§ 4 — Audit report from subscription log

The audit question a DACH supervisor asks — BaFin, FINMA, or an internal audit — is almost always the same shape:

"Show me who accessed the customer table in the last 90 days, with the query text, the requesting role, and the policy that granted the access."

Getting to that answer from CloudTrail + DataZone events takes real engineering but no exotic tricks. The catch is that the events are split across accounts (per-domain producer accounts for the storage-layer events, the governance account for the grant-layer events), so cross-account event stitching is the actual work.

The event stitching pattern:

Diagram read-out. A consumer's Athena query in one account starts a session (A1). The session assumes a cross-account role into the producer account (A2). LakeFormation issues temporary credentials for the Glue table (P1), which evaluates the LF-tag policy in the governance account (G1) — that policy is a materialised grant from a DataZone subscription (G2). The S3 read (P2) happens under the temporary credentials. To answer "who accessed the customer table", the audit query has to walk this chain backwards from S3 access to the human identity that started the session.

The query that answers the supervisor's question, running against a CloudTrail data lake (e.g. Athena over the CloudTrail S3 export):

-- Last-90-day audit query for the customer table
WITH s3_reads AS (
  SELECT
    event_time,
    user_identity.session_context.session_issuer.arn AS assumed_role,
    request_parameters.bucket_name                   AS bucket,
    request_parameters.key                           AS object_key,
    recipient_account_id                             AS producer_account
  FROM cloudtrail_data_events
  WHERE event_source     = 's3.amazonaws.com'
    AND event_name       = 'GetObject'
    AND request_parameters.bucket_name = 'cordata-lake-customer'
    AND event_time > NOW() - INTERVAL '90' DAY
),
role_sessions AS (
  SELECT
    event_time                                       AS assume_time,
    user_identity.arn                                AS human_identity,
    request_parameters.role_arn                      AS assumed_role,
    response_elements.assumed_role_user.arn          AS session_arn
  FROM cloudtrail_events
  WHERE event_source = 'sts.amazonaws.com'
    AND event_name   = 'AssumeRole'
    AND event_time > NOW() - INTERVAL '90' DAY
),
grant_context AS (
  SELECT
    dz.approval_time,
    dz.subscriber_project,
    dz.asset_name,
    dz.approved_by,
    dz.lf_tag_expression                             AS granting_policy
  FROM datazone_subscription_events dz
  WHERE dz.status = 'APPROVED'
    AND dz.asset_name LIKE '%customer%'
)
SELECT
  r.event_time,
  s.human_identity,
  r.assumed_role,
  r.object_key,
  g.granting_policy,
  g.approved_by                                       AS grant_authorised_by
FROM s3_reads r
JOIN role_sessions s ON s.session_arn = r.assumed_role
                     AND r.event_time > s.assume_time
LEFT JOIN grant_context g ON r.object_key LIKE '%' || g.asset_name || '%'
ORDER BY r.event_time DESC;

Sample output — two rows from the last 90 days, ARNs abbreviated for readability:

Every read of a cordata-lake-customer object is now stitched back to (a) the human identity behind the session, (b) the assumed role, (c) the LF-tag policy that allowed it, and (d) the DataZone subscription approval that materialised that policy — with the name of the domain owner who approved.

That is the format a BaFin auditor accepts. Not a spreadsheet reconstructed from memory.

§ 5 — Change-authorisation workflow

The pillar's § 5 second move — "a change-authorisation workflow that is itself audit-evidenced" — turns out to be the least mysterious of the four mechanisms once you accept the operating pattern: GitOps for policy-as-code, with signed commits as the change record.

This is not a governance-specific invention. It is the same policy-as-code access capability named as one of the five Fabric moves in the Data Engineering pillar, on the same IaC-first substrate the Fabric + Mesh on AWS reference treats as prerequisite in its § 2. Governance runs on the same rails as everything else the platform ships — that is the point of behaviour-first as an operating stance, not just as a phrase.

The shape:

• The LF-tag ontology, the grant expressions, the DataZone project configuration, and the subscription approval workflow config all live in a Terraform repo under terraform/governance/.
• Every proposed change is a pull request. The PR carries the output of terraform plan as an attached text artifact — the reviewer sees exactly what will change before approving.
• Merged PRs are GPG-signed commits (git commit -S) where the reviewer's key maps to their identity in the internal identity provider.
• A GitHub Actions workflow applies the plan on merge, and the CloudTrail PutResourcePolicy / CreateLFTag events that result carry the git SHA as a request-metadata tag.

# .github/workflows/apply-governance.yml
name: Apply governance changes
 
on:
  push:
    branches: [main]
    paths:
      - "terraform/governance/**"
 
jobs:
  apply:
    runs-on: ubuntu-latest
    environment:
      name: production
      # Environment requires 2 approvers from the governance CODEOWNERS
      # (defined in .github/CODEOWNERS below). No lone-wolf changes.
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - name: Verify signed commit
        run: |
          if [ "$(git log -1 --pretty=format:%G?)" != "G" ]; then
            echo "Commit is not GPG-verified — refusing to apply."
            exit 1
          fi
      - name: Apply with SHA tag
        env:
          COMMIT_SHA: ${{ github.sha }}
        run: |
          cd terraform/governance
          terraform init
          terraform apply -auto-approve \
            -var="change_reference=${COMMIT_SHA}"

# .github/CODEOWNERS
terraform/governance/lf-tags/**       @governance-council
terraform/governance/grants/**        @platform-team @governance-council
terraform/governance/datazone/**      @platform-team

CODEOWNERS enforces the federated-decision contract from pillar § 5: any change to the LF-tag ontology (the shared vocabulary) requires governance-council approval; grants require platform-team AND governance-council; DataZone project config is platform-team owned.

The change record IS the audit evidence. When a supervisor asks "when was the sensitivity=regulator_only tag added and who approved it?", the answer is a git log query, not a wiki archaeology dig:

git log --show-signature --patch \
  --follow terraform/governance/lf-tags/sensitivity.tf

Output includes the commit message, the diff, the GPG signature, the signer identity, and the timestamps. All immutable. All evidence.

Works-council notification integration

The pillar's § 8 works-council rhythm becomes a concrete part of this workflow. Any PR touching terraform/governance/lf-tags/** (RBAC-scope changes) auto-generates a works-council notification draft:

# .github/workflows/works-council-notification.yml
on:
  pull_request:
    paths:
      - "terraform/governance/lf-tags/**"
 
jobs:
  draft-notification:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Generate works-council notification draft
        env:
          PR_NUMBER: ${{ github.event.pull_request.number }}
          PR_TITLE:  ${{ github.event.pull_request.title }}
        run: |
          python .github/scripts/draft_notification.py \
            --pr "$PR_NUMBER" \
            --title "$PR_TITLE" \
            --diff "$(git diff origin/main -- terraform/governance/lf-tags/)" \
            --template docs/works-council/notification-template.de.md \
            --out /tmp/notification.md
      - name: Attach to PR
        uses: mshick/add-pr-comment@v2
        with:
          message-path: /tmp/notification.md

The generated draft is a filled-in copy of the works-council notification template (the template file is in German — .de.md — because the works council receives a German-language document) — auto-populated with the change scope (which tag values are added/removed/redefined), the affected data product surface, the estimated number of grants affected, and the platform team's rollback plan. The works council receives a real draft they can review; the platform team pays for the review cycle-time up front instead of eating a rollback three sprints later.

§ 6 — What this does not cover

The four mechanisms above are the artefact-generation side of behaviour-first governance. Three specific things they deliberately do not do:

• They do not decide policy. The projections generate the record of what policies exist and what they enforce. The governance council still has to reach the decisions themselves — federated computational governance means computational enforcement, not computational decision.
• They do not reconcile across jurisdictions. A multi-jurisdiction shop (DE + CH + AT with separate legal bases) needs a per-jurisdiction RoPA and a per-jurisdiction audit trail. The projection above is single-jurisdiction; extending it is a specific engagement scope, not a generic template.
• They do not migrate legacy artefacts. If a shop already has a manually-maintained RoPA and a paper RACI, the migration is a separate effort — reconciling the paper with the derived view, deciding which historical entries carry forward, and negotiating with Legal on the transition. The derived view is stable; the migration is where the political work lives.

Anyone selling generated-governance as a full-stop answer to a governance programme is missing all three.

Where this fits

The Data Governance pillar established the theory — governance as behaviour with evidence. The Fabric + Mesh on AWS reference established the technical foundation. This post is the wiring between them.

For a shop building this for real, the two-week discovery from the pillar's engagement model is where you would start: audit the current governance surface, run the existing paper-RACI against the derived one, count the divergences, decide which of the four mechanisms delivers the most compliance leverage first. RoPA generation is the usual answer — it saves the DPO half a week of manual maintenance per month, and the freshness improvement is the artefact a supervisor notices first.

If you have wired one of these four mechanisms yourself — or hit a specific projection problem the SQL above doesn't cover — I would genuinely like to hear where the seams showed. The door is open at cordata.tech/contact. Related reading: the Data Governance pillar for the operating-model argument this walkthrough concretises; the Fabric + Mesh on AWS reference for the LakeFormation + DataZone substrate the projections read from; and the forthcoming DSGVO Art. 17 across a mesh post covering the erasure mechanics this walkthrough does not.