Fabric + Mesh on AWS — LakeFormation, DataZone, and one governance plane

Companion to the Data Engineering pillar. The pillar makes the argument for the Fabric + Mesh Hybrid at the level of what to build and why. This post picks up where it left off, at how to build it on AWS, in the syntax you would actually check into Git.

What the pillar left on the table

The pillar named a guiding architectural choice and then, deliberately, walked past it: the Fabric needs exactly one governance plane per data domain — and picking it is the decision the whole architecture hangs on. Pure-AWS shops pick LakeFormation + DataZone — AWS's fine-grained access-control layer for data-lake tables plus the data-product catalog above it. Multi-cloud shops with a strong Databricks footprint pick Unity Catalog — Databricks's cross-workspace governance layer. The trap that ends most Fabric programmes is running two planes in parallel over the same data and hoping "governance will federate later." It never does.

This post walks the AWS-native half of that decision. Cordata's team has shipped Terraform and Terragrunt across the AWS estate since 2021 — through DACH banking, insurance, telco and enterprise-software engagements — with LakeFormation, Glue, DMS, EMR and Redshift as the daily moving parts, and DataZone joining the toolkit as it matured. We have not shipped the exact Fabric + Mesh composition below end-to-end at a single client; each engagement has covered a subset. What follows is the assembly we would ship today, in the syntax we have been shipping for five years — the trade-offs are the ones we live with, not the ones we heard about at a conference.

§ 1 — The topology

The guiding principle for an AWS-native Fabric + Mesh reveals itself only after the second or third migration: domains own accounts, not folders in a shared bucket. Shared-bucket "mesh" is what a Fabric programme decays into when the platform team is under-resourced and IAM feels like too much work. It looks the same on day one; by year two, every audit reduces to "who has s3:GetObject on that prefix" and the answer takes a week.

But account ownership is not a free-for-all. Each domain account is created from a platform-owned blueprint — the account structure, network topology, IAM baseline, KMS key policy, logging targets, and Terraform module set all come from the platform team. The domain owns the content of the account (its tables, its pipelines, its SLAs, its subscription decisions); the platform owns the shape of it (how the account is bootstrapped, secured, monitored, and connected to shared services). Without this split, one of two failures follows: domains without infra expertise flounder in AWS console-clicking, or every domain builds its account so differently that the estate becomes ungovernable within a year — the same shared-bucket rot in a different guise.

The account-per-domain topology fixes that structurally. Producer accounts hold the storage and the Glue catalog (AWS's metadata registry for tables and schemas) for one domain. A central governance account holds the LakeFormation admin plane and the DataZone domain. Consumer accounts subscribe to data products through DataZone; access is provisioned by LakeFormation cross-account grants; audit is AWS CloudTrail in the governance account for the grant events, and CloudTrail in the producer account for the data-plane reads.

Diagram read-out. Three account tiers, bottom to top. Producer accounts (bottom) each hold one domain's storage (S3) and its Glue catalog — one account per business domain (fraud, policy, claims). Producers publish assets into the central governance account (middle) through DataZone's CreateAssetType and PublishAsset APIs, which register the data product with its schema, business-glossary terms, and owner. The governance account holds three things: the LakeFormation admin plane (which owns the cross-account grants), the DataZone domain and its projects (which owns the publish + subscribe workflow), and the LF-tag ontology — the small controlled vocabulary of tags like domain=fraud, sensitivity=high, residency=eu that policies are written against. Consumer accounts (top) subscribe to data products through DataZone; an approved subscription materialises as a LakeFormation cross-account permission and the consumer's IAM role can then query the underlying table via Athena, Redshift Spectrum, or EMR.

Three properties this topology buys you, none of which you get from shared-bucket mesh:

1. Blast radius is an account. A compromised role in the fraud account cannot touch the claims account's data, ever, regardless of how many misconfigured policies exist elsewhere.
2. Cost attribution is free. Producer-account cost is domain cost. No tag-based allocation gymnastics.
3. Regulatory boundaries line up with technical boundaries. BaFin, FINMA and DSGVO all reason about "who processes what data" — the account boundary is the answer in a language auditors already speak.

§ 2 — Terraform: the domain-account skeleton

Prerequisite before any of the below matters: use infrastructure-as-code (IaC), full stop. Console-clicked accounts are not reviewable, not reproducible, not disaster-recoverable, and do not survive personnel change. Everything else in this post — reproducible environments, auditability, DORA's asset inventory, the guardrails-not-gates promise from the pillar — is downstream of that one decision. The choice of which IaC tool matters much less than the discipline itself: Terraform, OpenTofu, Pulumi, AWS CDK, all fine; pick the one your team already reads.

What matters more than the tool is the shape it enforces: separation of concerns between infrastructure code (what a module provisions) and environment configuration (which values it receives, per environment, per domain). Terraform + Terragrunt is one way to enforce that split; Pulumi with a stack layer, Terraform CDK with per-env config, or plain Terraform with per-env tfvars would all achieve the same shape. The value is in the separation, not the choice of tool.

Environments are separated deliberately. Every domain account exists three times: dev, staging, prod. That parity is what makes DORA's critical-ICT-asset inventory maintainable — a change proven in staging is the same change applied in prod, with the same configuration file, days later. Without this, DORA's incident-window requirement (know what breaks when component X goes down, in hours) gets answered by memory and guesswork instead of a plan you already tested.

The Terragrunt layout we use for this on client engagements looks like the block below. _envcommon/domain-account/terragrunt.hcl holds the infrastructure module wiring; each live/prod/domains/<domain>/terragrunt.hcl is a ~20-line configuration file that supplies the domain's inputs — that is the separation on disk.

# live/prod/domains/fraud/terragrunt.hcl
include "root" { path = find_in_parent_folders() }
include "common" { path = "${get_parent_terragrunt_dir()}/_envcommon/domain-account.hcl" }
 
inputs = {
  domain_name          = "fraud"
  domain_owner_email   = "[email protected]"
  data_residency       = "eu-central-1"
  sensitivity_default  = "high"    # PII-heavy domain
  works_council_review = true      # works-council notification required on tag changes
}

The shared module provisions the producer account's S3 bucket, KMS CMK, Glue catalog database, and (critically) registers those Glue databases with LakeFormation as managed. Once a database is under LakeFormation management, IAM-only access is disabled for its tables — every query goes through an LF permission check. This is the switch most Fabric programmes never flip; without it, LakeFormation is decorative.

# _envcommon/domain-account.hcl → data-lake module (excerpt)
resource "aws_lakeformation_resource" "domain_bucket" {
  arn                     = aws_s3_bucket.domain.arn
  use_service_linked_role = true
}
 
resource "aws_glue_catalog_database" "domain" {
  name = "${var.domain_name}_curated"
  location_uri = "s3://${aws_s3_bucket.domain.id}/curated/"
}
 
# Hand the Glue database to LakeFormation. Without this, IAM still wins.
resource "aws_lakeformation_permissions" "revoke_iam_default" {
  principal   = "IAM_ALLOWED_PRINCIPALS"
  permissions = ["ALL"]
  database { name = aws_glue_catalog_database.domain.name }
}
# ...paired with a lifecycle { prevent_destroy = true } outer resource
# that removes the IAM_ALLOWED_PRINCIPALS default. Belt and braces.

§ 3 — The LF-tag ontology, and why grants are written against it

Grant sprawl is the second-year killer of every LakeFormation programme we have seen. The first year, someone writes aws_lakeformation_permissions per table. The second year, there are 4,000 of them, nobody knows which are still needed, and every access review is a full-scan.

The underlying cause is process, not tool: nobody schedules a recurring review of the grant set, so it only grows. Two habits keep it healthy — do both. First, a quarterly access-review cadence that revokes any grant not exercised in CloudTrail during the review window; this is also the artefact DORA and SOC 2 auditors expect. Second, authoring every grant through a Terraform PR so new grants are peer-reviewed at creation and revocation is a one-line diff, not a console hunt.

The tool complement — the one that stops the sprawl from re-forming — is to grant against LF-tag expressions, not table names, and let new tables inherit their permissions from their tags.

The ontology is small on purpose. Ours, on most engagements:

resource "aws_lakeformation_lf_tag" "domain" {
  key    = "domain"
  values = ["fraud", "policy", "claims", "marketing", "finance"]
}
 
resource "aws_lakeformation_lf_tag" "sensitivity" {
  key    = "sensitivity"
  values = ["public", "internal", "confidential", "high"]
}
 
resource "aws_lakeformation_lf_tag" "residency" {
  key    = "residency"
  values = ["eu", "ch", "us", "global"]
}

Three keys. Twelve values. Every grant is written against expressions like domain=fraud AND sensitivity<=confidential AND residency=eu. New tables get the right access the moment they are tagged; tagging is the only operation reviewers care about; and the entire grant model fits on a whiteboard.

The consumer-facing grant then looks like this. Note how the resource references no table names at all:

resource "aws_lakeformation_permissions" "analyst_read_fraud_low_pii" {
  principal   = aws_iam_role.fraud_analyst.arn
  permissions = ["SELECT", "DESCRIBE"]
 
  lf_tag_policy {
    resource_type = "TABLE"
    expression {
      key    = "domain"
      values = ["fraud"]
    }
    expression {
      key    = "sensitivity"
      values = ["public", "internal", "confidential"]  # not "high"
    }
    expression {
      key    = "residency"
      values = ["eu"]
    }
  }
}

And column-level redaction — the thing every audit asks for, in every industry — collapses to a single tag on the column:

resource "aws_lakeformation_resource_lf_tags" "iban_column_high" {
  database { name = "policy_curated" }
  table    { name = "customer" }
  column   { name = "iban" }
 
  lf_tag {
    key    = "sensitivity"
    values = ["high"]
  }
}

The analyst role above, which excludes sensitivity=high, now sees the customer table but with iban redacted. No view. No masking function. No table-per-role duplication. The column is invisible because the grant expression cannot reach it.

Looking forward — agentic ontology access. The next affordance this ontology deserves is an MCP-exposed catalog tool: domain engineers ask an assistant "which sensitivity level applies to a column that stores an EU IBAN?" and the tool answers from the live LF-tag policy, not from a stale wiki. That is defence in depth — the ontology is small enough to reason about, and an assistant makes it queryable at the moment of tagging. The same tool later becomes how agentic consumers understand the catalog they are pointed at, instead of guessing. It is a natural next step; it is not in the reference implementation yet.

§ 4 — DataZone: publishing a data product

LakeFormation is the enforcement layer. DataZone is the product layer — where domain teams publish an asset with a business glossary, and consumers request it through a workflow rather than paging the data owner on Slack.

Segment DataZone into two layers. The shape that maps cleanly onto the account topology from § 1 is a platform-owned root and domain-owned branches:

• Platform layer — one root DataZone domain, owned by the platform team. Holds the LF-tag vocabulary (§ 3), the environment blueprints (§ 2), the IAM naming conventions, the subscription approval workflow config, and the shared business-glossary terms used across the org.
• Domain layer — one DataZone sub-domain per business domain (fraud, policy, claims), owned by that domain's team. Sub-domains hold the domain's projects — the working units where assets are published — and the domain owns subscription decisions inside its own sub-domain.

Ownership at a glance:

Note on DataZone's data model: the current DataZone API does not surface first-class "sub-domain" resources — the two-layer separation lives in convention (project naming, IAM boundaries, ownership) rather than in schema. The convention holds where it matters: platform and domain teams edit different Terragrunt files, with different reviewers, and never step on each other's grants.

The Terraform to stand up the root domain (platform team owns) and the fraud sub-domain + project (fraud team owns):

# Platform-owned: the root domain
resource "aws_datazone_domain" "cordata" {
  name                  = "cordata-analytics"
  domain_execution_role = aws_iam_role.datazone_execution.arn
  kms_key_identifier    = aws_kms_key.datazone.arn
}
 
# Domain-owned: the fraud sub-domain expressed as a DataZone project
resource "aws_datazone_project" "fraud" {
  domain_identifier = aws_datazone_domain.cordata.id
  name              = "fraud-domain"
  description       = "Fraud detection data products — owned by the fraud team"
}
 
# Associate the producer account so its Glue catalog is discoverable
resource "aws_datazone_environment_blueprint_configuration" "fraud_aws" {
  domain_id                = aws_datazone_domain.cordata.id
  environment_blueprint_id = data.aws_datazone_environment_blueprint.default_data_lake.id
  enabled_regions          = ["eu-central-1"]
  provisioning_role_arn    = aws_iam_role.datazone_provisioning.arn
}

The publish flow, from producer keystroke to consumer query:

Diagram read-out. Producer publishes an asset with attached glossary terms; DataZone catalogs it. A consumer searches, files a subscription request, and DataZone routes the approval to the sub-domain's owning team — not to the platform team. On approval, DataZone triggers a LakeFormation cross-account grant against the LF-tag expression that matches the consumer's role; no engineer touches IAM. Query time, LakeFormation enforces the LF-tag policy and applies column-level redaction transparently.

The same flow, one line each:

1. Producer — domain engineer calls create-asset + publish-asset on a Glue table, attaching business-glossary terms (Transaction, PII-EU).
2. Consumer — discovers the asset in DataZone search, files a subscription request with a business justification.
3. Approval — DataZone routes the request to the sub-domain owners. Approval is one click and produces an auditable record.
4. Provisioning — approval triggers a LakeFormation cross-account grant against the LF-tag expression matching the consumer's role. No engineer edits any IAM policy. The grant is logged and revocable through the same workflow.
5. Query — consumer runs Athena; LakeFormation checks the LF-tag policy per query; column-level redaction applies transparently.

The tier DataZone does not give you. DataZone catalogs assets — tables, schemas, sample rows, freshness metadata. It does not catalog metrics — the business definitions ("monthly recurring revenue", "policy attach rate", "fraud recall at 30 days") that consumers actually ask for. The current best fit for that missing tier on the AWS-native stack is dbt Semantic Layer, deployed against the same LF-tag-governed tables and publishing typed metrics that BI tools, notebooks and LLM agents query through the semantic API rather than reinventing the SQL. This matters most for the AI consumer: raw SQL against a governed lake is unsafe for agents; a semantic layer above it is what makes agentic consumption of a data product possible without hallucination. Cordata treats the semantic layer as its own tier and its own decision — a follow-up post covers it in depth.

This is the self-serve half of the Data Fabric contract from the pillar: the platform team built the substrate; domain teams publish through it; consumers subscribe through it; nobody's Slack DMs are on the critical path of the workflow.

§ 5 — The catalog-of-catalogs trap

The single hardest failure mode for a Fabric programme is not tool choice — it is the moment a second governance catalog appears over the same data. Two industries where we have seen this pattern begin: banks that adopted Databricks first for their data-science teams and then landed AWS-native governance later for the group; insurers that inherited a second catalog through a subsidiary and never consolidated. Unity Catalog is the most common counterparty because Databricks arrived first at many DACH data-science teams — but the trap generalises. Any second catalog plane over the same S3 objects produces the same failure: Unity Catalog, Microsoft Purview, Snowflake Horizon, or a legacy Collibra with hard grants.

The temptation is always to phrase it as future work: "Catalog A governs the assets under tool A, Catalog B governs the assets under tool B — we will federate later."

Federation does not happen. What happens instead:

Diagram read-out. The same S3 objects sit under two catalog planes at once. Each plane thinks it owns the truth of who can see what. When Catalog A revokes an analyst's access, the object is still reachable through Catalog B — because Catalog B did not observe the revocation. Auditors reconstructing "who read this table" have to compose two audit trails and hope they overlap; they rarely do.

The failure modes, in more detail:

• Dual grants drift. The same underlying S3 object is governed by two separate ACL systems. When a domain revokes an analyst's access in Catalog A, that plane revokes; Catalog B does not, because it does not know. The analyst can still query the same table through Catalog B's engine.
• Audit trails split. BaFin asks "who has read the customer table in the last 90 days?" You answer twice — once from each catalog's audit trail — and hope the two lists compose. They rarely do.
• IAM roles double-purpose. Catalog B's engine (a Databricks-cluster instance profile, a Purview scan role) ends up with s3:GetObject grants that duplicate Catalog A's permission set. The moment those duplicates diverge, you have a permission your primary governance plane cannot see.
• The ontology forks. Each catalog's tag vocabulary becomes an overlapping dialect that means almost the same thing as the others. Nobody documents the deltas. Every new table asks two or three questions instead of one.

What about "catalog of catalogs" federation? Unity Catalog can federate a Glue catalog — attach it read-only, resolve names, but grants stay on the AWS side. On paper this looks like the escape hatch: pick one catalog as the "catalog of catalogs" and let it read the others without owning their policies. In practice the pattern collapses because "read-only" is a promise a data engineer can break in fifteen minutes by writing a Databricks table onto the federated S3 path — and once that write exists, Catalog B has effective policy authority whether you granted it or not. Read-only federation is only safe when both catalogs and the underlying storage enforce the read-only boundary — which almost no real deployment does.

The refusal is architectural, not political: pick one governance plane per data domain, ever. If a domain lives on Databricks, it lives under Unity Catalog end to end; if a domain lives on the AWS-native stack, it lives under LakeFormation + DataZone end to end. Domains can differ from each other — that is fine. What cannot happen is a single domain's data governed by two planes.

§ 6 — Three DACH compliance checkpoints, hit along the way

The topology above earns its complexity by making three DACH-specific compliance conversations tractable that are otherwise slow, expensive, or unanswerable.

Outsourcing register (BaFin § 25b KWG · FINMA outsourcing circular). DataZone's domain-to-account mapping is an auditable outsourcing register out of the box. Every producer account is a service provider (AWS), every consumer account is a consuming business function, every subscription is a documented data flow between them with an owner and an approval trail. When BaFin or FINMA asks for the outsourcing inventory of data-processing activities, the answer is a DataZone export, not a spreadsheet reconstructed from memory.

DSGVO Art. 28 (processor obligations). AWS is a processor; the residency-pinning half of the compliance answer lives in the residency=eu LF-tag combined with a region-scoped Glue catalog and S3 bucket. The LF-tag is the enforcement — a grant carrying residency=eu mathematically cannot resolve to a table registered outside eu-central-1. This turns the processor-boundary conversation from "trust us" into "here's the Terraform, and here's the LakeFormation grant expression that would fail if we broke it."

Works-council review of RBAC changes (AVR / Betriebsrat). The DACH-specific trap of naive LakeFormation is that every individual grant becomes a works-council notification, which nobody sustains past month three. The LF-tag ontology fixes this at the primitive level: individual grants are exempt from AVR review because they operate against a pre-approved policy expression; only changes to the tag ontology itself — adding a new sensitivity level, redefining residency=eu, expanding the tag vocabulary — require works-council review. That is a review cycle the works council can actually run, at a cadence that stays honest, without either side burning out.

Deletion, masking, and DSGVO Art. 17 across a mesh. The one DACH conversation this post deliberately does not close is right-to-erasure. When a customer requests deletion under DSGVO Art. 17, their data lives in eight producer accounts, has been joined into ML feature stores, is cached in analytical extracts, and has been queried into audit logs. The architecture above supports it — LF-tags carry the customer-domain and residency dimensions the erasure job needs to enumerate — but the mechanics of distributed deletion (tombstoning, forgotten indexes, backup expiry, embedding-store re-ingestion, masking versus purge) deserve their own treatment. A companion post is planned: DSGVO Art. 17 in a data mesh — how erasure actually works when data lives in eight accounts.

§ 7 — What this does not do

The topology above is the Fabric half. It does not, on its own, produce a data mesh — a self-serve substrate with no domain teams to publish through it is still an abandoned museum. Getting the domain teams to own their data, publish through DataZone, honour their own SLAs, and participate in the review cadence is the Mesh half: the cultural adoption, the operating model, the accountability. That is the subject of Cordata's next pillar — Data Governance — and there is no Terraform for it.

It also does not, on its own, produce the semantic layer above the data products — the versioned metric definitions that let a BI dashboard, a notebook and an LLM agent compute "monthly recurring revenue" the same way, from the same governed source. That tier lives above DataZone, not inside it, and it deserves its own decision — its own follow-up post.

What is fair to claim about the code above: it is the assembly the components have been asking for since DataZone reached feature parity for cross-account subscriptions, it composes cleanly with the config-vs-infra separation patterns we have applied at DACH clients over the last five years, and it holds up under the three regulatory conversations that matter most in DACH. What is not fair to claim: that this is a substitute for the harder half of the problem. The harder half is people.

Return to the pillar: Data Engineering — the Fabric + Mesh Hybrid. Follow-ups planned: the semantic layer as its own tier above the data products; DSGVO Art. 17 across a mesh — how erasure actually works when data lives in eight accounts; a DataEng-only companion on the pipeline half (ETL, OpenLineage, Great Expectations); and the Governance pillar covering the Mesh half — cultural adoption, ownership, DORA's oversight chain.