Episode Details
Back to Episodes
Stop Wasting Money: The 3 Architectures for Fabric Data Flows Gen 2
Season 1
Published 5 months, 3 weeks ago
Description
Fabric Dataflows Gen2 architectures: in this episode of M365.fm, Mirko Peters explains why most Microsoft Fabric Dataflows Gen2 deployments quietly burn far too much compute—and how three clear architectures for staging, transforming, and serving data can cut your capacity bill while improving governance and performance. He shows how treating Dataflows Gen2 like “Power BI dataflows 2.0” leads to duplicated ingestion, repeated refreshes, and multiple workspaces pulling the same source data over and over again.
Mirko starts with the core misunderstanding: in Fabric, compute—not storage—is what you pay for. Every refresh spins up distributed compute, lands delta files, and tears clusters down again, so copying the same data into multiple workspaces multiplies your costs without adding value. He explains why Fabric assumes a shared lakehouse model—data lands once in OneLake and is reused many times—and how Dataflows Gen2 were redesigned as pipelines in Power Query clothing to support that pattern with lineage and reuse instead of one‑off imports.
The first architecture he introduces is the Staging (Bronze) Dataflow. Here, each external system—CRM, ERP, HR, line‑of‑business SQL—lands once into standardized delta tables in a shared lakehouse. Mirko shows how to keep logic minimal at this layer (types, basic cleanup, incremental refresh), so refresh jobs are cheap, repeatable, and reusable for every downstream team. This “ingest once, share everywhere” pattern stops five departments from hammering the same API with five near‑identical dataflows.
The second architecture is the Transform (Silver) Dataflow, where business logic, joins, and normalization happen on top of the bronze layer instead of directly against external sources. Mirko explains how to centralize entity logic (customer, product, calendar) into curated silver tables that multiple domains share, avoiding each workspace inventing its own slightly different version. He shows why running transformations against delta data instead of external systems is cheaper, more reliable, and easier to govern.
The third architecture is the Serve (Gold) pattern, where lightweight, consumption‑ready Dataflows or shortcuts feed semantic models, Direct Lake datasets, and downstream tools. Mirko explains how this layer should be thin—final shaping, field naming, and aggregations instead of heavy ETL—so refreshes stay fast and compute stays low. He walks through how Staging–Transform–Serve fits together as a reusable blueprint you can replicate across domains, instead of reinventing pipelines for every new project.
WHAT YOU WILL LEARN
Fabric Dataflows Gen2 are not just a nicer way to import—they are your front door for lakehouse architectures. Once you adopt a Staging–Transform–Serve pattern, each source lands once, transformations become reusable assets, and your capacity spend reflects business value instead of duplicated refresh cycles.
WHO THIS EPISODE IS FOR
This episode is ideal for Fabric architects, data engineers, BI leads, and Power BI professionals who are moving from classic Power BI to Fabric and want to avoid building a sprawling, expensive tangle of Gen2 dataflows. It is especially valuable if you are seeing rising capacity costs, duplicated ingestion across workspaces, or unclear lineage and want a simple, three‑archite
Mirko starts with the core misunderstanding: in Fabric, compute—not storage—is what you pay for. Every refresh spins up distributed compute, lands delta files, and tears clusters down again, so copying the same data into multiple workspaces multiplies your costs without adding value. He explains why Fabric assumes a shared lakehouse model—data lands once in OneLake and is reused many times—and how Dataflows Gen2 were redesigned as pipelines in Power Query clothing to support that pattern with lineage and reuse instead of one‑off imports.
The first architecture he introduces is the Staging (Bronze) Dataflow. Here, each external system—CRM, ERP, HR, line‑of‑business SQL—lands once into standardized delta tables in a shared lakehouse. Mirko shows how to keep logic minimal at this layer (types, basic cleanup, incremental refresh), so refresh jobs are cheap, repeatable, and reusable for every downstream team. This “ingest once, share everywhere” pattern stops five departments from hammering the same API with five near‑identical dataflows.
The second architecture is the Transform (Silver) Dataflow, where business logic, joins, and normalization happen on top of the bronze layer instead of directly against external sources. Mirko explains how to centralize entity logic (customer, product, calendar) into curated silver tables that multiple domains share, avoiding each workspace inventing its own slightly different version. He shows why running transformations against delta data instead of external systems is cheaper, more reliable, and easier to govern.
The third architecture is the Serve (Gold) pattern, where lightweight, consumption‑ready Dataflows or shortcuts feed semantic models, Direct Lake datasets, and downstream tools. Mirko explains how this layer should be thin—final shaping, field naming, and aggregations instead of heavy ETL—so refreshes stay fast and compute stays low. He walks through how Staging–Transform–Serve fits together as a reusable blueprint you can replicate across domains, instead of reinventing pipelines for every new project.
WHAT YOU WILL LEARN
- Why treating Fabric Dataflows Gen2 like old Power BI dataflows explodes compute and refresh costs.
- How a Staging (Bronze) Dataflow layer lands each external source once into reusable delta tables.
- How a Transform (Silver) layer centralizes business logic and joins on top of shared lakehouse data.
- How a Serve (Gold) layer delivers thin, consumption‑ready outputs for Direct Lake and semantic models.
- How to design lineage, workspaces, and refresh patterns so one ingestion serves many consumers without duplication.
Fabric Dataflows Gen2 are not just a nicer way to import—they are your front door for lakehouse architectures. Once you adopt a Staging–Transform–Serve pattern, each source lands once, transformations become reusable assets, and your capacity spend reflects business value instead of duplicated refresh cycles.
WHO THIS EPISODE IS FOR
This episode is ideal for Fabric architects, data engineers, BI leads, and Power BI professionals who are moving from classic Power BI to Fabric and want to avoid building a sprawling, expensive tangle of Gen2 dataflows. It is especially valuable if you are seeing rising capacity costs, duplicated ingestion across workspaces, or unclear lineage and want a simple, three‑archite