Episode Details

Opening Hook & Teaching PromiseSomewhere right now, a data analyst is heroically exporting a hundred‑megabyte CSV from Microsoft Fabric—again. Because apparently, the twenty‑first century still runs on spreadsheets and weekend refresh rituals. Fascinating. The irony is that Fabric already solved this, but most people are too busy rescuing their own data to notice.Here’s the reality nobody says out loud: most Fabric projects burn more compute in refresh cycles than they did in entire Power BI workspaces. Why? Because everyone keeps using Dataflows Gen 2 like it’s still Power BI’s little sidecar. Spoiler alert—it’s not. You’re stitching together a full‑scale data engineering environment while pretending you’re building dashboards.Dataflows Gen 2 aren’t just “new dataflows.” They are pipelines wearing polite Power Query clothing. They can stage raw data, transform it across domains, and serve it straight into Direct Lake models. But if you treat them like glorified imports, you pay for movement twice: once pulling from the source, then again refreshing every dependent dataset. Double the compute, half the sanity.Here’s the deal. Every Fabric dataflow architecture fits one of three valid patterns—each tuned for a purpose, each with distinct cost and scaling behavior. One saves you money. One scales like a proper enterprise backbone. And one belongs in the recycle bin with your winter 2021 CSV exports.Stick around. By the end of this, you’ll know exactly how to design your dataflows so that compute bills drop, refreshes shrink, and governance stops looking like duct‑taped chaos. Let’s dissect why Fabric deployments quietly bleed money and how choosing the right pattern fixes it.Section 1 – The Core Misunderstanding: Why Most Fabric Projects Bleed MoneyThe classic mistake goes like this: someone says, “Oh, Dataflows—that’s the ETL layer, right?” Incorrect. That was Power BI logic. In Fabric, the economic model flipped. Compute—not storage—is the metered resource. Every refresh triggers a full orchestration of compute; every repeated import multiplies that cost.Power BI’s import model trained people badly. Back there, storage was finite, compute was hidden, and refresh was free—unless you hit capacity limits. Fabric, by contrast, charges you per activity. Refreshing a dataflow isn’t just copying data; it spins up distributed compute clusters, loads staging memory, writes delta files, and tears it all down again. Do that across multiple workspaces? Congratulations, you’ve built a self‑inflicted cloud mining operation.Here’s where things compound. Most teams organize Fabric exactly like their Power BI workspace folders—marketing here, finance there, operations somewhere else—each with its own little ingestion pipeline. Then those pipelines all pull the same data from the same ERP system. That’s multiple concurrent refreshes performing identical work, hammering your capacity pool, all for identical bronze data. Duplicate ingestion equals duplicate cost, and no amount of slicer optimization will save you.Fabric’s design assumes a shared lakehouse model: one storage pool feeding many consumers. In that model, data should land once, in a standardized layer, and everyone else references it. But when you replicate ingestion per workspace, you destroy that efficiency. Instead of consolidating lineage, you spawn parallel copies with no relationship to each other. Storage looks fine—the files are cheap—but compute usage skyrockets.Dataflows Gen 2 were refactored specifically to fix this. They support staging directly to delta tables, they understand lineage natively, and they can reference previous outputs without re‑processing them. Think of Gen 2 not as Power Query’s cousin but as Fabric’s front door for structured ingestion. It builds lineage graphs and propagates dependencies so you can chain transformations without re‑loading the same source again and again. But that only helps if you architect them coherently.Once you grasp how compute mu