Episode Details
Back to Episodes
Fabric Rewrote Data Engineering
Published 1 month, 2 weeks ago
Description
Microsoft Fabric didn’t make data engineering easier.
It made ambiguity cheaper to ship. This episode explains why teams feel faster and more out of control at the same time after adopting Fabric and Copilot—and why that isn’t a tooling problem. Fabric removed the ceremony that used to slow bad decisions down. Copilot removed the typing, not the consequences. The result is architectural erosion that shows up first as cost spikes, conflicting dashboards, and audit discomfort—not broken pipelines. If your Fabric estate “works” but feels fragile, this episode explains why. What You’ll Learn 1. What Fabric Actually Changed (and What It Didn’t) Fabric didn’t rewrite data engineering because of better UI or nicer tools. It rewrote it by collapsing:
They fail expensively. Because all workloads draw from a shared capacity meter:
Teams without enforcement get faster at shipping entropy. 5. Why Raw Tables Become a Cost and Security Liability When raw tables are queryable:
Raw tables are not a consumption API. 6. Case Study: The “Haunted” Capacity Spike A common Fabric incident pattern:
Warehouses are expected to enforce structure—but they can’t enforce contracts that never existed. Without explicit schema enforcement:
It made ambiguity cheaper to ship. This episode explains why teams feel faster and more out of control at the same time after adopting Fabric and Copilot—and why that isn’t a tooling problem. Fabric removed the ceremony that used to slow bad decisions down. Copilot removed the typing, not the consequences. The result is architectural erosion that shows up first as cost spikes, conflicting dashboards, and audit discomfort—not broken pipelines. If your Fabric estate “works” but feels fragile, this episode explains why. What You’ll Learn 1. What Fabric Actually Changed (and What It Didn’t) Fabric didn’t rewrite data engineering because of better UI or nicer tools. It rewrote it by collapsing:
- Storage
- Compute
- Semantics
- Publishing
- Identity
- Environment boundaries
- Tool handoffs
- Deployment friction
- Separate billing surfaces
- Dashboards refresh on time and disagree
- Pipelines succeed while semantics fragment
- Capacity spikes without deployments
- Audits surface ownership gaps no one noticed forming
They fail expensively. Because all workloads draw from a shared capacity meter:
- Bad query shapes
- Unbounded filters
- Copilot-generated SQL
- Refresh concurrency
- Plausible output
- Fast completion
- Syntax correctness
- Deterministic cost
- Schema contracts
- Security intent
- Long-term correctness
Teams without enforcement get faster at shipping entropy. 5. Why Raw Tables Become a Cost and Security Liability When raw tables are queryable:
- Cost becomes probabilistic
- Schema drift becomes accepted behavior
- Access intent collapses into workspace roles
- Copilot becomes a blast-radius multiplier
Raw tables are not a consumption API. 6. Case Study: The “Haunted” Capacity Spike A common Fabric incident pattern:
- No deployments
- No pipeline failures
- Dashboards still load
- Capacity spikes mid-day
- Non-sargable predicates
- Missing time bounds
- SELECT *
- Copilot-generated SQL under concurrency
- Views and procedures as the only query surface
- Execution plans as acceptance criteria
- Cost treated as an engineered property
Warehouses are expected to enforce structure—but they can’t enforce contracts that never existed. Without explicit schema enforcement:
- Drift moves downstream
- Semantic models become patch bays
- KPIs fork silently
- “Power BI is wrong” becomes a recurring sentence