Episode Details

From Code Cruncher to Creative Thinker: How Microsoft Copilot in Fabric Rewired My Data Engineering JourneyEver spent what felt like an entire summer afternoon just transforming a CSV file? I have—and to say it sapped my motivation would be an understatement. But that was before Microsoft Copilot entered the chat. In this post, I’ll share the winding, sometimes embarrassing, sometimes revelatory path I took from dreading routine data engineering work to rediscovering why I loved building things with code in the first place—all thanks to a little AI magic (and a few hard-learned lessons).When Burnout Met Automation: A Cautionary TaleI used to lose entire weekends to CSV file conversions. Not kidding. My Saturdays would dissolve into a blur of error messages while debugging Spark code that refused to cooperate. Coffee cups would pile up as the sun went down, and I'd realize another day had vanished into the digital void.Sound familiar?The Weekend-Eating MonsterConverting files from CSV to Delta Parquet tables was my personal nemesis. What should have been simple became a soul-crushing time sink. I'd start Friday evening thinking, "This'll take an hour, tops." By Sunday night, I'd be questioning my career choices.Research backs up my pain – automation can reduce routine task times by up to 40%. But knowing that didn't help when I was knee-deep in code errors.Skepticism: My Default SettingWhen Copilot promised to handle these tasks, I laughed. Seriously? Hand over my code to an AI assistant? The trust issues were real.* What if it made mistakes I wouldn't catch?* What if it created more problems than solutions?* What if I became... replaceable?But desperation eventually trumped skepticism.Old Me vs. New MeThe transformation was almost embarrassing:Old me: Spent 6+ hours creating a fiscal calendar, cursing at my screen.New me: Types a prompt, reviews the generated code, done in 15 minutes.Manual data transformation tasks that once devoured my weekends now take minutes. ETL workflows that used to require days of coding and debugging? Handled through natural language prompts."Sometimes, freeing yourself from a tedious workflow is the most creative thing you can do." – Inder RanaRana's words hit different now. The relief of letting go was unexpected. I found myself having actual free time. I rediscovered hobbies. I remembered what my family looked like.The Surprising AftermathThe biggest shock wasn't the efficiency gain - it was the mental space that opened up. Without the dread of endless debugging sessions, my mind wandered to bigger questions and creative solutions.Yes, I still review everything Copilot generates. Yes, I sometimes need to tweak the code. But the 40% time savings? In my case, that's a conservative estimate.My burnout didn't just meet automation. It was thoroughly defeated by it.The Lost Art of Prompt Engineering (Or: Talking To Robots For Fun And Profit)I never thought I'd develop a creative relationship with an AI, but here we are. Writing prompts for Copilot has somehow become one of the most unexpectedly creative parts of my job as a data engineer.Remember when programming meant memorizing exact syntax? Those days feel distant now.The Accidental Monster FactoryLast month, I was exhausted after a long day of data wrangling. My brain was fried. I needed to create a simple data transformation table, but somehow typed: "create fantasy monster table with damage stats and special abilities."Copilot's response? A bizarre mix of SQL syntax and fantasy RPG content that made absolutely no sense. It tried to create columns for "acidBreath" and "tentacleCount" alongside my actual data fields.I laughed for five minutes straight. Then realized something important: I was talking to my development environment. Not coding. Talking.The Prompt-Review-Improve LoopI've developed a workflow now:* Write a natural language prompt* Review what Copilot generates* Refine my prompt with more details* Repeat until perfectIt's less like progr