One Month Into Learning Data Engineering in Public: Here’s What I Didn’t Write About

2026-06-25 · Source: Towards Data Science · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering · Depth: Novice, medium

Summary

The author details their first month learning data engineering, deviating from a planned 12-month self-study roadmap. Instead of a sequential approach (SQL, Python, Git, Spark, Airflow, Databricks), they adopted an iterative method, continuously building and refining a small ETL pipeline. This process revealed that technical hurdles like idempotency, persistence, and portability often masked deeper conceptual "thinking lessons" about assumptions and environmental dependencies. The author also grappled with waning motivation and "shiny object syndrome," nearly pivoting to AI engineering, before a LinkedIn contact reaffirmed their original roadmap. A key learning was the effectiveness of completing small, focused projects over starting large, potentially unfinished ones for sustained learning and skill acquisition.

Key takeaway

For aspiring Data Engineers transitioning from other roles, your self-study roadmap should be a flexible guide, not a rigid contract. Focus on building small, iterative projects to uncover practical challenges and conceptual lessons, rather than strictly following a sequential curriculum. Publicly documenting your journey can provide crucial accountability and unexpected external validation, helping you stay on track when motivation wanes or "shiny object syndrome" strikes.

Key insights

Learning data engineering effectively involves iterative building, revealing deeper conceptual lessons beyond just tools.

Principles

• Plans are starting points, not contracts.
• Technical walls often hide thinking lessons.
• Small, finished projects drive learning.

Method

The author's method involved iteratively building a small ETL pipeline, pushing it until it broke, and learning tools (SQL, Python, Git) as demanded by the pipeline's evolving needs, rather than following a strict sequential curriculum.

In practice

• Build small, complete mini-projects.
• Document your learning journey publicly.
• Distrust "it worked once" assumptions.

Topics

• Data Engineering
• Self-Study Roadmap
• ETL Pipelines
• Learning Strategies
• Career Transition
• Public Learning

Best for: Data Engineer, Data Analyst, Software Engineer

Related on AIssential

• The Day I Realized Learning Databricks Is More About Patience Than Speed — Effective learning, especially for complex platforms like Databricks, prioritizes deep understanding and making connections over rapid topic coverage.
• The Internship That Made Me Stop Practicing and Start Building — A structured data science internship bridges the gap between theoretical knowledge and practical, end-to-end project execution.
• I Quit My $130,000 ML Engineer Job After Learning 4 Lessons — Big Tech's stability and mature systems can stifle innovation and personal growth for action-oriented engineers.
• You Will Learn More From One Failed Kaggle Submission Than From Ten Completed Online Courses — Genuine data science competence stems from practical problem-solving and learning from failure, not passive course consumption.
• STEM PhD's transitioning to MLE/Data [R] — A postdoc in an ML lab is a proven path for STEM PhDs to transition into MLE/Data Science roles.
• Day 30 of SPE Boot Camp on AI & Machine Learning: My 30-Day Learning Journey — Hands-on project-based learning significantly enhances practical AI/ML skill acquisition.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.