From OpenSky to DynamoDB: Building a Real-Time Aircraft Data Pipeline for Wilvor — Aviation…

2026-07-12 · Source: Data Engineering on Medium · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure, Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

The Wilvor aviation operations intelligence platform successfully implemented a real-time aircraft data pipeline, moving live aircraft state vectors from OpenSky into AWS. This foundational system archives raw OpenSky responses to S3, processes data through Kinesis streams, and stores the latest aircraft state in DynamoDB. The pipeline incorporates a raw processor Lambda for validating and mapping OpenSky's array-based data into a clean internal schema, publishing valid records to a "Kinesis Clean" stream, and archiving invalid records with specific error reasons to S3. Key challenges addressed included AWS credential management, DynamoDB float-to-Decimal conversion, Terraform state drift, and Python Lambda packaging. The final system features comprehensive monitoring via structured JSON logs, CloudWatch custom metrics, and dashboards, ensuring observability and reliability for future advanced operational decision-making capabilities.

Key takeaway

For MLOps Engineers or Data Engineers building real-time data pipelines, this project demonstrates critical considerations for robust system design. You should prioritize explicit error handling, archiving invalid records with clear reasons, and implementing comprehensive monitoring from the outset. Ensure your deployment workflow accounts for Python packaging and manage Terraform state diligently to prevent drift. This approach builds trust in your data foundation, enabling reliable downstream analytics and operational decision-making.

Key insights

Building robust data pipelines requires iterative development, explicit error handling, and comprehensive observability.

Principles

• Separate producers from consumers using streaming services.
• Archive raw data for debugging and recovery.
• Treat bad records as a feature, not a failure.

Method

Implement a multi-stage data pipeline: local poller -> S3 raw archive & Kinesis Raw -> Raw Processor Lambda (validate, map, publish to Kinesis Clean, archive bad records to S3) -> Current State Writer Lambda -> DynamoDB.

In practice

• Use Kinesis to decouple data ingestion from processing.
• Implement conditional writes in DynamoDB to prevent stale data.
• Convert Python floats to Decimal for DynamoDB writes.

Topics

• Real-time Data Pipelines
• AWS Kinesis
• DynamoDB
• OpenSky Network
• Data Validation
• CloudWatch Monitoring
• Terraform

Best for: Data Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Why Your Data Pipeline Works Perfectly… Until Real Users Arrive — Data pipelines often fail due to unexpected characteristics of real-world data, not faulty code or infrastructure.
• Self-Hosting Airflow at Home: Automating Stock Price Data Collection — Self-hosting Airflow enables robust, automated data pipelines for MLOps, even on a homelab.
• Expected Goals, End-to-End: An MLOps Pipeline for Hockey on Databricks — Production-ready MLOps infrastructure, not model complexity, ensures reliable and governed machine learning deployments.
• From Chaos to Control: Taming Our Data Cleanup with a Python-Based Tool — Structured data pipelines with clear contracts and operational features are crucial for reliable data processing.
• Telemetry that matters: Designing sustainable, high-impact observability pipelines — Over-collection of telemetry data creates significant waste and hinders effective incident response in complex cloud-native systems.
• DataOps with Dagster: A Practical Guide to Building a Reliable Data Platform — DataOps integrates development and operations principles to build trustworthy, visible, and controllable data systems.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.