Building a Real-Time Weather Data Pipeline using Kafka, Spark, and Grafana

2026-03-21 · Source: Data Engineering on Medium · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, short

Summary

A real-time weather data pipeline has been developed using a modern data engineering stack to fetch, process, store, and visualize live weather information. The pipeline begins by ingesting real-time weather data from the OpenWeatherMap API via a Python producer script, which sends JSON-formatted messages containing city, temperature, humidity, wind speed, and timestamp to an Apache Kafka topic every 30 seconds. Apache Spark Structured Streaming then consumes this data from Kafka, performing necessary transformations like binary-to-string conversion, JSON parsing, and timestamp formatting. The processed data is stored in TimescaleDB, a PostgreSQL-based time-series database optimized for efficient storage and querying. Finally, Grafana connects to TimescaleDB to display real-time dashboards, featuring temperature, humidity, and wind speed trends for multiple cities, with time-based aggregation applied to ensure smooth, meaningful visualizations. All components, including Kafka, Zookeeper, Spark, TimescaleDB, and Grafana, are containerized using Docker and Docker Compose for portability and consistent deployment.

Key takeaway

For Data Engineers building real-time monitoring or analytics systems, this pipeline architecture offers a robust blueprint. You should consider integrating Kafka for reliable streaming, Spark Structured Streaming for efficient processing, and TimescaleDB for optimized time-series data storage. Leveraging Grafana with time-based aggregation will ensure your dashboards provide clear, actionable insights from noisy streaming data, while Docker containerization simplifies deployment and environment consistency.

Key insights

Modern data pipelines integrate Kafka, Spark, TimescaleDB, and Grafana for real-time data ingestion, processing, storage, and visualization.

Principles

• Containerization ensures portability.
• Time-series databases optimize storage.
• Aggregation improves visualization clarity.

Method

Data is ingested from an API via Python to Kafka, processed by Spark Structured Streaming, stored in TimescaleDB, and visualized in Grafana, with all components containerized using Docker.

In practice

• Use OpenWeatherMap API for weather data.
• Apply time_bucket() for smooth Grafana trends.
• Containerize with Docker Compose for setup.

Topics

• Real-time Data Pipelines
• Apache Kafka
• Spark Structured Streaming
• TimescaleDB
• Grafana

Best for: Data Engineer, MLOps Engineer, Software Engineer

Related on AIssential

• Apache Kafka for Data Engineers — Beginner Guide — Apache Kafka provides a scalable, fault-tolerant, and decoupled architecture for real-time data streaming.
• Ask Your Factory Floor Anything: Structuring Industrial Data for AI Agents — Integrating time-series data and equipment context in a single PostgreSQL database simplifies contextual queries and enhances data integrity.
• How Figma Upgraded Data Pipeline from Multi-Day Latency to Real-Time — Incremental synchronization via Change Data Capture significantly improves data freshness and reduces costs for large-scale data pipelines.
• Building Your First Streamlit App: Uploads, Charts, and Filters (Part 1) — Streamlit enables rapid development of interactive data exploration apps with minimal code.
• How Modern Data Pipelines Drive AI Application Development in Dubai — Modern data pipelines are foundational for scalable, real-time AI applications, ensuring data quality and efficient processing.
• From data lake to AI-ready analytics: Introducing new data source with S3 Tables in Amazon Quick — Amazon Quick now directly queries Apache Iceberg tables in S3, enabling near real-time, AI-powered analytics on data lakes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.