Data Engineer

JD - Data Engineer Role Overview: We are looking for a skilled Data Engineer to build and optimize data pipelines for a real-time Digital Twin platform powering mobility simulation, complex event processing, and multi-agent learning. You'll design the backbone for scalable, low-latency ingestion and processing of high-volume sensor, vehicle, and infrastructure data to feed prediction models and simulations. Responsibilities:
• Design and implement streaming data pipelines from IoT sensors, camera, vehicle telemetry, and infrastructure systems.
• Build scalable infrastructure using Kinesis, Apache Flink / Spark for real-time and batch workloads.
• Enable time-series feature stores and sliding window processing for mobility patterns.
• Integrate simulation outputs and model predictions into data lakes in AWS.
• Maintain data validation, schema versioning, and high-throughput ingestion
. • Collaborate with Data Scientists and Simulation Engineers to optimize data formats (e.g., Parquet, Protobuf, Delta Lake).
• Deploy and monitor pipelines on AWS cloud and/or edge infrastructure. Requirements:
• 3+ years of experience in data engineering, preferably with real-time systems. Overall 5+ Years of experience
• Proficient with Python, SQL, and distributed data systems (Kinesis, Spark, Flink, etc.).
• Strong understanding of event-driven architectures, data lakes, and message serialization.
• Experience with sensor data processing, telemetry ingestion, or mobility data is a plus
. • Familiarity with Docker, CI/CD, Kubernetes, and cloud-native architectures.
• Familiarity with bulding data pipelines & its workflows (eg: Airflow