Description:
Design, build, and maintain scalable and efficient real-time data pipelines using PySpark and SQL on Azure Databricks.
Develop data processing logic in Python, ensuring maintainability and scalability of code.
Implement and manage streaming-first ETL/ELT pipelines for processing high-volume event data.
Design and implement pipelines for end-to-end processing of behavioral event streams, including ingestion, transformation, enrichment, and serving.
Work with high-volume clickstream and event-based datasets, including web and app tracking data with dynamic and evolving schemas.
Design data solutions optimized for low-latency processing and near real-time availability.
Integrate and manage streaming data sources, including Kafka (or Azure Event Hubs), ensuring reliable ingestion and processing.
