"Key responsibilities:
Design and implement scalable data pipelines using PySpark.
Build and optimize big data solutions leveraging Apache Spark.
Ingest and transform data using Azure Data Factory.
Store and manage structured and unstructured data in Azure Data Lake Storage.
Develop scalable data models in Azure Synapse Analytics.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Implement data quality, monitoring, and logging frameworks.
Collaborate with Data Scientists, Analysts, and DevOps teams to support analytics and ML initiatives.
Ensure data security, governance, and compliance standards are met.
Automate deployment using CI/CD pipelines (Azure DevOps preferred).
Technical Skills
Languages: Python (PySpark), SQL
Frameworks: Apache Spark
Cloud Platform: Microsoft Azure
Data Tools: Azure Data Factory, Azure Databricks, Azure Synapse, ADLS
DevOps: Azure DevOps, CI/CD"
