Big Data Engineer- Pune
Job Title: Big Data Engineer
Experience Level: Mid-Level (4+ Years)
Job Summary:
We are seeking a highly skilled Big Data Engineer to join our data engineering team. The ideal candidate will have experience in designing, developing, and optimizing large-scale data processing systems. You will work with big data technologies to build scalable data pipelines, support advanced analytics, and enable real-time decision-making across the organization.
Key Responsibilities:
Design and develop scalable data pipelines to ingest, transform, and store structured and unstructured data.
Build and maintain distributed processing systems using tools like Apache Spark, Hadoop, Kafka, and Hive.
Work closely with data scientists, analysts, and business users to understand data requirements.
Ensure data quality, integrity, and lineage across all stages of the pipeline.
Develop ETL/ELT processes and optimize existing workflows for performance and reliability.
Integrate data from various sources including APIs, databases, and cloud storage systems.
Monitor and troubleshoot data pipeline issues and ensure uptime and reliability.
Implement data security and governance best practices.
Stay updated with the latest trends and technologies in big data and cloud platforms.
