Big Data Engineer

This role is for one of the Weekday's clients

Salary range: Rs 1500000 - Rs 2500000 (ie INR 15-25 LPA)

Min Experience: 4+ years

Location: Bengaluru

JobType: full-time

We are looking for a highly skilled and motivated Big Data Engineer with 4-8 years of experience in designing, developing, and optimizing large-scale data processing systems. The ideal candidate will have strong expertise in Scala, Spark, and Hadoop ecosystems, along with hands-on experience in building scalable data pipelines and distributed data processing applications. This role requires a strong analytical mindset, problem-solving ability, and passion for working with high-volume, high-velocity data environments.

As a Big Data Engineer, you will be responsible for developing robust data solutions that support business intelligence, analytics, and data-driven decision-making across the organization. You will collaborate closely with data architects, data scientists, software engineers, and business stakeholders to deliver efficient and reliable data platforms.

Requirements

Key Responsibilities

• Design, develop, and maintain scalable big data solutions using Scala, Apache Spark, and Hadoop technologies.
• Build and optimize batch and real-time data processing pipelines for large datasets.
• Develop reusable frameworks and components for data ingestion, transformation, validation, and storage.
• Work extensively with distributed computing technologies such as HDFS, Yarn, Hive, and MapReduce.
• Implement high-performance Spark applications and optimize data workflows for speed and scalability.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Monitor, troubleshoot, and enhance existing big data infrastructure and applications.
• Ensure data quality, integrity, and consistency across multiple systems and platforms.
• Participate in architecture discussions, code reviews, and technical documentation.
• Support deployment, automation, and performance tuning activities in production environments.
• Stay updated with emerging big data technologies and recommend improvements to the existing ecosystem.

Required Skills

Must-Have Skills

• Strong hands-on experience in Scala
• Expertise in Apache Spark
• Solid understanding of the Hadoop ecosystem
• Experience in developing distributed data processing applications
• Strong knowledge of data structures, algorithms, and parallel processing concepts
• Experience with large-scale ETL and data transformation pipelines
• Good understanding of performance optimization and troubleshooting in big data environments

Good-to-Have Skills

• Proficiency in Python
• Experience with Hive, HDFS, Yarn, and MapReduce
• Exposure to enterprise-level Big Data platforms and cloud-based data ecosystems
• Familiarity with workflow orchestration and scheduling tools
• Knowledge of real-time streaming technologies is an added advantage
• Understanding of DevOps practices and CI/CD pipelines for data engineering

Experience & Qualifications

• 4-8 years of experience in Big Data Engineering or Data Platform Development
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field
• Strong experience in handling structured and unstructured data at scale
• Experience working in Agile development environments
• Excellent communication and collaboration skills

Preferred Candidate Profile

The ideal candidate is a self-driven professional who thrives in fast-paced environments and enjoys solving complex data engineering challenges. You should possess strong debugging and optimization skills, attention to detail, and the ability to work independently while contributing effectively within a collaborative team. Candidates with experience in high-volume data environments, enterprise data lakes, or large-scale analytics platforms will be highly preferred.