PySpark Developer | Python | SQL

Job Title: PySpark Developer

Location: Chennai / Bangalore / Hyderabad / Pune

Notice Period: Immediate to 30 Days

Job Description :

We are seeking a skilled PySpark Developer with strong experience in Python, PySpark, SQL, and Data Warehousing concepts. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing pipelines and ETL solutions using Spark-based technologies.

Key Responsibilities :

\n
• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.
\n
• Build and optimize Spark jobs for performance, reliability, and scalability.
\n
• Process and transform large datasets using Spark SQL, DataFrames, and RDDs.
\n
• Develop batch and real-time data processing solutions.
\n
• Integrate data pipelines with Hive, HDFS, Snowflake, Redshift, and other data platforms.
\n
• Collaborate with data engineers, analysts, and business stakeholders.
\n
• Monitor data pipelines, troubleshoot issues, and ensure SLA compliance.
\n
• Follow coding best practices, version control, and CI/CD processes.
\n
• Work with Hadoop ecosystem tools and cloud platforms such as AWS, Azure, or GCP.
\n

Required Skills:

\n
• Strong hands-on experience in Python and PySpark
\n
• Expertise in Spark SQL, DataFrames, and RDDs
\n
• Good knowledge of Hadoop (Hive, HDFS, YARN)
\n
• Strong SQL and query optimization skills
\n
• Experience with Data Warehousing concepts
\n
• Knowledge of Parquet, Avro, JSON data formats
\n
• Experience with Git version control
\n
• Familiarity with Airflow, Oozie, or similar scheduling tools
\n
• Exposure to AWS, Azure, or GCP is an added advantage
\n