Posted 22 June, 2026
Senior Data Engineer Databricks/Pyspark
PRI India IT Services Pvt Ltd
Bangalore, Karnataka, IN
Full Time
Reference: 26-00239-740-2
Senior Data Engineer – Databricks (PySpark)
We are seeking a Databricks Consultant with strong expertise in PySpark and Python to design, develop, and optimize large-scale data processing solutions on cloud-based data platforms.
The ideal candidate will build scalable data pipelines, ETL workflows, and analytics solutions using Databricks and Apache Spark, supporting enterprise data platforms in IQVIA.
Key Responsibilities
1. Data Engineering & Databricks Development
2. PySpark & Python Development
3. Data Pipeline & Architecture
4. Performance Tuning & Optimization
5. Integration & Cloud Enablement
6. Best Practices & Governance
7. Collaboration & Delivery
Required Skills & Experience
Core Skills
Technical Skills
Data & Platform Skills
Cloud & Tools
Preferred Qualifications
Typical Experience Range
Value Proposition of Role
We are seeking a Databricks Consultant with strong expertise in PySpark and Python to design, develop, and optimize large-scale data processing solutions on cloud-based data platforms.
The ideal candidate will build scalable data pipelines, ETL workflows, and analytics solutions using Databricks and Apache Spark, supporting enterprise data platforms in IQVIA.
Key Responsibilities
1. Data Engineering & Databricks Development
- Design and develop pipelines using Databricks and PySpark
- Build and maintain distributed data processing workflows using Spark
- Develop reusable data transformation frameworks and components
- Work with Delta Lake for optimized data storage and versioning
2. PySpark & Python Development
- Implement Spark jobs using PySpark for large-scale data processing
- Develop Python-based data processing, automation, and orchestration scripts
- Optimize Spark transformations (joins, aggregations, partitioning)
3. Data Pipeline & Architecture
-
Build and optimize:
- Data pipelines
- Data ingestion frameworks
- Data lake architectures
- Ensure high scalability, fault tolerance, and performance
4. Performance Tuning & Optimization
-
Tune Spark jobs for:
- Performance
- Cost efficiency
- Resource utilization
- Identify bottlenecks in data pipelines and optimize workloads
- Monitor cluster and workload performance
5. Integration & Cloud Enablement
-
Integrate Databricks pipelines with:
- Azure Data Factory / AWS Glue
- Data lakes, APIs, databases
-
Work with cloud platforms:
- Azure / AWS / GCP
6. Best Practices & Governance
-
Implement best practices for:
- Logging
- Monitoring
- Error handling
- Ensure data quality, security, and governance standards
- Follow CI/CD and DevOps practices using Git and pipelines
7. Collaboration & Delivery
-
Collaborate with:
- Data engineers and architects
- Product and business teams
- Translate requirements into technical solutions and pipelines
- Participate in Agile (Scrum) delivery model
Required Skills & Experience
Core Skills
- Strong experience in Databricks and Apache Spark
- Hands-on expertise in PySpark programming
- Strong proficiency in Python for data engineering
- Experience in ETL/ELT pipeline development
Technical Skills
-
PySpark:
- Data Frames, Spark SQL, transformations
-
Python:
- Data processing, scripting, automation
-
SQL:
- Query optimization and data modeling
Data & Platform Skills
- Data warehousing and data modeling concepts
-
Experience with:
- Delta Lake
- Distributed data systems
- Handling structured and semi-structured data
Cloud & Tools
-
Experience with:
- Azure Databricks / AWS Databricks
- Azure Data Factory / AWS Glue
- CI/CD & DevOps tools (Git, pipelines)
- Exposure to monitoring and logging tools
Preferred Qualifications
- Databricks or Spark-related certifications
-
Experience with:
- Data pipeline orchestration (Airflow, Databricks Workflows)
- Large-scale enterprise data platforms
-
Knowledge of:
- Streaming (Spark Structured Streaming)
- ML pipelines (optional)
- Strong analytical and problem-solving skills
- Ability to work in cross-functional teams
- Good communication with stakeholders
- Agile mindset and delivery orientation
Typical Experience Range
- 4–8 years in data engineering
- Minimum 2–3 years in Databricks / PySpark
Value Proposition of Role
- Build scalable big data pipelines on cloud-native platforms
- Enable high-performance analytics and data-driven decision making
