| Job Description: |
Req ID- 10487512
Job Title: Developer
Work Location: KOLKATA, WB
Skill Required: Digital : Databricks
Experience : 6-8 years
Job Description:
Platform & Infrastructure:• Oversee Databricks platform configuration, resource management, workspace structuring, and cluster optimization.• Monitor and troubleshoot performance issues across clusters, jobs, notebooks, and pipelines.• Implement governance, security, compliance and data access control using Role-Based Access Control (RBAC) and Unity Catalog.Pipeline Development & Architecture:• Design and implement end-to-end data pipelines using PySpark, SQL, and Delta Lake within a medallion architecture using Data factory And Data bricks.• Build real-time and batch DLT pipelines using Databricks Delta Live Tables with a focus on reliability and scalability.• Optimize Lakehouse architecture for performance, cost-efficiency, and data integrity.• Automate data ingestion, transformation, and validation, including support for streaming (Autoloader) and scheduled workflows.• Perform data transformations, cleansing and validations using data quality rules for consistent and accurate data sets.• Manage and monitor job orchestration, ensuring efficient pipelines run and reliability.CI/CD & DevOps:• Design and maintain CI/CD pipelines for Databricks artifacts (notebooks, jobs, libraries) using tools such as Azure DevOps, GitHub Actions, Terraform or Jenkins.• Support trunk-based development, deployment workflows, and infrastructure-as-code practices.• Manage version control and automated testing using Git and related DevOps practices.Collaboration & Delivery:• Collaborate with product owners, business stakeholders, and data teams to gather requirements and translate them into technical solutions.• Drive the adoption of best practices in coding, versioning, testing, deployment, monitoring, and security.• Provide thought leadership on the best practices in Data Engineering, Architecture and Cloud Computing.Performance Optimization:• Deliver optimized spark jobs and SQL queries for large scale data processing.• Implement partitioning, caching and indexing strategies to improve performance and scalability of big data workloads.• Conduct POCs for capacity planning and recommend appropriate infrastructure optimizations for cost effectiveness.Documentation & Knowledge Sharing:• Created detailed documentations and review them for data workflows, SOPs, Architectural reviews etc.• Mentor junior team members and promote a culture of learning and innovation.• Promote the culture of optimization and cost saving and enable research driven development.________________________________________Required Qualifications:Technical Expertise:• 5+ years in data engineering, with a strong focus on Databricks and Azure ecosystems.• Deep hands-on experience with Data Factory, Databricks Lakehouse Architecture, Delta Lake, PySpark, and Spark job optimization.• Proficiency in Python, SQL, and optionally Scala for building scalable ETL/ELT pipelines.• Strong SQL skills are essential, with hands-on experience in SQL Server or other RDBMS platforms.•
Essential Skills:
Azure Databricks Engineer We are seeking an experienced Senior Azure Databricks Engineer to lead the design, development and optimization of scalable and fault tolerant data solutions using pipelines and notebooks, implementing strong data quality and governance solutions in Lakehouse architecture. You will work across engineering, analytics, and business teams to build robust, high-performant data workflows and ensure best-in-class delivery on Azure and Databricks platforms. |
| Comments for Suppliers: |
|
|
