Data Engineer

Data Engineer

Location- Kochi/Coimbatore/Chennai

Job Overview

We are seeking a strong Data Engineer - Databricks Migration with 10+ years of experience and hands-on expertise in Databricks, PySpark, and SQL/T-SQL interpretation to support a legacy-to-Databricks migration program. The ideal candidate should have worked predominantly as a Data Engineer throughout their career, ideally beginning with Oracle, SQL Server/T-SQL, ETL, or Data Warehouse technologies and later transitioning into modern cloud-based Data Engineering. The role requires the ability to understand existing T-SQL logic and migrate transformation rules into scalable Databricks and PySpark solutions. Important screening criterion: The candidate must have worked predominantly and consistently as a Data Engineer throughout their career. Profiles that are primarily BI, reporting, dashboarding, analytics, or only recently shifted into data engineering should not be prioritized.

Key Responsibilities:

• Analyze legacy data warehouse, SQL Server, Oracle, ETL, and T-SQL based implementations for migration to Databricks.
• Read and interpret complex T-SQL scripts, stored procedures, functions, views, joins, transformation rules, and data flows.
• Convert business logic from legacy SQL/T-SQL processes into optimized Databricks and PySpark workflows.
• Design, develop, and maintain scalable ETL/ELT pipelines in Databricks.
• Support ingestion, transformation, validation, and reconciliation of enterprise data across legacy and modern platforms.
• Collaborate with architects, senior engineers, business analysts, and migration teams to understand source-to-target mapping requirements.
• Perform data quality checks, unit testing, integration testing, and defect resolution during migration execution.
• Optimize PySpark jobs, SQL queries, and Databricks notebooks for better performance and reliability.
• Document migration logic, transformation rules, dependencies, and technical design details.
• Participate in code reviews and follow Data Engineering best practices.

Key Skills:

• 5+ years of relevant Data Engineering experience, with the majority of career spent in Data Engineering roles.
• Strong hands-on experience in Databricks development.
• Strong PySpark programming experience for data transformation and pipeline development.
• Strong SQL/T-SQL understanding, especially the ability to read and interpret existing T-SQL code.
• Experience in legacy data technologies such as SQL Server, Oracle, traditional ETL tools, or Data Warehouse platforms.
• Experience in ETL/ELT development, data ingestion, transformation, and data pipeline implementation.
• Good understanding of Data Warehousing, Data Modeling, and dimensional concepts.
• Ability to migrate or re-engineer legacy SQL/ETL logic into Databricks/PySpark.
• Strong analytical, debugging, and problem-solving skills.
• Good communication skills to work with technical and business teams.

Preferred Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Azure Data Factory or Azure Data Platform exposure.
• Delta Lake experience.
• Unity Catalog exposure.
• Azure Synapse Analytics experience.
• CI/CD exposure for data pipelines.
• Experience working in Agile/Scrum delivery models.
• Data governance, security, and lineage awareness.
• Relevant certifications in Databricks, Azure Data Engineering, or Cloud Data Platforms will be an added advantage.
• Strong, hands-on Data Engineering background throughout career, not a partial or recent transition.
• Avoid profiles that are mainly BI Developers, Power BI Developers, Data Analysts, or reporting-only candidates.