Job Description: • Lead the end-to-end migration of data pipelines and workloads from Teradata to Databricks. • Analyze existing Teradata workloads to identify optimization opportunities and migration strategies. • Design and implement scalable, efficient, and cost-effective data pipelines using PySpark/Scala on Databricks Skills Required: • Strong experience with Databricks, Apache Spark, and Delta Lake. • Strong problem-solving and communication skills. • Proficiency in PySpark, SQL, and optionally Scala. • Strong understanding of Spark, Databricks performance tuning techniques • Understanding of Teradata utilities, BTEQ scripts, stored procedures • Experience with cloud platforms (Azure, AWS, or GCP), preferably Azure Databricks. • Familiarity with data warehousing concepts, data modeling, and big data best practices. • Experience with version control and CI/CD tools.
