Databricks, Apache Spark

Job Description:

• Lead the end-to-end migration of data pipelines and workloads from Teradata to Databricks.
• Analyze existing Teradata workloads to identify optimization opportunities and migration strategies.
• Design and implement scalable, efficient, and cost-effective data pipelines using PySpark/Scala on Databricks

Skills Required:

• Strong experience with Databricks, Apache Spark, and Delta Lake.
• Proficiency in PySpark, SQL, and optionally Scala.
• Strong understanding of Spark, Databricks performance tuning techniques
• Understanding of Teradata utilities, BTEQ scripts, stored procedures
• Experience with cloud platforms (Azure, AWS, or GCP), preferably Azure Databricks.
• Familiarity with data modeling, data warehousing concepts, and big data best practices.
• Strong problem-solving and communication skills.