Sr. Data Engineers - Databricks Pipeline Developer
Role Overview
Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.
Key Responsibilities
- Develop PySpark pipelines for
- API ingestion
- FTP file ingestion
- Data cleansing
- Enrichment transformations
- Implement incremental & batch loads
- Develop Bronze Silver Gold transformations
- Translate legacy SQL stored procedures into Spark jobs
- Optimize Delta Lake tables
- Implement validation checks
- Support testing & reconciliation
Required Skills
- Strong PySpark / Spark SQL
- Strong SQL (read & interpret legacy procedures)
- Experience with API integration
- Experience handling flat files (FTP ingestion)
- Experience with Delta Lake
- Knowledge of partitioning & optimization
