Description
• Design and build robust, scalable ETL/ELT pipelines using PySpark to ingest data from diverse sources (databases, logs, APIs, files).
• Transform and curate raw transactional and log data into analysis-ready datasets in the Data Hub and analytical data marts.
• Develop reusable and parameterized Spark jobs for batch and micro-batch processing.
• Optimize performance and scalability of PySpark jobs across large data volumes.
• Ensure data quality, consistency, lineage, and proper documentation across ingestion flows.
• Collaborate with Data Architects, Modelers, and Data Scientists to implement ingestion logic aligned with business needs.
• Work with cloud-based data platforms (e.g., AWS S3, Glue, EMR, Redshift) for data movement and storage.
• Support version control, CI/CD, and infrastructure-as-code where applicable.
• Participate in Agile ceremonies and contribute to sprint planning, story grooming, and demos.
|
