Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks
Developing scalable and re-usable frameworks for ingesting of data sets
Integrating the end to end data pipleline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times
Working with event based / streaming technologies to ingest and process data
Working with other members of the project team to support delivery of additional project components (API interfaces, Search)
Evaluating the performance and applicability of multiple tools against customer requirements
