Azure data engineeer

Role Descriptions: Job descriptionData Product Factory is seeking a proactive| dynamic| and adaptable member to join our global| diverse and inclusive team. Due to our product continually growing and evolving| we look for team members who are open to change and respond well in ambiguous environments by demonstrating curiosity and proactiveness.Reporting into a Technical Lead| the new member will play a pivotal role in supporting the Data Ingestion team| Business and Technical Product Owners| to expand and promote our offering throughout EYs Service Lines| Products| Functions and engagement teams. This role covers a broad area of activity| including technical design and development of near real time data processing| consumer interaction| business documentation and reporting. Your key responsibilitiesYou will be responsible forCollaboratively identify and ideate opportunities to continuously improve Data Product Factorys (DPF) data asset and the services it provides to consumers.Engage in design and development sessions to further DPFs data asset creation and ingestion pipelines with a focus on stability| optimisation and traceability.Develop| test and maintain DPFs data pipelines processes.Developing and testing resilient| fault tolerant| modular code| to ensure DPFs processes| pipelines and services are robust and highly available.Research| explore and if feasible implement new methods and processes across DPF to build world class data products and associated services.Actively participate in cross-team collaboration to ensure smooth transition through Ingestion| Consumption and ActivationSkills and attributes for successAzure Data Factory (ADF) Expertise in Developing and managing ADF pipelines| activities| triggers| data flows Azure Databricks Proficient in using Databricks for data processing with notebooks| jobs| and clusters experience with Apache Spark. Programming Strong skills in Python| PySpark| and SQL for efficient data transformation and processing. ETL Data Integration Experience with ETL processes and data transformation in ADF and Databricks| integrating various data sources. Cloud Data Storage Familiarity with Azure storage solutions like Azure SQL Database| Blob Storage| and Data Lake Storage. Big Data Processing Experience with big data frameworks| specifically Apache Spark| for processing large datasets. Data Warehousing Modelling Understanding of data warehousing principles and scalable data modelling techniques. Good to have Experience in working with Open-source storage formats such as Apache Delta| Apache Parquet would be beneficial Awareness of compliance regulations relevant to data handling and processing. Knowledge of data security practices| including encryption| access control| and data governance. Knowledge of other Azure services that integrate with ADF and ADB| such as Azure Functions| Azure Logic Apps| and Azure Event Hubs. An understanding of Financial Statements and Business Ownership structures would
Essential Skills: Job descriptionData Product Factory is seeking a proactive| dynamic| and adaptable member to join our global| diverse and inclusive team. Due to our product continually growing and evolving| we look for team members who are open to change and respond well in ambiguous environments by demonstrating curiosity and proactiveness.Reporting into a Technical Lead| the new member will play a pivotal role in supporting the Data Ingestion team| Business and Technical Product Owners| to expand and promote our offering throughout EYs Service Lines| Products| Functions and engagement teams. This role covers a broad area of activity| including technical design and development of near real time data processing| consumer interaction| business documentation and reporting. Your key responsibilitiesYou will be responsible forCollaboratively identify and ideate opportunities to continuously improve Data Product Factorys (DPF) data asset and the services it provides to consumers.Engage in design and