Data Integration Engineer

Mandatory Skills

Databricks & Python

Job Description

Core Technical Skills

Python - Advanced level (pandas, requests, boto3 libraries)

API Integration - REST APIs, authentication (OAuth, API keys), error handling

Cloud Platforms - AWS (S3, EC2, IAM) proficiency

Databricks/Spark - PySpark, Delta Lake, SQL optimization

Data Pipeline Tools - Airflow, Databricks Workflows, or equivalent orchestration

SQL - Complex queries, window functions, performance tuning

Security & Enterprise Knowledge

Understanding of security telemetry (logs, alerts, events, metrics)

Familiarity with at least 2-3 security platforms (JAMF, Intune, Qualys, CrowdStrike,

Okta)

Experience with AWS CloudTrail or similar cloud audit logs

Knowledge of data security and compliance (encryption, access controls)

Additional Competencies

Version control (Git)

CI/CD pipeline familiarity

Data quality and validation frameworks

Performance tuning and optimization mindset

Education Qualificaiton

BE, BTech, MCA

Roles & Responsibilities

Data Integration & Connectivity

Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys,

CrowdStrike, Okta, AWS CloudTrail) and Databricks

Establish secure API connections and authentication mechanisms with each source system

Implement data extraction logic using REST APIs, SDKs, and native connectors

Handle varying data formats and transformation requirements across heterogeneous sources

Data Pipeline Development

Develop automated Python scripts for continuous data ingestion

Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools

Implement error handling, retry mechanisms, and monitoring for pipeline reliability

Ensure near real-time or scheduled data synchronization based on requirements

AWS S3 & Data Staging

Configure S3 bucket architecture and data partitioning strategy

Land raw security telemetry data in S3 with proper folder structures

Implement lifecycle policies and data retention strategies

Ensure proper encryption and access controls

Databricks Platform Configuration

Create and optimize Databricks tables