Posted 25 June, 2026
Senior Data Engineer - Teradata/Databricks & AI/ML
VARITE INDIA PRIVATE LIMITED
Bangalore, Karnataka, IN
Full Time
Reference: 26-35432-2522-2
Company Name: VARITE India Private Limited
About The Client:
A global IT services and consulting company, multinational information technology (IT), headquartered in Tokyo, Japan. The Client offers a wide array of IT services, including application development, infrastructure management, and business process outsourcing. Their consulting services span business and technology, while their digital solutions focus on transformation and user experience design. It excels in data and intelligence services, emphasizing analytics, AI, and machine learning. Additionally, their cybersecurity, cloud, and application services round out a comprehensive portfolio designed to meet the diverse needs of businesses worldwide.
About The Job:
Teradata DBA Skills
Databricks Engineering Skills
AI / ML Skills
Preferred
Unlock Rewards: Refer Candidates and Earn.
If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the referred candidate completes a three-month assignment with VARITE.
Experience Level Bonus Referral:
About VARITE: VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.
Equal Opportunity Employer:
VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.
About The Client:
A global IT services and consulting company, multinational information technology (IT), headquartered in Tokyo, Japan. The Client offers a wide array of IT services, including application development, infrastructure management, and business process outsourcing. Their consulting services span business and technology, while their digital solutions focus on transformation and user experience design. It excels in data and intelligence services, emphasizing analytics, AI, and machine learning. Additionally, their cybersecurity, cloud, and application services round out a comprehensive portfolio designed to meet the diverse needs of businesses worldwide.
About The Job:
- Client Americas is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.
- The ideal candidate combines deep Teradata DBA expertise with hands-on Databricks engineering capability and applied AI/ML skills. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.
- Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
- Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
- Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
- Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
- Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
- Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
- Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
- Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.
- Experience ( Relevant) : 10+ Years
- Shift Timings : 2pm to 11pm IST
- Please focus on the Databricks AI/ML skills and we can be flexible on the cost for the right resource.
Teradata DBA Skills
| Skill / Technology | Min. Experience | Relevance to Work Order |
|---|---|---|
| Teradata DBA / Administration | 7+ years | Primary platform under analysis; deep knowledge of system internals required |
| Teradata DBQL (Database Query Logging) | 5+ years | Core data source — SQL text, object usage, timestamps, user/app IDs, row counts, steps |
| Teradata System Views & Space Metadata | 5+ years | Object inventory, space analysis, last read/write tracking |
| Teradata SQL & Performance Tuning | 7+ years | Writing complex analytical queries across DBQL and system catalog tables |
| Teradata ResUsage / Workload Statistics | 3+ years | Quantifying CPU/IO cost impact for dataset removal/archival recommendations |
| Data Classification (hot/warm/cold) | 3+ years | Categorizing datasets for tiered storage management recommendations |
| ETL Pipeline Assessment (DataStage, Autosys) | 4+ years | Identifying and recommending decommission of stale ingestion pipelines |
| Teradata BTEQ / FastExport / TPT | 3+ years | Data extraction, log export, and metadata collection utilities |
Databricks Engineering Skills
| Skill / Technology | Min. Experience | Relevance to Work Order |
|---|---|---|
| Databricks Platform (Workspaces, Clusters, Jobs) | 4+ years | Core analysis and solution delivery platform for the engagement |
| Apache Spark (PySpark / Spark SQL) | 5+ years | Large-scale log ingestion, transformation, and analytical processing |
| Delta Lake | 3+ years | Building reliable, ACID-compliant data pipelines for log-driven analysis |
| Databricks Notebooks & Workflows | 3+ years | Developing repeatable, shareable analytical notebooks for customer handoff |
| Python (pandas, numpy, matplotlib) | 5+ years | Data wrangling, statistical analysis, and visualization of usage heatmaps |
| SQL (Spark SQL / ANSI SQL) | 7+ years | Analytical querying across large DBQL datasets in Databricks |
| Unity Catalog / Data Governance in Databricks | 2+ years | Organizing deliverable assets and maintaining data lineage within the solution |
| Dashboarding (Databricks SQL / Power BI) | 3+ years | Usage heatmaps, partition access patterns, and recommendation visualizations |
AI / ML Skills
| Skill / Technology | Min. Experience | Relevance to Work Order |
|---|---|---|
| Machine Learning (scikit-learn, MLflow) | 3+ years | Anomaly detection on access patterns, predictive cold-data classification |
| LLM / Generative AI Integration | 2+ years | LLM-assisted log interpretation, automated recommendation narrative generation |
| Feature Engineering on Time-Series Log Data | 3+ years | Extracting access frequency, recency, and seasonality signals from DBQL logs |
| Clustering & Classification Algorithms | 3+ years | Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers |
| MLflow / Experiment Tracking | 2+ years | Tracking model runs for reproducible analysis and customer handoff artifacts |
| Natural Language Processing (NLP) | 2+ years | Parsing and classifying SQL text from DBQL for workload pattern analysis |
| Data Visualization (Plotly, Matplotlib, Seaborn) | 3+ years | Generating usage heatmaps and partition access charts for stakeholder readouts |
Preferred
- Familiarity with DataStage orchestration log parsing
- Prior work on data platform cost optimization or cloud migration readiness assessments
- Experience working in healthcare payer data environments (HIPAA awareness)
- Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs)
Unlock Rewards: Refer Candidates and Earn.
If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the referred candidate completes a three-month assignment with VARITE.
Experience Level Bonus Referral:
| 0-2 years | INR 5,000 |
| 2-6 years | INR 7,500 |
| 6+ years | INR 10,000 |
About VARITE: VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.
Equal Opportunity Employer:
VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.
