Work Locations : Any NTT location

Experience (Relevant) : 10+ Years

Shift Timings : 2pm to 11pm IST

Hybrid : 3 days/ week

Start date : ASAP

Position Overview :

NTT DATA Americas is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.

The ideal candidate combines deep Teradata DBA expertise with hands-on Databricks engineering capability and applied AI/ML skills. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.

Key Responsibilities:

- Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.

- Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.

- Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.

- Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.

- Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.

- Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.

- Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.

- Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.

Required Skills & Experience :

Teradata DBA Skills :

- Teradata DBA / Administration : Primary platform under analysis; deep knowledge of system internals required.

- Teradata DBQL (Database Query Logging) : Core data source - SQL text, object usage, timestamps, user/app IDs, row counts, steps.

- Teradata System Views & Space Metadata : 5+ years - Object inventory, space analysis, last read/write tracking.

- Teradata SQL & Performance Tuning : Writing complex analytical queries across DBQL and system catalog tables.

- Teradata ResUsage / Workload Statistics : Quantifying CPU/IO cost impact for dataset removal/archival recommendations.

- Data Classification (hot/warm/cold) : 3+ years - Categorizing datasets for tiered storage management recommendations.

- ETL Pipeline Assessment (DataStage, Autosys) : 4+ years - Identifying and recommending decommission of stale ingestion pipelines.

- Teradata BTEQ / FastExport / TPT : 3+ years - Data extraction, log export, and metadata collection utilities.

Databricks Engineering Skills :

- Databricks Platform (Workspaces, Clusters, Jobs) : Core analysis and solution delivery platform for the engagement.

- Apache Spark (PySpark / Spark SQL) : Large-scale log ingestion, transformation, and analytical processing.

- Delta Lake : Building reliable, ACID-compliant data pipelines for log-driven analysis.

- Databricks Notebooks & Workflows : Developing repeatable, shareable analytical notebooks for customer handoff.

- Python (pandas, numpy, matplotlib) : Data wrangling, statistical analysis, and visualization of usage heatmaps.

- SQL (Spark SQL / ANSI SQL) : Analytical querying across large DBQL datasets in Databricks.

- Unity Catalog / Data Governance in Databricks : Organizing deliverable assets and maintaining data lineage within the solution.

- Dashboarding (Databricks SQL / Power BI) : Usage heatmaps, partition access patterns, and recommendation visualizations.

AI / ML Skills :

- Machine Learning (scikit-learn, MLflow) : Anomaly detection on access patterns, predictive cold-data classification.

- LLM / Generative AI Integration : LLM-assisted log interpretation, automated recommendation narrative generation.

- Feature Engineering on Time-Series Log Data : 3+ years - Extracting access frequency, recency, and seasonality signals from DBQL logs.

- Clustering & Classification Algorithms : 3+ years - Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers.

- MLflow / Experiment Tracking : 2+ years - Tracking model runs for reproducible analysis and customer handoff artifacts.

- Natural Language Processing (NLP) : Parsing and classifying SQL text from DBQL for workload pattern analysis.

- Data Visualization (Plotly, Matplotlib, Seaborn) : Generating usage heatmaps and partition access charts for stakeholder readouts.

Preferred Qualifications :

- Familiarity with DataStage orchestration log parsing.

- Prior work on data platform cost optimization or cloud migration readiness assessments.

- Experience working in healthcare payer data environments (HIPAA awareness).

- Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs).