Job Title : AI Data Engineer

Location : Hyderabad ,Rai Durg

Work Mode : Hybrid /3 Days Work From Office

Experience : 3 to 8 Years

Job Overview :

We are looking for a highly skilled and hands-on AI Data Engineer to build and scale high-performance data infrastructure for autonomous AI agents. This role goes beyond traditional data engineeringfocusing on dynamic context systems, real-time semantic indexing, and intelligent data pipelines that power next-generation AI systems.

Must Have :

- AI Modules

- Vector database

- HNSW Indexes

- Metadata

- Scalar Quantization

- Python

- Apache

- Spark

- Flint

- Kafka

- llamaIndex

- langchain

- data version control

- Airflow

Key Responsibilities :

- Vector & Graph ETL : Design and maintain pipelines that transform unstructured data (PDFs, emails, logs, chats) into optimized embeddings for Vector Databases (Pinecone, Weaviate, Milvus).

- Semantic Data Modeling : Engineer data structures that optimize for Retrieval-Augmented Generation (RAG), ensuring agents find the "needle in the haystack" in milliseconds.

- Knowledge Graph Construction : Build and scale Knowledge Graphs (Neo4j) to represent complex relationships in our trading and support data that standard vector search misses.

- Automated Data Labeling & Synthetic Data : Implement pipelines using LLMs to auto-label datasets or generate synthetic edge cases for agent training and evaluation.

- Stream Processing for Agents : Build real-time data "listeners" (Kafka/Flink) that feed live context to agents, allowing them to react to market or support events as they happen.

- Data Reliability & "Drift" Detection : Build monitoring for "Embedding Drift", identifying when the statistical distribution of your data changes and the agent's "knowledge" becomes stale.

Preferred Skills :

- Proficiency in Rust or C++ for high-performance systems

- Experience with semantic chunking and parent-child retrieval

- Knowledge of cold/warm/hot storage strategies

- Experience building PII redaction pipelines

- Understanding of knowledge graphs and graph databas

Required Qualification :

- Bachelor of Engineering

- Bachelor of Technology (B.E./B.Tech.)