Capgemini - Data Engineer - Generative AI/LLM Pipelines

Core Responsibilities :

- GenAI Data Pipelines : Build ingestion pipelines for unstructured data (documents, text, media) and optimize advanced chunking and embedding workflows.

- Vector Storage : Deploy and manage vector databases (Pinecone, Qdrant, Weaviate, pgvector) for similarity search and low-latency retrieval.

- RAG & Agent Support : Engineer context-retrieval pipelines and APIs to power RAG systems, model fine-tuning datasets, and AI agent workflows.

- Data Platform & Streaming : Design batch (dbt, Airflow) and streaming (Kafka, Spark) pipelines on cloud lakehouses (Databricks, Snowflake).

- Governance & Quality : Ensure PII masking, data lineage, and security compliance before data reaches AI models.

Key Qualifications :

- Experience : 5+ years in Data Engineering, with hands-on experience supporting GenAI/LLM pipelines or NLP workloads.

- Languages : Expert in Python and SQL.

- Vector & AI Tools : Experience with vector databases and AI orchestration frameworks (LlamaIndex, LangChain).

- Data Stack : Proficiency with cloud lakehouses (Databricks/Snowflake/BigQuery) and orchestration tools (dbt, Airflow).

- Engineering : Strong knowledge of Git, Docker, and CI/CD pipelines.

Technical Stack :

- Languages : Python, SQL

- Data Platforms : Snowflake, Databricks, PostgreSQL

- Vector Databases : Pinecone, Qdrant, Weaviate, pgvector

- Orchestration & Processing : dbt, Apache Airflow, Kafka, Spark

- AI Tooling : LlamaIndex, LangChain, Hugging Face