- Role: Data Engineer (AI Knowledge Base Platform)

- Experience: 4 years to 7 years

- Work location: Anywhere in India (Remote)

- Notice period: Immediate to 15 days

- Shift timing: 3:00 PM to 12:00 AM (IST)

We are seeking a Data Engineer to design and maintain the ingestion, processing, and transformation pipelines that feed AI knowledge bases on GCP.

Required Skills & Qualifications :

- 4+ years of Data engineering experience

- Strong programming skills in Python and SQL

- Proficiency with GCP data services: BigQuery, Cloud Storage, Dataflow, Pub/Sub, Composer/Airflow

- Experience with ETL/ELT pipeline design and workflow orchestration

- Familiarity with vector databases/embeddings (e.g., Pinecone, FAISS, Vertex AI Matching Engine)

- Knowledge of data modeling, governance, and version control

- Experience with CI/CD, Docker, and Kubernetes for data pipeline deployment

- Strong analytical and debugging skills

Key Responsibilities :

- Design and implement data ingestion pipelines from GCS, BigQuery, and Google Drive

- Build workflows for knowledge base indexing/vectorization (RAG pipelines)

- Implement incremental updates, targeted revisions, and full rebuild options for AI agents

- Collaborate with ML Engineers to integrate data pipelines with Vertex AI models

- Optimize pipelines for cost, performance, and scalability within GCP

- Ensure data quality, governance, and security with IAM-based access control

- Maintain logging, monitoring, and alerting for pipeline health

- Work with DevOps to enable self-healing and rollback automation