Senior Data Engineering Lead
Posted on: 03/12/2025
Job Description
Description :
As the Data Engineering - Lead, you will be responsible for architecting, building, and scaling the core AI data infrastructure that fuels intelligent decisioning, personalization, and LLM-based use cases.
The core responsibilities for the job include the following :
Data Platform Architecture and Engineering :
- Lead the design and implementation of end-to-end data pipelines on the Databricks Lakehouse platform.
- Build metadata-driven ingestion and transformation frameworks for multimodal data.
- Enable real-time and batch data pipelines for downstream AI and analytics applications.
- Define and implement scalable data models, schemas, and lineage tracking mechanisms.
AI and Knowledge Graph Enablement:
- Architect knowledge graph pipelines for unified entity resolution and semantic enrichment.
- Partner with AI/ML engineers to design feature stores, embedding workflows, and reasoning layers.
- Integrate vector databases and graph systems (Neo4j, Neptune, etc.) to support intelligent retrieval and recommendations.
- Optimize data preparation and transformation workflows for LLM fine-tuning and inference.
Data Governance, Observability, and Reliability:
- Drive data quality, lineage, and cataloging standards.
- Implement automated validation and observability frameworks in data pipelines.
- Champion CI/CD automation and data versioning best practices using Terraform and GitHub Actions.
- Collaborate with cross-functional teams to enforce compliance, privacy, and data access controls.
Requirements :
- 9+ years of experience in data engineering, AI platform development, or enterprise data architecture.
- Expertise in Databricks, Delta Lake, PySpark, and distributed data processing.
- Advanced proficiency in Python, SQL, and Spark-based transformations.
- Experience with real-time streaming pipelines (Kafka, Debezium, etc.
- Hands-on with knowledge graphs, semantic data modeling, or graph-based analytics.
- Deep understanding of data governance, metadata management, and security frameworks.
- Ability to lead technical discussions across AI, engineering, and product teams.
Preferred Qualifications :
- Experience in AI-driven product organizations or Customer Data Platforms (CDP).
- Exposure to LLM data workflows, embeddings, or retrieval-augmented generation (RAG).
- Familiarity with AWS, GCP, or Azure cloud ecosystems.
- Knowledge of data observability platforms and impact analysis frameworks
Did you find something suspicious?
Posted By
12
Posted in
Data Engineering
Functional Area
Data Engineering
Job Code
1584513
