Director - Data Engineering

y'Role Overview :

We are looking for a Director Data Engineering to lead the design, development, and scaling of modern data platforms powering our analytics and AI solutions.

Key Responsibilities :

Platform Architecture & Engineering :

- Design and build scalable data platforms leveraging Databricks Lakehouse Architecture and Snowflake.

- Implement Medallion Architecture (Bronze, Silver, Gold layers) to standardize enterprise data pipelines.

- Optimize Delta Lake tables for performance, scalability, and cost efficiency.

- Establish governance frameworks using Unity Catalog and enterprise data modeling practices.

- Drive modern cloud-native data architecture and engineering best practices.

Data Pipeline Development :

- Architect and manage end-to-end batch and near real-time data pipelines.

- Enable ingestion, transformation, and serving of large-scale datasets, including :

1. Healthcare claims (Medical & Pharmacy)

2. Benefits and wellness data

3. Employee and workforce datasets

- Ensure pipelines are scalable, maintainable, and aligned to downstream analytics and AI requirements.

AI-Ready Data Platforms :

- Build data platforms that support AI and Machine Learning workloads.

- Enable feature engineering, model-ready datasets, and AI data pipelines.

- Leverage capabilities such as :

1. Databricks ML

2. MLflow

3. Databricks Feature Store / Feature Engineering

4. Vector Search and AI-powered data workflows (preferred)

5. Snowflake Cortex AI or similar AI capabilities (preferred)

- Collaborate closely with Data Science teams to operationalize ML solutions.

DataOps & Reliability :

- Implement CI/CD practices for data engineering workflows.

- Establish monitoring, observability, logging, and data quality frameworks.

- Drive high standards of reliability, governance, security, and operational excellence.

Collaboration & Solutioning :

- Partner with Product, Analytics, Consulting, and Client teams to deliver scalable data solutions.

- Participate in architecture discussions and solution design for client engagements.

- Support client implementations and technical solutioning.

Practice Development & Go-to-Market :

- Contribute to building and scaling company's Data Engineering / Databricks Services practice.

- Support pre-sales activities including technical presentations, solution architecture discussions, effort estimation, and proposal development.

- Participate in customer meetings, discovery workshops, solution demonstrations, and technical consulting.

- Help define reusable accelerators, frameworks, and best practices for enterprise data engineering engagements.

Team Leadership :

- Lead and mentor a team of Data Engineers, fostering technical excellence and accountability.

- Conduct architecture reviews, code reviews, and establish engineering best practices.

- Build a culture of ownership, continuous learning, innovation, and execution excellence.

Key Requirements :

Experience :

- 11 to 15 years of experience in Data Engineering, Data Platform Development, or Cloud Data Engineering.

- Proven experience leading engineering teams and delivering scalable enterprise data platforms in production environments.

- Experience working with enterprise analytics, AI/ML, or SaaS products is highly desirable.

- Experience in taking Data Engineering and/or Databricks-based solutions to market is preferred, including customer-facing solutioning, technical consulting, pre-sales support, architecture discussions, and client presentations.

Technical Expertise :

Mandatory :

- Strong hands-on expertise in Databricks (highest priority).

- Apache Spark / PySpark.

- Delta Lake.

- Databricks Lakehouse Platform.

- Unity Catalog.

- Cloud platforms (AWS, Azure, or GCP).

Preferred :

- Snowflake (strongly preferred alongside Databricks).

- Data Lake / Lakehouse architectures.

- Workflow orchestration tools (Airflow or similar).

- Infrastructure as Code and CI/CD pipelines.

AI & Modern Data Platform Capabilities :

Experience with one or more of the following is preferred :

- Databricks ML.

- MLflow.

- Feature Engineering / Feature Store.

- Databricks Mosaic AI.

- Vector Search / Retrieval-Augmented Generation (RAG) architectures.

- Snowflake Cortex AI.

- AI-enabled data pipelines and model operationalization.

- Integration with Large Language Models (LLMs) and Generative AI applications.

Core Capabilities :

- Strong problem-solving and structured thinking.

- Ability to translate business requirements into scalable technical architectures.

- Strong stakeholder management and executive communication skills.

- Experience working directly with global clients.

- High ownership with a bias for execution.

- Ability to balance technical depth with business impact.

What Success Looks Like :

- Scalable, secure, and high-performance data platforms supporting analytics and AI solutions.

- Reliable, well-governed data pipelines with strong quality and observability standards.

- A high-performing and accountable data engineering team.

- Successful enablement of analytics, AI, and Machine Learning use cases.

- Strong contribution to the growth of company's Data Engineering and Databricks practice through customer engagement and solution leadership.

Why Join Us :

- Opportunity to build and scale modern enterprise data platforms using leading cloud technologies.

- Work with complex, real-world datasets at global scale.

- High ownership and visibility in a fast-growing, AI-first, product-led organization.

- Opportunity to shape our Data Engineering practice and client offerings.

- Collaborative culture focused on innovation, engineering excellence, and solving meaningful business problems.

Nice to Have :

- Experience with healthcare, HR, Total Rewards, or employee benefits datasets.

- Exposure to analytics products or SaaS platforms.

- Databricks and/or Snowflake certifications.

- Experience with Generative AI, LLM applications, or AI-powered data engineering solutions.