Principal Data Engineer - Spark/Data Pipeline

Responsibilities :

- Define and lead the data architecture vision and strategy, ensuring it supports analytics, ML, and business operations at scale.

- Architect and manage cloud-native data platforms using Databricks and AWS, leveraging the lakehouse architecture to unify data engineering and ML workflows.

- Build and optimize large-scale batch and streaming pipelines using Apache Spark, Airflow, and AWS Glue, ensuring high availability and fault tolerance.

- Design and develop data marts, warehouses, and analytics-ready datasets tailored for BI, product, and data science teams.

- Implement robust ETL/ELT pipelines with a focus on reusability, modularity, and automated testing.

- Enforce and scale data governance practices, including data lineage, cataloging, access management, and compliance with security and privacy standards.

- Partner with ML Engineers and Data Scientists to build and deploy ML pipelines, leveraging Databricks MLflow, Feature Store, and MLOps practices.

- Provide architectural leadership across data modeling, data observability, pipeline monitoring, and CI/CD for data workflows.

- Evaluate emerging tools and frameworks, recommending technologies that align with platform scalability and cost-efficiency.

- Mentor data engineers and foster a culture of technical excellence, innovation, and ownership across data teams.

Requirements :

- 8+ years of hands-on experience in data engineering, with at least 4 years in a lead or architect-level role.

- Deep expertise in Apache Spark, with proven experience developing large-scale distributed data processing pipelines.

- Strong experience with Databricks platform and its internal ecosystem (e. g., Delta Lake, Unity Catalog, MLflow, Job orchestration, Workspaces, Clusters, Lakehouse architecture).

- Extensive experience with workflow orchestration using Apache Airflow.

- Proficiency in both SQL and NoSQL databases (e. g., Postgres, DynamoDB, MongoDB, Cassandra) with a deep understanding of schema design, query tuning, and data partitioning.

- Proven background in building data warehouse/data mart architectures using AWS services like Redshift, Athena, Glue, Lambda, DMS, and S3

- Strong programming and scripting ability in Python (preferred) or other AWS-compatible languages.

- Solid understanding of data modeling techniques, versioned datasets, and performance tuning strategies.

- Hands-on experience implementing data governance, lineage tracking, data cataloging, and compliance frameworks (GDPR, HIPAA, etc. ).

- Experience with real-time data streaming using tools like Kafka, Kinesis, or Flink.

- Working knowledge of MLOps tooling and workflows, including automated model deployment, monitoring, and ML pipeline orchestration.

- Familiarity with MLflow, Feature Store, and Databricks-native ML tooling is a plus.

- Strong grasp of CI/CD for data and ML pipelines, automated testing, and infrastructure-as-code (Terraform, CDK, etc. ).

- Excellent communication, leadership, and mentoring skills with a collaborative mindset and the ability to influence across functions.