Job Description :

Senior Data Engineer with a deep focus on data quality, validation frameworks, and reliability engineering. This role will be instrumental in ensuring the accuracy, integrity, and trustworthiness of data assets across our cloud-native infrastructure. The ideal candidate combines expert-level Python programming with practical experience in data pipeline engineering, API integration, and managing cloud-native workloads on AWS and Kubernetes.

Roles & Responsibilities :

- Design, develop, and deploy automated data validation and quality frameworks using Python.

- Build scalable and fault-tolerant data pipelines that support quality checks across data ingestion, transformation, and delivery.

- Integrate with REST APIs to validate and enrich datasets across distributed systems.

- Deploy and manage validation workflows using AWS services (EKS, EMR, EC2) and Kubernetes clusters.

- Collaborate with data engineers, analysts, and DevOps to embed quality checks into CI/CD and ETL pipelines.

- Develop monitoring and alerting systems for real-time detection of data anomalies and inconsistencies.

- Write clean, modular, and reusable Python code for automated testing, validation, and reporting.

- Lead root cause analysis for data quality incidents and design long-term solutions.

- Maintain detailed technical documentation of data validation strategies, test cases, and architecture.

- Promote data quality best practices and evangelize a culture of data reliability within the engineering teams.

Required Skills :

- Experience with data quality platforms such as Great Expectations, Collibra Data Quality, or similar tools.

- Proficiency in Docker and container lifecycle management.

- Familiarity with serverless compute environments (e.g., AWS Lambda, Azure Functions), Python, PySpark

- Relevant certifications in AWS, Kubernetes, or data quality technologies.

- Prior experience working in big data ecosystems and real-time data environments.