Data Engineer - Validation & Quality

Description :

Responsibilities :

Validation Framework Development :

- Build scalable rule-based, statistical, and pattern-driven data validation frameworks using Pandas, NumPy, and distributed processing tools.

- Develop mechanisms for contradiction detection, record reconciliation, drift analysis, and anomaly flagging across structured and semi-structured datasets.

- Design and compute confidence scores/metrics for each evidence record to quantify trust and support downstream decision-making.

Data Quality Automation :

- Automate schema compliance checks, sampling workflows, checksums, and cross-source consistency validation to ensure complete dataset integrity.

- Integrate validation at ingestion, transformation, and output stages to enforce continuous quality gates throughout the data lifecycle.

- Build reusable validation pipelines that can scale across new domains, data types, and ingestion patterns.

Platform Integration :

- Work closely with the Kernel / core platform engineering team to embed validation results, flags, and confidence metadata into every output artifact.

- Contribute to monitoring dashboards, alerting rules, and automated quality reporting to provide operational transparency.

- Optimize validation workflows for performance, parallelism, and cost efficiency.

Requirements :

Experience :

- 5+ years in data engineering, data validation/quality engineering, or MLOps-focused quality roles.

- Strong SQL proficiency with experience in query optimization, indexing, and high-throughput ETL pipelines.

- Familiarity with data lineage, DQ frameworks, and compliance/regulatory standards such as SOC 2, GDPR, or equivalent.

Skills & Attributes :

- Deep understanding of data profiling, statistical checks, and anomaly detection techniques.

- Ability to reason about data correctness, consistency, and reproducibility in complex systems.

- Strong Python engineering background with the ability to implement efficient validation logic.

- Excellent communication skills and comfort collaborating with cross-functional platform teams.