Big Rattle Technologies - QA Engineer - Machine Learning Systems

Position : QA Engineer - Machine Learning Systems (5 - 7 years)

Location : Remote (Company in Mumbai)

Immediate Joiners only.

Summary :

The QA Engineer will own quality assurance across the ML lifecyclefrom raw data validation through feature engineering checks, model training/evaluation verification, batch prediction/optimization validation, and end-to-end (E2E) workflow testing.

The role is hands-on with Python automation, data profiling, and pipeline test harnesses in Azure ML and Azure DevOps.

Success means probably correct data, models, and outputs at production scale and cadence.

Key Responsibilities :

Test Strategy & Governance :

- Define an ML-specific Test Strategy covering data quality KPIs, feature consistency checks, model acceptance gates (metrics + guardrails), and E2E run acceptance (timeliness, completeness, integrity).

- Establish versioned test datasets & golden baselines for repeatable regression of features, models, and optimizers.

Data Quality & Transformation :

- Validate raw data extracts and landed data lake data: schema/contract checks, null/outlier thresholds, time window completeness, duplicate detection, site/material coverage.

- Validate transformed/feature datasets: deterministic feature generation, leakage detection, drift vs. historical distributions, feature parity across runs (hash or statistical similarity tests).

- Implement automated data quality checks (e.g., Great Expectations/pytest + Pandas/SQL) executed in CI and AML pipelines.

Model Training & Evaluation :

- Verify training inputs (splits, windowing, target leakage prevention) and hyperparameter configs per site/cluster.

- Automate metric verification (e.g., MAPE/MAE/RMSE, uplift vs. last model, stability tests) with acceptance thresholds and champion/challenger logic.

- Validate feature importance stability and sensitivity/elasticity sanity checks (price/volume monotonicity where applicable).

- Gate model registration/promotion in AML based on signed test artifacts and reproducible metrics.

Predictions, Optimization & Guardrails :

- Validate batch predictions : result shapes, coverage, latency, and failure handling.

- Test model optimization outputs and enforced guardrails : detect violations and prove idempotent writes to DB.

- Verify API push to third party system (idempotency keys, retry/backoff, delivery receipts).

Pipelines & E2E :

- Build pipeline test harnesses for AML pipelines (data-gen nightly, training weekly, prediction/optimization) including orchestrated synthetic runs and fault injection (missing slice, late competitor data, SB backlog).

- Run E2E tests from raw data store - ADLS - AML - RDBMS - APIM/Frontend; assert freshness SLOs and audit event completeness (Event Hubs -> ADLS immutable).

Automation & Tooling :

- Develop Python-based automated tests (pytest) for data checks, model metrics, and API contracts; integrate with Azure DevOps (pipelines, badges, gates).

- Implement data-driven test runners (parameterized by site/material/model-version) and store signed test artifacts alongside models in AML Registry.

- Create synthetic test data generators and golden fixtures to cover edge cases (price gaps, competitor shocks, cold starts).

Reporting & Quality Ops :

- Publish weekly test reports and go/no-go recommendations for promotions; maintain a defect taxonomy (data vs. model vs. serving vs. optimization).

- Contribute to SLI/SLO dashboards (prediction timeliness, queue/DLQ, push success, data drift) used for release gates.

Required Skills (hands-on experience in the following) :

- Python automation (pytest, pandas, NumPy), SQL (PostgreSQL/Snowflake), and CI/CD (Azure DevOps) for fully automated ML QA.

- Strong grasp of ML validation : leakage checks, proper splits, metric selection (MAE/MAPE/RMSE), drift detection, sensitivity/elasticity sanity checks.

- Experience testing AML pipelines (pipelines/jobs/components), and message-driven integrations (Service Bus/Event Hubs).

- API test skills (FastAPI/OpenAPI, contract tests, Postman/pytest-httpx) + idempotency and retry patterns.

- Familiar with feature stores/feature engineering concepts and reproducibility.

- Solid understanding of observability (App Insights/Log Analytics) and auditability requirements.

Required Qualifications :

- Bachelors or Masters degree in Computer Science, Information Technology, or related field.

- 5- 7 years in QA with 3+ years focused on ML/Data systems (data pipelines + model validation).

- Certification in Azure Data or ML Engineer Associate is a plus