We are looking for an experienced Data Engineer 2.0 with strong expertise in testing and validating modern data platforms. The role involves designing and executing comprehensive test strategies for real-time streaming systems, cloud-native data pipelines, and distributed data processing frameworks to ensure scalability, reliability, and data quality.

Key Responsibilities :

- Design and implement manual test strategies for real-time streaming use cases using Azure Service Bus, Azure Event Hubs, Kafka, and Azure Functions.

- Validate Spark Streaming applications, including unbounded data flows, streaming DataFrames, checkpoints, watermarking, and streaming joins.

- Develop and execute test plans for containerized microservices deployed on Kubernetes, ensuring scalability, resilience, and fault tolerance.

- Test and validate data ingestion and transformation workflows across open table formats such as Delta Lake, Apache Iceberg, and Apache Hudi.

- Ensure data accuracy, consistency, and reliability across distributed and real-time data pipelines.

- Collaborate with data engineers, platform teams, and product stakeholders to identify risks and improve overall data quality

Good to Have/Preferred Skills :

- Experience in monitoring and troubleshooting system performance using observability stacks like Prometheus, Grafana, and ELK.

- Exposure to functional and performance testing on analytical databases and query engines such as Trino, StarRocks, and ClickHouse.

- Experience in testing and validating data products built on Data Mesh architecture, ensuring domain-oriented data quality, governance, and ownership.

- Understanding of cloud-native and distributed system testing best practices.