Big Data Testing Engineer - Java/Python

About the Role :

We are looking for a highly skilled and detail-oriented Big Data Testing Engineer to join our QA team. The ideal candidate will have strong hands-on experience in advanced SQL, ETL validation, and Big Data testing on platforms such as Azure and Cloudera. You will play a critical role in ensuring the quality and accuracy of large-scale data pipelines and analytics systems.

Youll work closely with data engineers, developers, and business analysts to validate data ingestion, transformation, and storage processes in complex Big Data environments. A good understanding of distributed data systems and excellent scripting and troubleshooting skills are essential for this role.

Key Responsibilities :

- Perform data validation, reconciliation, and transformation testing on large datasets across various data sources and targets.

- Work with data ingestion and processing frameworks such as Apache Spark, Hive, Pig, Sqoop, etc.

- Ensure data integrity, quality, and completeness across Hadoop Distributed File Systems (HDFS) or similar architectures.

- Write complex SQL queries to validate and reconcile large volumes of data.

- Validate data transformations and aggregations from source to target in data lakes or data warehouses.

- Compare outputs between legacy and modern systems as part of migration testing.

- Perform testing in cloud-based Big Data platforms like Microsoft Azure and Cloudera.

- Validate data integration across platforms like Azure Data Factory, Synapse Analytics, or HDInsight.

- Test data ingestion from cloud storage systems like Azure Blob, Data Lake, etc.

- Leverage scripting languages like Python, Java, or Scala to automate repetitive testing tasks or

validations.

- Develop reusable scripts for file comparison, log validations, and batch job verifications.

- Write and execute Unix/Linux shell scripts for job monitoring, file movement, and log analysis.

- Collaborate with cross-functional teams in Agile environments using tools like JIRA for sprint planning and defect tracking.

- Participate in daily stand-ups, retrospectives, and testing-related meetings to ensure alignment with

development and business teams.

- Document defects clearly and work with developers to ensure prompt resolution.

Required Skills & Qualifications :

- Strong working knowledge of SQL and at least one relational database such as Oracle, Netezza, or SQL

Server.

- Solid understanding and testing experience with Big Data tools and query languages (HiveQL, Pig, Spark SQL).

- Hands-on experience in at least one cloud platform (preferably Azure) and Big Data ecosystem tools (Cloudera, Hadoop).

- Proficiency in one or more scripting languages: Python, Java, or Scala.

- Familiarity with Unix/Linux shell scripting.

- Strong knowledge of Agile development methodologies and tools like JIRA and Confluence.

- Excellent analytical, problem-solving, and communication skills.