Data Engineer - Azure Databricks

Desired Competencies (Technical/Behavioral Competency) :

Must-Have : ETL Knowledge, SQL Programming, Azure, Databricks

Good-to-Have : Python Automation

Responsibility of / Expectations from the Role :

- Should have experience in database/ETL concepts & testing

- Should have experience in understanding business requirements and translating them (writing) and resolving

complex sql queries.

- Should be able to write new code (SQL/java/python/excel macro) to compare two tables, create test scenario/ test data

- Experience working with Cloud (Azure,AWS)

- Experience in handling semi structure data like csv, json, text

- Ability to influence and negotiate with others to meet project deadlines without compromising quality

- Should be very strong in writing SQL queries using complex data types

- Should be able to create testcases, scenarios, test-plan, test scripts

- Should have basic knowledge in Selenium using TestNG or BDD (cucumber)

Responsibilities :

- Design, develop, and test ETL processes, data pipelines, and data solutions using Azure, Databricks, and SQL.

- Collaborate with cross-functional teams to understand business requirements and translate them into technical specifications.

- Develop and maintain complex SQL queries, data models, and data visualizations.

- Write new code (SQL, Java, Python, Excel macro) to compare two tables, create test scenarios, and test data.

- Work with semi-structured data (CSV, JSON, text) and cloud-based data platforms (Azure, AWS).

- Create test cases, scenarios, test plans, and test scripts.

- Participate in agile development methodologies and collaborate with team members to meet project

deadlines.

Requirements :

- 3+ years of experience in ETL development, with focus on Azure, Databricks, and SQL.

- Strong knowledge of ETL concepts, data warehousing, and data integration.

- Proficiency in SQL programming, including complex queries and data modeling.

- Experience with Azure, Databricks, and cloud-based data platforms.

- Ability to write clear and concise technical documents.