Job Summary:

We are looking for an experienced Azure Data Engineer with strong hands-on expertise in designing and implementing cloud-based data solutions on Microsoft Azure. The ideal candidate should have experience with Azure data warehouses, NoSQL databases, hybrid data environments, Azure Data Factory, Databricks, and Azure streaming services.

Key Responsibilities:

- Design, develop, and implement scalable Azure cloud data solutions and data pipelines.

- Develop and maintain cloud-based data warehouses, Azure databases, NoSQL databases, and hybrid data solutions.

- Build robust ETL/ELT pipelines using Azure Data Factory (ADF).

- Develop and manage ADF components including Azure Functions, Logic Apps, Triggers, and Integration Runtime (IR).

- Design and implement data processing solutions using Azure Databricks.

- Develop data transformation and processing workflows using PySpark and Scala.

- Implement real-time and streaming data solutions using Azure Stream Analytics and Event Hubs.

- Work with HDInsight and other Azure data services for distributed data processing.

- Integrate data from multiple structured and unstructured sources into enterprise data platforms.

- Implement data quality, validation, monitoring, and error-handling mechanisms across data pipelines.

- Optimize data pipelines and processing workloads for performance, scalability, and cost efficiency.

- Collaborate with data architects, analysts, developers, and business stakeholders to understand data requirements.

- Troubleshoot data pipeline failures and production issues and implement appropriate solutions.

- Follow Azure security, governance, data management, and development best practices.

- Contribute to the design and implementation of modern cloud and hybrid data architectures.

Required Skills & Experience:

- 5+ years of hands-on experience in data engineering and cloud data platforms.

- Strong experience implementing Azure Cloud data warehouses.

- Hands-on experience with Azure databases, NoSQL databases, and hybrid data scenarios.

- Strong experience with Azure Data Factory (ADF).

- Good knowledge of Azure Functions, Logic Apps, Triggers, and Integration Runtime (IR).

- Strong hands-on experience with Azure Databricks.

- Experience with PySpark and/or Scala for large-scale data processing.

- Experience with Azure Stream Analytics and Event Hub.

- Knowledge of HDInsight and distributed data processing.

- Strong understanding of ETL/ELT processes, data integration, and data pipeline architecture.

- Good understanding of cloud data security, governance, monitoring, and performance optimization.