Data Engineer - Azure Databricks

We are looking for a Data Engineer with hands-on experience in Azure Data Factory, Azure Databricks, PySpark, and .NET Framework to design and maintain efficient, scalable data pipelines and analytics platforms.

The ideal candidate will be responsible for application integration, workflow automation, and leveraging Azure Synapse for advanced data warehousing and analytics.

Experience with machine learning techniques and production support will be an added advantage.

Key Responsibilities :

- Develop and maintain data pipelines using Azure Data Factory (ADF) and Azure Databricks (PySpark) to support real-time and batch data processing.

- Use Azure Synapse Analytics for data warehousing, reporting, and big data analytics.

- Apply machine learning models and frameworks to drive predictive insights and automation.

- Build and consume APIs and data services using the .NET Framework, integrating them into the data ecosystem.

- Automate and orchestrate data workflows for performance, reliability, and scalability.

- Ensure high data quality, consistency, and integrity across multiple data sources and platforms.

- Provide BAU (Business As Usual) support : monitor systems, resolve incidents, and ensure SLAs are met.

- Collaborate with data analysts, architects, business stakeholders, and cross-functional teams to translate requirements into technical solutions.

- Maintain technical documentation including data flows, architecture diagrams, pipeline logic, and operational runbooks.

- Stay current on emerging Azure and big data technologies, best practices, and trends.

Required Skills & Qualifications :

- Strong experience in Azure Data Factory (ADF) and Azure Databricks with PySpark.

- Solid understanding of Azure Synapse Analytics, data lake architecture, and structured/unstructured data handling.

- Programming experience with Python (for PySpark) and C#/.NET Framework for integration/API development.

- Experience with data orchestration, automation, and ETL/ELT design.

- Familiarity with machine learning concepts, models, and libraries (e., Scikit-learn, MLflow)

- Experience with CI/CD pipelines, version control (Git), and DevOps practices.

- Understanding of data governance, security, and compliance on cloud platforms.

Preferred Qualifications :

- Microsoft Azure certification

- Experience with Power BI or other reporting tools.

- Familiarity with Kafka/Event Hub for streaming data.

- Exposure to other cloud platforms like AWS/GCP is a plus.

Educational Qualification :

- Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent work experience)