About the Role :

We are seeking a highly skilled and motivated Senior Data Engineer with proven experience in Azure Databricks, Azure Data Factory (ADF), and Scala. The ideal candidate will have a strong background in building scalable data processing pipelines and implementing big data solutions on Microsoft Azure. You will work closely with cross-functional teams to deliver reliable, performant, and secure data solutions.

Key Responsibilities :

- Design and develop scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Databricks.

- Write optimized Spark applications in Scala to perform complex data transformations and aggregations.

- Develop data ingestion and transformation processes using Delta Lake and Azure Data Lake Storage (ADLS).

- Collaborate with data architects, analysts, and business teams to understand data requirements and translate them into technical solutions.

- Optimize Spark jobs for performance and cost-efficiency on the Azure platform.

- Implement data quality checks, monitoring, logging, and error handling for robust data pipelines.

- Develop CI/CD pipelines using Azure DevOps or GitHub Actions for deploying and maintaining data solutions.

- Ensure security and compliance of data pipelines by following cloud data governance best practices.

- Participate in code reviews, knowledge sharing, and mentoring junior developers.

Required Skills & Experience :

- 6+ years of overall experience in Data Engineering or Big Data Development.

- Minimum 4 years of hands-on experience with Azure Databricks and ADF.

- Strong programming skills in Scala and experience with Apache Spark.

- Solid understanding of distributed data processing, partitioning, and performance tuning in Spark.

- Experience with Azure services : Azure Data Lake, Azure Blob Storage, Delta Lake.

- Experience in version control systems (Git) and CI/CD processes.

- Excellent problem-solving skills and the ability to work independently and in a team environment.

Preferred / Good to Have :

- Knowledge of Azure Synapse Analytics, Azure Functions, or Event Hubs.

- Exposure to PySpark, Python, or SQL.

- Familiarity with data security, encryption, and access control mechanisms in Azure.

- Experience with Agile methodologies and tools like JIRA or Azure Boards.