EXL - Data Engineer - Azure Data Services

Role Summary :

We are looking for a skilled Data Engineer with solid experience in the Azure data ecosystem and hands-on expertise in Azure Databricks. The ideal candidate will have worked on designing and developing scalable, cloud-native data pipelines and solutions using Azure data services. You will contribute to building and optimizing data lakehouse and warehouse architectures, enabling efficient data ingestion, transformation, and analytics in a collaborative team environment.

Roles and Responsibilities :

- Design, develop, and maintain scalable data pipelines and workflows using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Storage (ADLS).

- Implement ETL/ELT processes using Azure Data Factory and Databricks to enable efficient data ingestion and transformation.

- Work on building data solutions leveraging Delta Lake and implement best practices such as the Medallion Architecture.

- Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver effective Azure-based data solutions.

- Participate in code reviews, testing, deployment, and documentation to ensure high-quality data engineering deliverables.

- Follow data governance, security, and compliance standards while handling data pipelines and sensitive information.

- Implement and manage secure authentication and authorization using Azure Active Directory (AAD) and Azure Key Vault for secrets management.

- Support CI/CD automation and monitoring of data pipelines using Azure DevOps.

- Stay current with Azure data platform developments and contribute ideas to improve existing data infrastructure and processes.

Must-Have Skills :

- 4+ years of experience in data engineering with a strong focus on Azure data services.

- Hands-on experience with Azure Databricks, Spark, PySpark, Delta Lake, and implementing Medallion Architecture is highly desirable.

- Proficiency with Azure Data Factory for building ETL/ELT pipelines.

- Working knowledge of Azure Synapse Analytics and Azure Data Lake Storage (ADLS).

- Strong programming skills in Python and SQL (T-SQL). PowerShell experience is a plus.

- Experience with Azure Active Directory (AAD) for authentication and role-based access control.

- Familiarity with Azure Key Vault for secrets and certificate management.

- Understanding of modern data lakehouse architectures and cloud-native data platform principles on Azure.

- Basic knowledge of CI/CD and DevOps processes, preferably using Azure DevOps.

- Good communication skills with the ability to work collaboratively in cross-functional teams.

Good-to-Have Skills :

- Experience with Delta Live Tables, MLflow, or Unity Catalog on Azure Databricks.

- Familiarity with Azure Event Hubs, Azure Stream Analytics, or other streaming data services.

- Awareness of Azure Purview or other data governance tools.

- Basic exposure to Snowflake in an Azure environment, including data modeling and performance tuning.

- Experience with Infrastructure as Code (IaC) tools like ARM templates or Terraform on Azure.

- Understanding of machine learning workflows and MLOps integration within Azure.