Job Summary :

We are looking for a highly experienced Senior Azure Data Engineer to design, build, and manage scalable cloud-based data platforms. The role requires strong expertise in Azure Databricks, Python, SQL, ETL pipelines, Azure Data Factory, and modern data lake and data warehouse architectures.

Key Responsibilities :

- Design and develop end-to-end ETL/ELT pipelines using Azure Data Factory and Databricks.

- Build and optimize data processing workflows using Python, PySpark, and SQL.

- Implement Azure Databricks notebooks and jobs for large-scale data processing.

- Manage data ingestion, transformation, and migration using Azure Data Migration tools.

- Design and maintain Data Warehousing and Data Lake architectures on Azure.

- Work with Delta Lake for scalable, reliable, and ACID-compliant data pipelines.

- Handle structured, semi-structured, and unstructured data using big data file formats (Parquet, Avro, ORC, JSON).

- Develop and monitor pipeline orchestration and scheduling.

- Ensure data quality, security, performance tuning, and cost optimization.

- Collaborate with architects, analysts, and business teams for data integration and analytics needs.

- Mentor junior engineers and lead best practices across data engineering initiatives.

Required Skills & Qualifications :

- 8 to 15 years of experience in Data Engineering or related roles.

- Strong hands-on experience with Azure Databricks and Apache Spark.

- Advanced proficiency in Python and SQL.

- Extensive experience in building ETL pipelines using Azure Data Factory (ADF).

- Hands-on experience with Azure Data Migration Tool / Services.

- Solid knowledge of Data Warehousing concepts (fact/dimension modeling).

- Strong understanding of Data Lakes, Delta Lake, and modern data architectures.

- Experience with cloud-based data engineering solutions on Azure.

- Knowledge of big data file formats (Parquet, Avro, ORC).

- Experience with pipeline development, orchestration, and monitoring.

Good to Have :

- Experience with CI/CD pipelines for data engineering workflows.

- Knowledge of Azure Synapse Analytics.

- Familiarity with data governance, security, and compliance.

- Experience working in Agile/Scrum environments.