Description :

Role : Senior Data Engineer Azure.

Experience Level : 4 to 6 Years.

Work location : Mumbai, Bangalore, Trivandrum (Hybrid).

Notice Period : 0-30 days.

Key Responsibilities :

- Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks (Spark), and Azure Synapse Analytics.

- Mandatory hands-on experience with Microsoft Azure and Databricks.

- Understanding of the Medallion Architecture (specifically the Bronze/Raw layer) and data modeling concepts for data lakes.

- Experience building robust Orchestration pipelines (scheduling, error handling, logging) and managing full/incremental data loads.

- Knowledge of Delta format best practices, including partitioning, Z-ordering, vacuuming, and storage tiering for cost efficiency.

- Familiarity with CI/CD pipelines, automated deployment scripts, version control (Git), and Unit/Integration testing frameworks.

- Experience implementing Data Quality rules, PII handling/masking, and metadata management.

- Develop and optimize PySpark/Spark SQL jobs for large-scale batch and streaming data transformations.

- Ingest data from various sources including Apache Kafka, REST APIs, and RDBMS, ensuring real-time or near-real-time availability.

- Implement data modeling strategies (star schema, snowflake schema) for analytics consumption layers in Synapse or ADLS.

- Collaborate with DevOps teams to automate deployment using CI/CD pipelines (Azure DevOps, GitHub Actions, etc.

- Monitor, troubleshoot, and optimize data workflows for performance, cost-efficiency, and reliability.

- Follow coding standards, participate in peer reviews, and maintain version-controlled code in Git repositories.

- Support data quality checks, logging, alerting, and observability mechanisms for production workloads.

- Participate in sprint ceremonies and contribute to task estimation and delivery planning.

Must have skills :

- 3+ years of experience in data engineering roles.

Hands-on experience with :

- Azure Data Factory (ADF) - building pipelines, triggers, linked services.

- Azure Databricks - building and managing Spark jobs in PySpark.

- Azure Synapse Analytics - data warehousing, SQL queries, workspace orchestration.

- Apache Kafka - consuming and processing real-time data streams.

- Strong in SQL, Python, and Spark for data manipulation and transformation.

- Exposure to CI/CD practices (Azure DevOps, Git workflows, build/release pipelines).

- Understanding of data lake architecture and modern data warehousing principles.