We are looking forward to your interview for the Senior Azure Data Engineer role at Algoworks. Please join the interview on time and ensure you have a stable internet connection through laptop.

Additionally, choose a quiet location to minimize distractions and be prepared to use your video camera during the interview.

Job Title : Senior Data Engineer - Azure

Experience Level : 6 to 8 years of hands-on data engineering experience, with strong expertise in Azure Databricks and Azure Data Factory.

Role Overview :

The Senior Data Engineer will be responsible for building, optimizing, and maintaining scalable data pipelines across Bronze, Silver, and curated layers.

This role focuses on high-performance ETL/ELT development, Delta Lake optimization, and close collaboration with DWH teams for downstream consumption.

The engineer will act as a senior contributor, ensuring reliability, performance, and data quality across the platform.

Key Responsibilities :

Pipeline Development :

- Build and maintain data pipelines using Azure Databricks and Azure Data Factory.

- Implement ingestion and transformation logic across Bronze and Silver layers.

- Support batch and incremental processing patterns.

Curated Layer Logic :

- Implement hydration, merge, and upsert logic using Delta Lake.

- Ensure curated datasets meet data quality and business requirements.

- Handle late-arriving data and incremental updates.

Performance & Storage Optimization :

- Optimize Delta Lake tables for performance and cost.

- Select and tune appropriate storage formats (Parquet / Delta).

- Apply partitioning, compaction, and file sizing strategies.

- Tune Spark jobs for large-scale data processing.

Downstream & DWH Collaboration :

- Work closely with DWH and BI teams to support downstream consumption.

- Provide optimized datasets for Synapse and reporting workloads.

- Support data validation and reconciliation with Gold layer outputs.

Engineering Best Practices :

- Implement basic CI/CD practices for data pipelines.

- Follow coding standards, documentation, and version control practices.

- Support production troubleshooting and performance tuning.

Required Skills & Experience :

Experience :

- 6 to 8 years of experience in data engineering.

- Strong hands-on experience building pipelines on Azure.

- Experience working with large datasets and distributed processing.

Technical Skills :

- Strong proficiency in PySpark.

- Hands-on experience with Azure Databricks.

- Strong experience with Azure Data Factory.

- Deep knowledge of Delta Lake tuning and optimization.

- Experience with storage optimization (Parquet, Delta).

- Strong SQL skills for transformation and validation.

Tools & Practices :

- Experience with Git and basic CI/CD pipelines.

- Familiarity with data quality and validation techniques.

- Experience working in Agile delivery models.

Soft Skills :

- Strong analytical and problem-solving skills.

- Ability to work independently on complex pipelines.

- Good communication and collaboration skills.

Nice to Have :

- Experience supporting Synapse Dedicated SQL Pool.

- Exposure to streaming or near real-time pipelines.

- Familiarity with data governance or metadata tools.