Job Description :

- 5+ years of professional experience in data engineering, with a proven track record of building enterprise-grade data solutions.

- 3+ years of hands-on experience with Microsoft Azure data services, specifically :

a. Azure Databricks : Expert-level proficiency in developing, tuning, and debugging Spark clusters and notebooks.

b. Azure Data Lake Storage (Gen2) : Deep experience in managing data lakes, including directory structure, security (RBAC & ACLs), and performance optimization.

c. Medallion Architecture

- Expert programming skills in Python for data engineering tasks (e.g., Pandas, API interactions, unit testing).

- Expert-level proficiency in PySpark for large-scale data processing, including DataFrame API, Spark SQL, and understanding of Catalyst Optimizer.

- Strong experience in writing and optimizing complex SQL.

- Solid understanding of data modeling concepts (e.g., star schema, snowflake schema, data vault).

- Experience with data pipeline orchestration tools such as Azure Data Factory or Apache Airflow.

- Experience with version control systems (Git) and collaborative development practices.

Preferred Qualifications :

- Microsoft Azure Data Engineer Associate (DP-203) certification or similar.

- Experience with Delta Lake (format and features like ACID transactions, time travel, schema enforcement).

- Experience with real-time data processing using Azure Stream Analytics or Spark Streaming.

- Knowledge of other Azure services (Synapse Analytics, Event Hubs, Azure SQL DB, Cosmos DB, Purview).

- Experience with DevOps/DataOps principles and tools (CI/CD, Terraform, Azure DevOps).

- Familiarity with data visualization tools like Power BI.

Personal Attributes :

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills, with the ability to explain complex technical concepts to non-technical stakeholders.

- A proactive, self-motivated attitude with a strong sense of ownership and accountability.

- A continuous learner who stays updated with the latest trends in cloud data technologies.