Required Skills:

Azure Databricks, PySpark, SQL, dbt, and Airflow, Medallion Architecture (Bronze/Silver/Gold), data modelling skills (Star/Snowflake schemas).

Job Description:

Skills:

- 5 to 8 years of experience in Azure Data Engineering.

- Strong expertise in Azure Databricks, PySpark, SQL, dbt, and Airflow.

- Experience building scalable ETL/ELT pipelines on Azure Data Lake.

- Hands-on experience with Medallion Architecture (Bronze/Silver/Gold).

- Strong data modelling skills (Star/Snowflake schemas).

- Knowledge of CI/CD, GitHub/Azure DevOps, and data quality/governance practices.

- Good understanding of analytics, reporting, and AI/ML data platforms.

- Databricks certification is required.

- DBT Experience required.

Key Responsibilities :

- Design and implement robust end-to-end data pipelines using Azure Databricks and PySpark to process large-scale structured and unstructured data for downstream consumption.

- Orchestrate complex data workflows using Apache Airflow to ensure timely delivery and reliability of data across the enterprise.

- Develop and optimize sophisticated SQL queries and stored procedures to support high-performance reporting and data retrieval requirements.

- Partner with cross-functional engineering teams to troubleshoot data quality issues and implement automated monitoring solutions that maintain system integrity.

- Lead the migration and modernization of legacy data systems to the Azure cloud, enhancing scalability and reducing operational overhead for the organization.

Required Skillset :

- Demonstrated expertise in building distributed data processing systems using Azure Databricks and PySpark, with a deep understanding of performance tuning and cluster management.

- Proven ability to design complex ETL/ELT workflows using Airflow, ensuring seamless integration across diverse data sources.

- Advanced proficiency in SQL for data modeling, complex transformations, and performance optimization within large-scale database environments.

- Strong interpersonal skills with the ability to translate technical data challenges into clear, actionable insights for non-technical stakeholders.

- A Bachelors or Masters degree in Computer Science, Engineering, or a related quantitative field, paired with 5 - 7 years of hands-on experience in data engineering.

- Adaptability to thrive in a hybrid work environment, maintaining high levels of productivity and collaboration across distributed teams in Bangalore and Hyderabad.