We are seeking a highly skilled Senior Data Engineer with 3-5 years of professional experience, specializing in the Azure Databricks platform. The ideal candidate will have a strong track record of designing, building, and optimizing scalable data pipelines. You will play a crucial role in enhancing our data infrastructure to support critical business intelligence and advanced analytics.

Key Responsibilities :

- Design, develop, and maintain robust data pipelines on Azure Databricks using PySpark.

- Implement and manage ETL/ELT processes for both batch and streaming data.

- Work with data scientists and analysts to understand their data requirements and deliver reliable data solutions.

- Optimize Databricks clusters and jobs for performance and cost efficiency.

- Ensure data quality, integrity, and security across all data pipelines.

Required Skills & Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 3-5 years of demonstrable experience as a Data Engineer with a strong focus on Azure Databricks mandatory .

- Proficient in PySpark and the Spark ecosystem.

- Experience with Delta Lake for building data lakes and managing ACID transactions is a must.

- Strong command of SQL and solid understanding of data warehousing concepts.

- Hands-on experience with other Azure data services, including Azure Data Lake Storage (ADLS) and Azure Synapse Analytics.

- Excellent problem-solving and communication skills.