Senior Data Engineer - Azure Databricks

Job Description :

Key Responsibilities :

- Design, develop, and maintain scalable batch and real-time data pipelines.

- Build and optimize data solutions using Databricks, PySpark, Delta Lake, and Apache Kafka.

- Develop ETL/ELT workflows for ingesting, transforming, and processing large-scale data.

- Design cloud-native data architectures and modernize legacy data platforms.

- Implement real-time streaming and event-driven data processing solutions.

- Ensure data quality, governance, security, and compliance across enterprise data platforms.

- Optimize pipeline performance, scalability, and cost efficiency.

- Collaborate with Data Scientists, ML Engineers, Product teams, and business stakeholders to deliver reliable data solutions.

Required Skills :

- Strong experience in Python, PySpark, SQL, and distributed data processing.

- Hands-on expertise with Databricks, Apache Spark, Delta Lake, and Apache Kafka.

- Experience in ETL/ELT development, data pipelines, and streaming technologies.

- Knowledge of cloud platforms such as Azure, AWS, or GCP.

- Experience with data lakes, lakehouse architecture, and modern data engineering practices.

- Familiarity with CI/CD, Git, Docker, and monitoring tools is preferred.

Mandatory Requirements :

- 4 - 9 years of experience in Data Engineering.

- Healthcare/HealthTech domain experience is mandatory.

- Strong experience in building scalable data pipelines and streaming applications.

- Experience working with cloud-based Big Data platforms.

Preferred Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Databricks or Cloud Data Engineering certifications are a plus.