Description :

Job Description :

We are hiring an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, Python, and SQL, and proven experience in migrating Hadoop-based data platforms to Azure Databricks. The role focuses on building scalable data pipelines, modernizing legacy big data systems, and delivering high-performance data solutions on Azure.

Key Responsibilities :

- Design, develop, and maintain data pipelines using Azure Databricks & PySpark

- Execute Hadoop (HDFS/Hive) to Azure Databricks migration initiatives

- Write clean, efficient, and scalable Python code for data processing

- Develop and optimize complex SQL queries

- Handle large-scale datasets with batch and incremental processing

- Optimize Databricks workloads for performance and reliability

- Collaborate with cross-functional teams to deliver data solutions

- Ensure data quality, security, and best practices

Required Skills & Qualifications :

Mandatory Skills :

- Azure Databricks (hands-on experience)

- PySpark

- Python (strong coding experience is mandatory)

- SQL

- Hadoop ecosystem (HDFS, Hive, etc.)

- Hadoop to Databricks migration experience

Good to Have :

- Azure Data Factory / Azure Synapse

- Delta Lake & Lakehouse architecture

- Performance tuning and optimization

- CI/CD exposure for data pipelines

Ideal Candidate Profile (Recruiter Insight) :

- Strong hands-on data engineer, not just architecture-level

- Real-world migration project experience

- Comfortable working with large datasets

- Can independently own Databricks development