Location : Noida

Work Mode : Work From Office

Experience : 9-12 Years

Role Overview :

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, Python and SQL to design, develop and optimize scalable enterprise data engineering solutions.

The ideal candidate should have extensive experience in building robust ETL/ELT pipelines, processing large-scale datasets, implementing modern Lakehouse architectures and optimizing data platforms for performance, scalability and cost efficiency.

The candidate will also drive Databricks serverless migration initiatives, CI/CD implementation and engineering best practices.

Key Responsibilities :

- Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark, Python and SQL.

- Build and optimize ETL/ELT workflows for structured and semi-structured data.

- Develop and manage data pipelines using Apache Spark, Spark SQL and Delta Lake.

- Lead migration of existing Databricks workloads to serverless compute architecture.

- Implement and maintain CI/CD pipelines using GitHub Actions or equivalent tools.

- Manage code repositories, monitor pull requests, conduct code reviews and ensure deployment quality.

- Optimize Spark jobs, data processing workflows and Databricks workloads for performance and cost efficiency.

- Implement data governance, security, access control and compliance best practices.

- Develop modular, reusable and maintainable code; refactor existing pipelines where required.

- Work closely with cross-functional teams to understand business requirements and deliver reliable data solutions.

- Support automated testing, data validation, monitoring and production troubleshooting.

Mandatory Technical Skills :

- Azure Databricks : Minimum 2+ years of hands-on experience.

- Data Engineering : 8+ years of relevant experience in Data Engineering, ETL/ELT and large-scale data processing.

- Programming : Strong hands-on expertise in Python and PySpark.

- Apache Spark : Strong understanding of Spark architecture, Spark SQL and performance optimization.

- SQL : Advanced SQL skills, including complex queries and data transformations.

- ETL/ELT : Experience designing and implementing enterprise-grade data pipelines.

- Data Architecture : Strong understanding of Delta Lake, Lakehouse architecture and data modelling.

- Orchestration : Experience with Apache Airflow and/or equivalent orchestration tools.

- CI/CD : Hands-on experience with GitHub Actions, Azure DevOps or similar CI/CD platforms.

Preferred / Good-to-Have Skills :

- Delta Live Tables (DLT).

- Unity Catalog and data governance.

- Delta Sharing.

- Databricks SQL Warehouse, including serverless SQL Warehouse.

- Serverless compute architecture and migration.

- dbt or similar data transformation frameworks.

- Cloud security, access management and compliance.

- Experience with financial data, pricing logic or complex business calculations.

- Exposure to AI/ML-based data solutions.

- Experience leading technical teams, mentoring engineers and conducting code reviews.

Ideal Candidate Profile :

- Strong hands-on engineering background rather than only project coordination or team management.

- Proven experience developing production-grade Databricks and PySpark solutions.

- Experience handling large-scale data migration and transformation projects.

- Strong problem-solving, analytical and communication skills.

- Ability to independently drive technical initiatives and collaborate with stakeholders.