Hiring Senior Data Engineer - Databricks | PySpark | Spark | Delta Lake | AWS/Azure

Total Experience : 7-9 years

About the Role :

We are looking for a hands-on Senior Data Engineer / Databricks Engineer who can build scalable, reliable and high-performance data pipelines using Databricks, Apache Spark, PySpark, Python and SQL.

You will play a key role in designing and delivering modern Lakehouse, ETL/ELT and data engineering solutions, transforming business requirements into production-ready pipelines, Delta Lake tables and reusable data components.

This is an opportunity for a strong Data Engineer who enjoys solving complex data problems, optimizing Spark workloads and building production-grade data platforms across AWS or Azure.

Key Responsibilities :

- Design, develop and maintain scalable ETL/ELT data pipelines using Databricks, Spark and PySpark.

- Build and manage production-grade Delta Lake tables and Lakehouse data solutions.

- Translate business and technical requirements into robust, reusable and maintainable data engineering solutions.

- Develop high-quality Python, SQL and PySpark code for large-scale data processing.

- Optimize Apache Spark / PySpark jobs for performance, scalability and cloud cost efficiency.

- Implement Databricks Workflows for reliable data pipeline orchestration and scheduling.

- Apply Unity Catalog best practices for data governance, security and access management.

- Develop reusable data engineering frameworks, reference patterns and components.

- Work with Databricks SQL and modern Lakehouse architecture.

- Support AI/ML data pipelines, model development workflows and production data requirements.

- Integrate data from multiple sources, including APIs and cloud-based systems.

- Collaborate with Data Scientists, ML Engineers, Software Engineers and business stakeholders.

- Troubleshoot pipeline failures, data quality issues and performance bottlenecks.

- Follow engineering best practices for testing, documentation, version control and deployment.

- Contribute to DevOps / MLOps practices and infrastructure automation where required.

Required Skills & Qualifications :

- Bachelor's degree in Computer Science, Engineering, Information Technology or a related field.

- 7+ years of experience in Software Engineering, Data Engineering or a related technical discipline.

- Strong programming skills in Python, SQL, and PySpark.

- Strong understanding of ETL / ELT pipeline development.

- Hands-on experience with Delta Lake and Databricks Lakehouse architecture.

- Good knowledge of AWS or Microsoft Azure cloud platforms.

- Strong understanding of Data Warehousing and Lakehouse architecture.

- Experience building scalable data pipelines and processing large datasets.

- Strong understanding of data engineering concepts, data modeling and pipeline optimization.

Preferred Skills :

- Experience with Databricks Workflows, Databricks SQL, and dbt (Data Build Tool).

- Experience supporting AI/ML pipelines and MLOps / DevOps practices.

- Experience with Terraform or AWS CloudFormation.

- Strong understanding of REST APIs / API-based data ingestion.

- Experience with CI/CD and automated deployment pipelines.

- Knowledge of cloud-native data engineering and modern data platforms.