Role Overview :

We are looking for an experienced Lead Data Engineer with strong hands-on expertise in Databricks, PySpark, and Cloud technologies (AWS or Azure) to lead the design, development, and implementation of scalable data engineering solutions.

The ideal candidate should have strong experience in building cloud-based data platforms, data pipelines, ETL/ELT workflows, and data processing solutions using Databricks and PySpark.

The candidate will be responsible for providing technical leadership while remaining hands-on with data engineering activities.

Primary Skills :

- Databricks

- PySpark

- AWS or Azure

Key Responsibilities :

- Lead the design and development of scalable data engineering and data processing solutions.

- Develop and optimize large-scale data pipelines using Databricks and PySpark.

- Design and implement robust ETL/ELT pipelines for batch and real-time data processing.

- Work with cloud data platforms on AWS or Azure.

- Develop data solutions following Lakehouse architecture and modern data engineering best practices.

- Design and implement data ingestion, transformation, validation, and integration processes.

- Optimize PySpark jobs, Databricks workloads, SQL queries, and data pipelines for performance and cost.

- Work with structured and unstructured data from multiple sources.

- Implement data quality, data validation, monitoring, and error-handling mechanisms.

- Collaborate with Data Architects, Data Scientists, BI teams, application teams, and business stakeholders.

- Provide technical leadership, conduct code/design reviews, and mentor junior and mid-level data engineers.

- Troubleshoot production data pipelines and resolve performance and data-quality issues.

- Participate in architecture discussions and contribute to technical standards and best practices.

Screening Questions :

- How many years of hands-on experience do you have with Databricks and PySpark

- Which cloud platform have you worked on hands-on - AWS or Azure? Please specify your experience in years.

- How many years of experience do you have in designing and developing large-scale ETL/ELT data pipelines, including Lakehouse architecture?