Lead Cloud Data Engineer - PySpark/SQL

About the Job :

We're hiring an cloud data engineering (preferably Azure) data pipelines and Spark.

- Work with Databricks platform using Spark for big data processing and analytics.

- Write optimized and efficient code using PySpark, Spark SQL and Python.

- Develop and maintain ETL processes using Databricks notebooks and workflows.

- Implement and optimize data pipelines for data transformation and integration.

- Knowledge of one or more SQL variant, preferably PL/SQL and Spark SQL.

- Write complex SQL queries for data retrieval, manipulation, and analysis.

- Debugging code when required and troubleshooting any Python, Pyspark or SQL related queries.

- Good Experience with version control (Git) and ci/cd.

- Excellent problem-solving ability with solid communication and collaboration skills

Key Responsibilities :

- Work with Databricks platform using Spark for big data processing and analytics.

- Write optimized and efficient code using PySpark, Spark SQL and Python.

- Develop and maintain ETL processes using Databricks notebooks and workflows.

- Implement and optimize data pipelines for data transformation and integration.

- Knowledge of one or more SQL variant, preferably PL/SQL and Spark SQL.

- Write complex SQL queries for data retrieval, manipulation, and analysis.

- Debugging code when required and troubleshooting any Python, Pyspark or SQL related queries.

- Good Experience with version control (Git) and ci/cd.

- Excellent problem-solving ability with solid communication and collaboration skills

Technical Requirements :

- 7 to 12 years of experience as a Data Engineer or related role

- Strong expertise with Databricks platform using Spark for big data processing

- Proficiency in Python programming and data manipulation libraries like Pandas, PySpark

- Strong working knowledge of SQL for data querying and transformations

- Solid understanding of ETL design patterns and data warehousing principles

- Azure Data Engineer certification is a plus