Capgemini - Pyspark/Azure Databricks Engineer

Job Description :

We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics.

Responsibilities :

- Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation.

- Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources.

- Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights.

- Optimize PySpark jobs and data pipelines for performance, scalability, and reliability.

- Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows.

- Ensure data quality and integrity throughout all stages of data processing.

- Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation.

- Troubleshoot and debug issues related to data pipelines and processing.

- Ensure compliance with regulatory requirements and industry standards in all data processing activities.

- Implement best practices for data security, compliance, and privacy within Azure environment.

- Document technical specifications, data flows, and solution architecture.

Requirements :

- Bachelors degree in computer science, Engineering, or a related field; Masters degree preferred.

- Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals.

- Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF).

- Proficiency in Python programming language and solid understanding of SQL.

- Experience designing and optimizing data pipelines for ETL/ELT processes.

- Familiarity with cloud platforms, preferably Microsoft Azure.

- Excellent problem-solving skills and ability to think critically.

- Strong communication skills with the ability to collaborate effectively in a team environment.

- Experience in Financial, Risk, Compliance, or Banking domains is a plus.

- Experience identifying and mitigating financial risks in data processes.

- Ability to analyse data for potential risk factors and develop strategies to minimize financial risk.

- Ensure all data processes comply with relevant regulatory requirements and industry standards.

Preferred Qualifications :

- Certification in Azure Data Engineering or related field.

- Knowledge of other big data technologies such as Hadoop, Hive, or Kafka.

- Familiarity with machine learning frameworks and techniques