Senior Consultant - Data Engineer

Experience : 8+ Years

Employment Type : Full-time

Job Overview :

We are seeking a highly skilled Senior Consultant - Data Engineer with 8+ years of experience, particularly within the Financial Services sector. You will be instrumental in designing, building, and optimizing data pipelines and ETL processes in Azure Databricks, ensuring high performance, reliability, and scalability. This role demands a commitment to strategic, optimal solutions, strong technical skills in SQL, Python, and PySpark, and the ability to collaborate effectively within an Agile team to deliver robust data solutions for our clients.

Responsibilities :

- Design, build, and optimize data pipelines and ETL processes specifically within Azure Databricks, ensuring high performance, reliability, and scalability for large datasets.

- Implement best practices for data ingestion, transformation, and cleansing to ensure exceptional data quality and integrity across all stages of the pipeline.

- Adhere strictly to client's best practice guidelines as set out by the Data Engineering Lead, ensuring architectural consistency and operational excellence.

- Collaborate closely with data modelers and testers to ensure that data pipelines are implemented correctly and meet all specified requirements.

- Work as an integral part of a cross-functional Agile team to understand complex business requirements and translate them into efficient and effective technical data solutions.
- Demonstrate a continual desire to implement strategic or optimal data solutions, actively avoiding workarounds or short-term tactical approaches wherever possible.

Role Requirements :

- Strong Data Engineer with extensive experience, particularly within the Financial Services industry.

- Proven knowledge of and hands-on experience building data pipelines in Azure Databricks.

- A strong commitment to implementing "strategic" or "optimal" solutions, prioritizing long-term maintainability and performance over quick fixes.

- Experience working effectively within an Agile team environment.

Experience/Skillset :

- 8+ years of experience in data engineering roles.

- Strong proficiency in SQL, Python, and PySpark.

- Good knowledge of Azure Databricks, including a solid understanding of delta tables, Apache Spark, and Unity Catalog.

- Extensive experience in writing, optimizing, and analyzing SQL and PySpark code, coupled with a robust capability to interpret complex data requirements and architect scalable solutions.

- Good knowledge of the Software Development Life Cycle (SDLC).

- Familiarity with Agile/Scrum ways of working.

- Strong verbal and written communication skills, capable of articulating complex technical concepts to diverse audiences.

- Ability to manage multiple priorities effectively and deliver high-quality solutions to tight deadlines.