Technical Lead - Data Engineering

Description:

We are looking for a skilled Data Engineer with 9+ years of experience in big data technologies, particularly Azure, Python, PYSpark, SQL and data lakehouse architectures. The ideal candidate will have a strong background in building scalable data pipelines and experience with modern data storage formats, including Apache Iceberg. You will work closely with cross-functional teams to design and implement efficient data solutions in a cloud-based environment.

Responsibilities:

- Design, build, and optimise scalable data pipelines using Apache Spark.

- Implement and manage large-scale data processing solutions across data lakehouses.

- Work with modern data lakehouse platforms (e. g. Apache Iceberg) to handle large datasets.

- Optimise data storage, partitioning, and versioning to ensure efficient access and querying.

- Write complex SQL queries to extract, manipulate, and transform data.

- Develop performance-optimised queries for analytical and reporting purposes.

- Integrate various structured and unstructured data sources into the lakehouse environment.

- Work with stakeholders to define data needs and ensure data is available for downstream consumption.

- Implement data quality checks and ensure the reliability and accuracy of data.

- Contribute to metadata management and data cataloguing efforts.

- Monitor and optimise the performance of Spark jobs, SQL queries, and overall data infrastructure.

- Work with cloud infrastructure teams to optimise costs and scale as needed.

Requirements:

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

- 9+ years of experience in data engineering, with a focus on Python, Spark and SQL Programming languages.

- Hands-on experience with Apache Iceberg, Snowflake, or similar technologies.

- Strong understanding of data lakehouse architectures and data warehousing principles.

- Experience with version control systems like Git and CI/CD pipelines.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Experience with containerization (Docker, Kubernetes) and orchestration tools like Airflow.

- Certifications in Azure cloud technologies.