Job Overview :

We are seeking a talented and experienced Lead Data Engineer to manage and optimise our organization's data architecture and infrastructure. In this senior role, you will lead a team of data engineers in designing, building, and maintaining scalable data pipelines and systems that transform raw data into actionable insights, ensuring data quality, security, and accessibility for various business stakeholders.

Key Responsibilities :

- Team Leadership & Mentorship : Lead, mentor, and coach a team of data engineers, fostering skill development and embedding industry best practices across the team.

- Architecture & Design : Drive the design, development, and implementation of robust, scalable data pipelines, data warehouses, and data lake solutions in a cloud environment.

- ETL/ELT Development : Oversee the development and maintenance of efficient Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) processes to integrate data from diverse sources.

- Data Governance & Quality : Implement processes for data accuracy, reconciliation, and quality assurance. Ensure compliance with data security and privacy regulations (e.g., GDPR, CCPA).

- Performance Optimisation : Optimise existing data systems and pipelines for maximum performance, efficiency, and scalability, troubleshooting and resolving bottlenecks in a timely manner.

- Collaboration : Work closely with data scientists, analysts, product managers, and executive teams to understand data requirements and deliver tailored solutions.

- Strategic Planning : Contribute to the strategic vision and road map for data services and products, evaluating and recommending new emerging technologies and tools.

- Documentation : Ensure comprehensive documentation of data pipelines, processes, and best practices for knowledge sharing and compliance.

Required Skills and Qualifications :

- Experience : Proven experience in a data engineering role, with extensive hands-on experience at a senior level, including managing a team.

- Programming Languages : Strong proficiency in programming languages such as Python, SQL, and ideally Java or Scala.

- Data Technologies : Expertise with big data technologies like Apache Spark, Hadoop, Kafka, and related ecosystems.

- Cloud Platforms : In-depth experience with major cloud-based data platforms (AWS, Azure, or GCP).

- Database Management : Strong understanding of data modeling, data warehousing concepts, and experience with relational and NoSQL databases.

Tools:

Programming: Python, SQL, Scala/Java. Cloud Platforms: AWS, GCP, Azure. Big Data: Spark, Hadoop, Databricks. Databases: SQL, NoSQL, Data Warehousing (Redshift). Tools: dbt, Docker, Kubernetes (often).

- Soft Skills: Leadership, problem-solving, communication, project management.

- Education : A Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field is typically required.