Role : Data Engineer - Data Warehouse and Data Pipelines

Core Responsibilities :

1. Data Pipeline Development and Maintenance :

- Design, implement, and maintain ETL/ELT data pipelines to reliably ingest high-volume, time-series metrics from diverse internal systems (servers, network devices, custom applications).

- Ensure data quality, integrity, and timely processing for all operational metrics.

- Collaborate with platform teams to integrate new data sources and evolve existing pipeline schemas.

2. Observability and Visualisation :

- Develop and manage dashboards and visualisations (e.g., using tools like Grafana, Kibana, or Power BI) that provide clear, actionable insights into system health, usage patterns, and performance bottlenecks.

- Focus on creating views that empower end-users and engineers to self-service their troubleshooting and system analysis needs.

3. Infrastructure and Scalability :

- Contribute to the optimisation and scaling of our data storage solutions (e.g., time-series databases, data lakes) to handle metric data from thousands of devices.

- Monitor pipeline and database performance, identifying and implementing efficiency improvements.

4. Cross-Functional Collaboration :

- Work closely with Site Reliability Engineers (SREs), Platform Engineers, and Product Owners to understand their data requirements and translate them into effective data solutions and visualisations.

Desired Skills for a Data Engineer :

This role requires a candidate with strong foundational programming and SQL skills, along with a keen interest in modern ELT tools and data warehousing concepts.

1. Foundational Programming & Development :

- Core Programming Proficiency : Proven ability in at least one major programming language (e.g., Python, Go, C#, Java, etc).

- Expectation : Understanding of core syntax, data structures, and the ability to write reliable, efficient scripts.

- SQL Expertise (Mandatory) : Strong ability to write and understand complex SQL queries, including JOINs, window functions, and optimisation techniques.

- Version Control : Familiarity with Git (basic commands like committing, branching, and pull requests).

- Development Practices : Basic understanding of testing (e.g., unit tests) and the software development lifecycle (SDLC).

2. Data Warehousing :

- Analytical Data Warehouse Concepts : Experience or strong academic understanding of a modern data warehouse platform (e.g., Snowflake, Iceberg, BigQuery, Dremio, ClickHouse, etc).

- Data Modelling : Foundational knowledge of relational database design and common data modelling patterns for analytics, particularly dimensional modelling (star/snowflake schemas).

- Data Model Implementation : Experience designing and implementing efficient and scalable data models under the dbt framework, following best practices.

- Data Quality : Ability to implement data quality checks and monitoring to ensure data accuracy, completeness, and consistency.

- Optimisation : Proficiency in optimising SQL queries, warehouse costs, and performance.

- Cloud Storage : Familiarity with concepts of large-scale cloud storage (e.g., AWS S3, Azure Data Lake Storage, or GCS).

3. General Skills & Mindset :

- Problem-Solving : Eagerness to debug issues, troubleshoot data quality problems, and investigate pipeline failures.

- Communication & Learning : Strong desire to learn new technologies quickly and ability to communicate technical concepts clearly.

- Attention to Detail : Focus on accuracy and consistency in data handling.