Role Overview:

We are seeking a Software Engineer with 3 - 5 years of data engineering experience to build and operate the data pipelines and services behind our platforms. You will own ingestion, transformation, and serving of data from diverse sources, keep it reliable and well-governed, and work closely with product, platform, and programme teams to deliver data systems at scale. This role also requires hands-on DevOps experience.

Required Skills & Qualifications:

- Experience: 3 - 5 years in data engineering or backend software engineering with a data focus.

- Education: Bachelor's/Master's in Computer Science or a related field.

- Programming: Strong skills in Python (or Java/Scala) and advanced SQL, with clean, testable coding practices.

- Data Pipelines: Hands-on experience building batch and streaming ETL/ELT pipelines using tools such as Apache Airflow, Spark, Kafka, or dbt.

- Databases & Storage: Solid knowledge of PostgreSQL or similar relational databases, plus at least one NoSQL, columnar, or time-series store; familiarity with data lake/warehouse concepts and formats such as Parquet, Avro, and JSON.

- APIs: Experience designing and consuming RESTful APIs and integrating data from varied sources and formats.

- Tools: Proficiency with Git, code reviews, automated testing, and CI workflows.

- DevOps: Hands-on experience with Docker, Kubernetes, CI/CD (GitHub Actions, GitLab CI, or Jenkins), infrastructure as code (Terraform/Ansible), monitoring (Prometheus, Grafana), and cloud or Linux environments.

- Soft Skills: Problem-solving, clear communication and documentation, teamwork, and ability to work independently.

Key Responsibilities:

- Design, build, and maintain scalable batch and streaming data pipelines for ingesting data from APIs, databases, files, and event streams.

- Develop ETL/ELT workflows and data models that are reliable, testable, and easy to evolve.

- Build and optimise data storage and serving layers, and expose data through well-designed APIs.

- Implement data quality checks, validation, lineage, monitoring, and alerting to ensure trusted data.

- Tune pipeline and query performance, and troubleshoot production data issues end to end.

- Collaborate with architects, product owners, and programme teams to translate requirements into technical solutions.

- Follow security, privacy, and data governance practices suited to government-facing systems.

- Write clean, documented, version-controlled code; participate in code reviews and design discussions.

- Deploy, monitor, and maintain the Data Exchange platform in production to keep it stable and dependable.