Data Engineer - Python/ETL

Scope of Work :

- Collect requirements and collaborate with city officials, agencies, system integrators, solution providers, and other data sources to integrate relevant datasets into the data exchange platform.

- Deploy and manage data exchange, while supporting application providers in building solutions for optimised management and citizen services.

Key Responsibilities :

- Develop ETL modules in Python to ingest data into data exchange using REST APIs and streaming protocols such as AMQP and MQTT.

- This includes containerizing the adapters, creating data models, and catalogue entries according to data exchange specifications.

- Follow best practices for software development and adhere to Agile methodology throughout.

Required Skills :

- Solid understanding of available data sources and APIs in city solutions, including the ability to evaluate data quality, availability frequency, and common issues like data stream repetitions, as well as familiarity with various data fields and their meanings, and data structure formats (JSON, GeoJSON).

- Experience in developing ETL modules in Python for data ingestion into data exchange using REST APIs and streaming protocols such as AMQP and MQTT, including containerizing adapters and creating data models and catalogue entries according to data exchange specifications, while adhering to best practices in software development and Agile methodology.

- Good understanding and expertise in working with geospatial data (vector and raster), with experience in processing geospatial data using GIS tools like QGIS and ESRI, and skilled in analysing, transforming, and visualising geospatial data.

- Proficient in Linux, with experience in GIT version control and cloud computing platforms.

- Strong understanding of IoT, GIS, Big Data, and Cloud applications aimed at improving operational efficiency and service delivery, with a commitment to creating a positive societal impact.

- Experience in containerization using Docker, with familiarity in Kubernetes for orchestration is a plus.

- Knowledge of monitoring and logging tools such as Prometheus, Grafana, and Logstash would be a strong plus.