Job Description :

Role summary :

We are looking for an Expert Software Engineer to build and scale core components of our Data Platform using Java, pipeline orchestration, and microservices architecture.

This role focuses on designing systems that capture, refresh, and compare (CCR) data across platforms to ensure consistency, correctness, and traceability. You will build backend services and pipeline frameworks that enable reliable data movement and validation across distributed systems.

You will operate at the intersection of data processing and backend engineering, with a strong emphasis on system reliability, data correctness, and platform scalability.

Key responsibilities :

Pipeline & Data Processing Systems :

- Design and implement pipeline management systems for data capture, refresh, and comparison workflows

- Build scalable mechanisms to capture data from multiple sources, process updates, and maintain synchronization

- Develop efficient data refresh strategies ensuring consistency and minimal latency

- Implement data comparison (CCR) frameworks to detect mismatches, anomalies, and drift across systems

Microservices & Platform Development :

- Design and develop Java-based microservices supporting data pipelines and CCR workflows

- Build RESTful APIs to trigger, monitor, and manage pipeline executions

- Ensure services are scalable, resilient, and fault-tolerant

- Apply domain-driven design (DDD) principles where applicable

Containerization & Deployment :

- Develop and deploy services using Docker-based containerization

- Ensure consistency across environments through containerized builds and deployments

- Support integration with orchestration environments where applicable

Data Consistency & Reliability :

- Build systems that ensure data integrity across distributed services

- Design mechanisms for idempotency, retry handling, and failure recovery

- Identify and resolve data discrepancies through automated comparison frameworks

Performance & Optimization :

- Optimize pipelines and services for throughput, latency, and resource efficiency

- Troubleshoot bottlenecks in data movement and comparison processes

- Ensure systems scale effectively with increasing data volume and complexity

Quality & Engineering Excellence :

- Write clean, maintainable, and testable code following engineering best practices

- Implement unit, integration, and system-level tests

- Participate in code reviews and design discussions

- Contribute to CI/CD pipelines for automated build and deployment

Collaboration & Ownership :

- Work closely with product, engineering, and data teams to understand data consistency requirements

- Take end-to-end ownership of features from design to production support

- Communicate progress, risks, and technical decisions clearly

Continuous Improvement :

- Identify gaps in existing pipeline and data validation systems and propose improvements

- Drive enhancements in pipeline reliability, monitoring, and maintainability

- Stay current with best practices in microservices and data platform engineering

Required qualifications (Hard requirements) :

- 8+ years of software engineering experience

- Strong programming expertise in Java

- Experience building microservices-based architectures

- Hands-on experience with pipeline management / workflow systems

- Experience working with Docker and containerized deployments

- Experience designing and implementing data capture, refresh, and comparison (CCR) systems or similar data validation frameworks

Strong understanding of :

- Distributed systems and data consistency challenges

- API design and backend service development

- Fault tolerance, retries, and idempotent processing

- Experience with CI/CD pipelines and modern development practices

Preferred qualifications :

- Experience with large-scale data systems or data platforms

- Familiarity with event-driven architectures

- Exposure to cloud platforms (Azure/AWS/GCP)

- Experience designing data reconciliation or audit systems

- Understanding of performance tuning for data-intensive services

What success looks like :

A successful engineer in this role will :

- Build robust systems that ensure data consistency across platforms

- Deliver scalable pipelines that reliably capture, refresh, and compare data

- Identify and resolve data discrepancies proactively

- Design services that are resilient, observable, and production-ready

- Take ownership of systems that directly impact data trust and platform reliability