Job Title : AWS Data Platform Framework Development Engineer

Experience : 8+ Years

Location : Noida , Mumbai , Pune , Bengaluru , Chennai , Hyderabad

Employment Type : Full-Time

Department : DAF Digital - Client IT Europe

About the Organization :

DAF Digital, a sub-department of Client IT Europe, is leading the development of a modern, Iceberg-based data lakehouse platform on AWS Cloud. This platform is built to deliver reusable, scalable, and standardized data processing frameworks that accelerate ingestion, transformation, and analytics for DAF Trucks N.V., Netherlands.

The architecture follows a medallion lakehouse pattern and leverages :

- AWS Glue (PySpark) for processing

- Apache Airflow for orchestration

- Debezium Kafka CDC pipelines for streaming ingestion

- Snowflake as Data Warehouse as a Service (DWHaaS)

- GitHub for CI/CD

- Terraform for Infrastructure as Code

This initiative focuses on building reusable frameworks rather than one-off pipelines, ensuring extensibility, scalability, and maintainability.

Role Overview :

We are seeking a highly skilled AWS Data Platform Framework Development Engineer with a strong software engineering foundation and extensive framework development experience.

This role requires a developer mindset focused on building extensible frameworks, reusable libraries, and standardized platform components rather than isolated data pipelines.

The ideal candidate will :

- Have strong command over OOP, SOLID principles, and design patterns

- Understand framework extension points and pluggable architectures

- Bring deep expertise in AWS, Kafka, and Debezium

- Possess 8+ years of software engineering experience

Key Responsibilities :

Framework Engineering & Architecture :

- Design, develop, and maintain reusable, scalable, and extensible data processing frameworks.

- Apply OOP principles (encapsulation, inheritance, abstraction, polymorphism).

- Implement SOLID principles (SRP, OCP, LSP, ISP, DIP).

- Utilize design patterns such as Factory, Singleton, Observer, Strategy, and Dependency Injection.

- Define extension points allowing developers to plug in custom logic without modifying core framework code.

AWS Data Platform Development :

- Build scalable data processing frameworks using AWS services :

1. AWS Glue (PySpark)

2. S3

3. IAM

4. CloudWatch

- Align data processing with medallion architecture (Bronze, Silver, Gold layers).

- Ensure scalability, fault tolerance, and performance optimization.

Kafka & Debezium CDC Frameworks :

- Develop and enhance Debezium-based CDC ingestion frameworks.

- Implement Kafka and Kafka Connect-based streaming pipelines.

- Design standardized ingestion patterns for scalable streaming solutions.

- Optimize event-driven data workflows.

Snowflake Integration & ETL Standardization :

- Develop reusable ETL frameworks for efficient data loading into Snowflake.

- Standardize ingestion, transformation, and loading patterns.

- Ensure schema evolution and data consistency.

Infrastructure Automation & DevOps :

- Automate infrastructure provisioning using Terraform.

- Implement CI/CD pipelines using GitHub workflows.

- Promote Git-based development and automated deployment strategies.

- Ensure environment consistency across dev, test, and production.

Platform Reliability & Engineering Excellence :

- Implement observability, logging, and monitoring best practices.

- Drive performance tuning and cost optimization initiatives.

- Conduct code reviews and enforce development standards.

- Advocate reusable tools and standardized engineering practices.

- Collaborate within international, cross-functional teams.

Required Skills & Experience :

- Master's degree in Computer Science, Software Engineering, IT, or related field.

- Minimum 8+ years of experience in software/application development.

- Proven experience in framework or library development.

- Strong expertise in :

1. AWS (Glue, S3, IAM, CloudWatch)

2. Kafka & Kafka Connect

3. Debezium (CDC)

- Strong understanding of :

1. OOP and SOLID principles

2. Design patterns

3. Framework extensibility and architecture design

- Experience with :

1. Java, C++, or C#

2. Python (PySpark preferred)

- Deep understanding of Object-Oriented, procedural, and functional programming paradigms.

- Strong knowledge of scalable data ingestion and processing architectures.

- Familiarity with GitHub-based CI/CD workflows.

- Excellent communication and collaboration skills.

Preferred Qualifications :

- AWS Certifications (Solutions Architect Associate, Developer Associate, Data Engineer Associate).

- Experience with Apache Iceberg or open table formats.

- Familiarity with containerization (Docker, Kubernetes).

- Experience with Terraform or CloudFormation.

- Knowledge of medallion architecture and lakehouse patterns.

- Experience working in big data environments.

What Success Looks Like in This Role :

- Delivery of extensible, reusable frameworks rather than ad-hoc pipelines.

- Standardized CDC ingestion patterns using Kafka and Debezium.

- Automated and reproducible infrastructure deployments.

- Improved developer productivity via framework-based architecture.

- Scalable, reliable, and maintainable AWS data platform.