Key Responsibilities :

- Design and develop scalable batch and real-time data pipelines on AWS.

- Build and maintain robust ETL/ELT pipelines using AWS data services.

- Develop data processing and transformation workflows using Python and PySpark.

- Ingest data from databases, APIs, files, SaaS applications, and other sources.

- Work with Amazon S3, AWS Glue, Redshift, Athena, Lambda, and other relevant AWS services.

- Develop and optimize SQL queries for large datasets.

- Design and implement data lakes, data warehouses, and data models.

- Implement data quality, validation, monitoring, and reconciliation processes.

- Optimize data pipelines for performance, scalability, reliability, and cost efficiency.

- Troubleshoot and resolve data pipeline failures and production issues.

- Implement appropriate security, IAM, and access-control practices for data platforms.

- Collaborate with Data Architects, Data Scientists, Analysts, and application teams.

- Participate in code reviews, testing, deployment, and CI/CD activities.

Required Skills :

- 5 - 10 years of experience in Data Engineering.

- Strong hands-on experience with AWS Data Engineering.

- Strong knowledge of AWS Glue and Amazon S3.

- Strong proficiency in Python, PySpark, and SQL.

- Experience developing ETL/ELT pipelines.

- Good understanding of data lakes, data warehouses, and data modelling.

- Experience with AWS services such as Redshift, Athena, Lambda, and CloudWatch.

- Strong understanding of distributed data processing.

- Experience with data quality, monitoring, and troubleshooting.

- Good understanding of AWS IAM and cloud security.

- Knowledge of Git and CI/CD practices.

Preferred Candidate Profile :

- Experience building production-grade and scalable data platforms on AWS.

- Hands-on experience handling large datasets using PySpark and AWS Glue.

- Strong understanding of cloud data architecture and best practices.

- Experience with streaming technologies such as Kafka/Kinesis is an advantage.

- Exposure to Terraform or Infrastructure as Code (IaC) is preferred.

- Strong analytical and problem-solving skills.

- Excellent communication and collaboration skills.