Key Responsibilities:

- Design, develop, and maintain scalable, fault-tolerant ETL/ELT pipelines on AWS.

- Build cloud-native data solutions using AWS Glue, S3, Athena, Redshift, EMR, Lambda, and other AWS services.

- Implement Medallion Architecture (Bronze, Silver, Gold) for modern data lake solutions.

- Develop batch and near real-time data processing pipelines using PySpark.

- Design and orchestrate workflows using Apache Airflow (DAGs).

- Work with Amazon SageMaker to support machine learning and AI-driven data solutions.

- Ensure data quality, governance, security, and compliance across enterprise data platforms.

- Optimize pipeline performance, scalability, and cost efficiency.

- Collaborate with architects, data scientists, analysts, and business stakeholders to deliver robust data solutions.

- Mentor junior engineers, conduct code reviews, and drive engineering best practices.

- Troubleshoot production issues and provide ongoing operational support.

Required Skills & Experience:

- 6+ years of experience in Data Engineering.

- Strong hands-on expertise in:

1. PySpark

2. AWS Glue

3. Apache Airflow (DAGs)

4. Amazon SageMaker

5. ETL/ELT Development

6. Data Lake Architecture

- Experience with AWS services:

1. Amazon S3

2. Athena

3. Redshift

4. EMR

5. Lambda

- Strong understanding of Medallion Architecture.

- Experience in data security, governance, and performance optimization.

- Strong SQL and Python programming skills.

- Excellent analytical, communication, and stakeholder management skills.

Preferred Skills:

- Experience with Delta Lake, Databricks, or Apache Spark.

- Knowledge of CI/CD pipelines, Git, and DevOps practices.

- Experience with Infrastructure as Code (Terraform or CloudFormation).

- Exposure to streaming technologies such as Kafka or Kinesis.

- AWS certifications are an added advantage.

Key Competencies :

- Data Engineering

- AWS Cloud Services

- PySpark & Apache Spark

- AWS Glue & Amazon SageMaker

- Apache Airflow

- Medallion Architecture

- ETL/ELT Development

- Data Lake & Data Governance

- Technical Leadership

- Performance Optimization & Cross-functional Collaboration