Key Responsibilities :

- Design, develop, and maintain end-to-end data pipelines for batch and near-real-time data processing.

- Write and optimize complex SQL queries, primarily on Amazon Redshift, to support analytics and reporting use cases.

- Build and manage ETL/ELT pipelines using AWS Glue, integrating data from S3, RDS, APIs, and flat files.

- Develop scalable data processing jobs using Apache Spark (PySpark), preferably on AWS EMR.

- Perform data modeling, query tuning, and performance optimization for large datasets.

- Create and maintain dashboards and visual analytics using AWS QuickSight.

- Implement data ingestion frameworks to handle structured and semi-structured data from multiple sources.

- Orchestrate workflows using Apache Airflow or similar scheduling tools.

- Monitor data pipelines and ensure data quality, reliability, and availability.

- Collaborate with DevOps teams to support deployment, monitoring, and troubleshooting.

- Ensure adherence to data security, governance, and compliance best practices.

Required Skills & Qualifications :

- 3-5 years of experience in Data Engineering or related roles.

- Expert-level proficiency in SQL, especially for Amazon Redshift.

- Hands-on experience with Apache Spark and PySpark.

- Strong experience with AWS Glue for ETL pipelines.

- Proficiency in Python for scripting and data transformation.

- Solid understanding of data warehousing concepts and data modeling.

- Experience with AWS services such as S3, Lambda, Athena, Step Functions, and CloudWatch.

- Familiarity with workflow orchestration tools like Apache Airflow.

- Working knowledge of version control systems (Git) and basic DevOps practices.

Preferred Skills :

- Experience with real-time data streaming platforms such as Kafka or Amazon Kinesis.

- Exposure to infrastructure as code tools like Terraform or AWS CloudFormation.

- Understanding of data security, access control, and compliance frameworks.

- Experience supporting analytics and BI teams.