About the Role :

We are looking for an experienced Data Engineer with strong expertise in PySpark, Python, AWS and Apache Airflow to design, develop and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience building robust ETL/ELT pipelines, processing large datasets and developing cloud-based data platforms.

Key Responsibilities :

- Design, develop and maintain scalable data pipelines using Python and PySpark.

- Build and optimize ETL/ELT workflows for processing structured and unstructured data.

- Develop and manage data processing solutions on AWS cloud services.

- Design, implement and orchestrate data workflows using Apache Airflow.

- Work with AWS services such as S3, Glue, EMR, Lambda, Redshift and Athena.

- Perform data transformation, cleansing, validation and integration across multiple sources.

- Optimize PySpark jobs for performance, scalability and cost efficiency.

- Implement data quality checks, monitoring and error-handling mechanisms.

- Collaborate with data architects, analysts and business stakeholders to understand data requirements.

- Ensure data security, governance and adherence to engineering best practices.

- Troubleshoot pipeline failures and resolve data processing issues.

Required Skills & Qualifications:

- 6 - 10 years of experience in Data Engineering.

- Strong hands-on experience with PySpark and Python.

- Expertise in building ETL/ELT pipelines and large-scale data processing solutions.

- Strong experience with AWS data services, particularly S3, Glue, EMR, Redshift and Athena.

- Hands-on experience with Apache Airflow for workflow orchestration and scheduling.

- Strong understanding of SQL, data warehousing and database concepts.

- Experience working with distributed computing and big data processing frameworks.

- Knowledge of data modelling, data integration and pipeline optimization.

- Experience with Git, CI/CD and deployment practices.

- Strong analytical, problem-solving and debugging skills.

Preferred Skills:

- Exposure to AWS data architecture and cloud-native data engineering solutions.

- Knowledge of Docker, Kubernetes and infrastructure automation.

- Understanding of data lake and lakehouse architectures.

- Experience implementing automated data quality and monitoring frameworks.