Role Overview:

We are looking for strong Data Engineering professionals with hands-on expertise in PySpark, AWS and SQL to build, optimize and support scalable ETL/ELT pipelines and cloud-based data solutions. Candidates should be comfortable working with large datasets, performance tuning and end-to-end data pipeline development.

Key Skills:

PySpark, AWS, SQL, Data Engineering, ETL/ELT, Data Pipelines

Key Responsibilities:

- Design, develop and maintain scalable data pipelines and ETL/ELT workflows using PySpark and SQL.

- Build and support AWS-based data engineering solutions using relevant services such as S3, Glue, EMR, Redshift, Lambda or similar services.

- Perform data extraction, transformation, ingestion and integration across structured and semi-structured data sources.

- Write and optimize complex SQL queries for data transformation, validation, reconciliation and performance.

- Process large datasets using Spark/PySpark and apply partitioning, caching and performance-tuning techniques where required.

- Monitor production pipelines, troubleshoot failures and resolve data-quality and performance issues.

- Collaborate with architects, analysts and engineering teams to translate requirements into reliable data solutions.

- Follow coding, testing, deployment, documentation and data-governance best practices.

Required Experience:

- 5 - 10 years of overall experience in Data Engineering / ETL development.

- Strong hands-on development experience in PySpark and Apache Spark concepts.

- Strong SQL skills, including joins, CTEs, window functions, query optimization and data validation.

- Hands-on experience with AWS cloud data services and cloud-based data pipelines.

- Good understanding of data warehousing, dimensional concepts, ETL/ELT and batch/large-scale data processing.

- Ability to independently troubleshoot pipeline, data and performance issues.

Preferred:

- Experience with AWS Glue, EMR, S3, Redshift, Lambda, Step Functions or comparable AWS data services.

- Experience with Airflow or other workflow orchestration tools, CI/CD and Git-based development practices.

- Immediate or short-notice availability is strongly preferred.