Data Engineer - Python/ETL

About the Role :

We are seeking an experienced Data Engineer - Python with 58 years of hands-on expertise in building scalable data solutions. The ideal candidate will design, develop, and optimize ETL pipelines, ensure data quality and reliability, and collaborate with cross-functional teams to enable data-driven decision-making.

Key Responsibilities :

- Design, develop, and maintain ETL/ELT pipelines using Python, PySpark, and SQL.

- Build and optimize scalable data pipelines leveraging AWS services (Glue, Lambda, S3, Athena, Step Functions).

- Implement and manage Data Warehousing solutions with strong knowledge of SCD (Type 1, Type 2) and Medallion Architecture.

- Develop efficient data models and ensure partitioning, indexing, and performance optimization in big data environments.

- Ensure high standards of data quality, governance, and security across pipelines and platforms.

- Collaborate with data scientists, analysts, and business stakeholders to translate business requirements into technical solutions.

- Monitor and troubleshoot production pipelines to ensure reliability and scalability.

- Contribute to automation, process improvements, and documentation for data engineering workflows.

Required Skillsets :

- 5-8 years of proven experience in Data Engineering.

- Strong proficiency in Python, PySpark, and SQL for data processing and transformations.

- Solid understanding of ETL/ELT design principles and experience with AWS services (Glue, Lambda, S3, Athena, Step Functions).

- Hands-on experience with Data Warehousing concepts, SCD (Type 1, Type 2), and Medallion Architecture.

- Expertise in data modeling, partitioning strategies, and query performance tuning.

- Strong problem-solving and debugging skills in big data environments.

- Excellent communication skills to explain technical concepts to non-technical stakeholders.

Nice-to-Have Skills :

- Experience with DBT for data transformation and testing.

- Exposure to Databricks and the Lakehouse architecture.

- Familiarity with CI/CD for data pipelines and infrastructure-as-code (Terraform, CloudFormation).

- Knowledge of data security, compliance, and governance best practices.