Senior Data Engineer - Python/PySpark

Description :

Senior Data Engineer

Duration : 6 Months with extension

Location : Pune

About the Role

We are looking for a Senior Data Engineer to design, build, and optimize scalable data pipelines and infrastructure. The ideal candidate will have deep expertise in AWS, Snowflake, Terraform, and strong programming skills in SQL, Python, and PySpark.

You will play a key role in collaborating with Data tech lead/lead data engineer while managing data workflows, ensuring data reliability, and implementing best practices for data governance and observability. This work will directly empower data-driven products, personalization, reporting, data science, machine learning, and our overall business success.

Key Responsibilities

- Developing reusable custom frameworks using cloud technologies like AWS , Snowflake and Managed Airflow.

- Design and develop scalable ETL/ELT pipelines using Python, SQL, and PySpark.

- Implement infrastructure-as-code (IaC) using Terraform for cloud-based data environments.

- Develop and maintain data models, transformations, and orchestration workflows.

- Ensure data quality, observability, and lineage tracking across the ecosystem.

- Optimize query performance, storage costs, and compute resources in Snowflake and AWS.

- Implement CI/CD pipelines for data infrastructure automation.

- Monitor and troubleshoot data pipelines, jobs, and cloud infrastructure to maintain SLAs.

Required Skills & Qualifications :

- Strong collaboration and communication skills

- Strong proficiency in SQL and Python for data processing and transformation.

- Hands-on experience with AWS (S3, Glue, Lambda, Redshift, etc.).

- Expertise in Snowflake (performance tuning, Snowflake SQL, schema design).

- Expertise with Terraform for infrastructure automation.

- Proficiency in Airflow or other orchestration tools.

- Understanding of data observability, monitoring, and governance best practices.

- Experience with version control (Git) and CI/CD for data pipelines.

- Strong problem-solving skills and ability to work independently in a fast-paced environment.

- Experience with any code base ETL/ELT tools.

Good to Have :

- Experience in implementing Datamesh and distributed data ownership

- Exposure to Docker, Kafka and Kinesis

- Knowledge of data security and compliance frameworks (GDPR, SOC2, etc.).

- Experience in cost optimization and performance tuning in cloud-based data architectures

- Experience in PySpark