Lead Data Engineer - Python/PySpark

Description :

- Build and manage data workflows using Informatica, Databricks, AWS Glue, Snowflake

- Work across multi-cloud platforms (AWS, Azure, GCP, Microsoft Fabric)

- Manage and optimize databases such as Redshift, SQL Server, AWS RDS

- Orchestrate workflows using Apache Airflow

- Implement real-time streaming solutions using Apache Kafka

- Deploy and manage containerized applications using Kubernetes

- Automate processes using Unix shell scripting

- Implement CI/CD pipelines and ensure data governance

- Collaborate with BI teams for Tableau, Looker, Power BI

- Troubleshoot and optimize performance of pipelines and queries

- Maintain technical documentation

Required Skills & Qualifications :

- Bachelors/Masters degree in Computer Science or related field

- 7+ years of experience in Data Engineering

- Strong skills in Python, PySpark, Scala, SQL

- Experience with Informatica, Databricks, AWS Glue

- Expertise in Snowflake & Redshift

- Experience in AWS / Azure / GCP

- Experience with Apache Airflow

- Knowledge of Apache Kafka & Kubernetes

- Experience with CI/CD pipelines

- Exposure to BI tools (Tableau, Looker, Power BI)