Data Engineer - Python/SQL

Description :

Job Title : Data engineer GCP

Experience needed : 5 to 8 years

Job Purpose :

We are looking for a skilled and proactive Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The ideal candidate will have strong experience in Google Cloud Platform (GCP) technologies, infrastructure as code, and modern data engineering practices, with a focus on delivering reliable and high-quality data systems.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and data processing systems.

- Work with GCP data services such as BigQuery, Dataflow, Pub/Sub, and Cloud Functions.

- Implement and manage cloud infrastructure using Terraform (Infrastructure as Code).

- Build and maintain CI/CD pipelines using Azure DevOps for automated deployment and testing.

- Collaborate with cross-functional teams to understand data requirements and deliver solutions.

- Ensure data quality, reliability, and performance optimization across pipelines.

- Develop and optimize data transformations using ANSI SQL and Python.

- Work with GKE/Kubernetes for containerized deployments.

- Leverage GCP Vertex AI where applicable for advanced data and ML workflows.

- Follow Agile/Scrum methodologies and actively participate in sprint planning, stand-ups, and retrospectives.

- Use tools like Jira and Confluence for tracking tasks and documentation.

- Monitor, troubleshoot, and resolve data pipeline issues proactively.

- Maintain version control and collaborate using Git.

- Continuously improve data engineering practices, tools, and processes.

Required Skills & Qualifications :

- Strong experience in cloud-based data architecture and platforms (preferably GCP).

- Hands-on experience with BigQuery, Dataflow, Pub/Sub, and Cloud Functions.

- Proficiency in Python for data engineering and pipeline development.

- Strong knowledge of ANSI SQL.

- Experience with Terraform for infrastructure management.

- Experience with Azure DevOps CI/CD pipelines.

- Working knowledge of GKE/Kubernetes.

- Solid understanding of data pipeline design, development, and testing.

- Familiarity with Git for version control.

- Experience working in Agile/Scrum environments.

- Familiarity with Jira and Confluence.

- Strong problem-solving skills with a proactive and ownership-driven mindset.

- Excellent communication and collaboration abilities.