Data Engineer with AI (Python & GCP)

Experience Required : 5 - 9 Years

Location : Open/As per Business Requirement

About the Role :

We are looking for a highly skilled Data Engineer with AI expertise to design, develop, and optimize scalable cloud-based data platforms on Google Cloud Platform (GCP). The ideal candidate will have strong experience in data engineering, cloud-native architectures, ETL/ELT pipelines, real-time data processing, infrastructure automation, and Generative AI solutions. The candidate will work closely with data architects, business stakeholders, and AI teams to build robust data ecosystems that support advanced analytics, machine learning, and GenAI applications.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP.

- Build and optimize ETL/ELT solutions using Python, PySpark, and GCP-native services.

- Develop real-time and batch data processing solutions using Kafka, Pub/Sub, Dataflow, and Dataproc.

- Create and manage BigQuery datasets, scheduled queries, performance tuning, and cost optimization.

- Orchestrate workflows using Airflow, Astronomer, or Cloud Composer.

- Develop and maintain Infrastructure as Code (IaC) using Terraform.

- Implement monitoring, alerting, and operational dashboards for data platforms.

- Collaborate with AI/ML teams to support GenAI and LLM-based applications.

- Integrate enterprise-grade security practices including secrets management and access controls.

- Participate in architecture discussions, code reviews, testing, and production support.

- Build CI/CD pipelines to automate deployment and release processes.

- Ensure adherence to data governance, quality, and security standards.

Must-Have Skills & Experience :

1. Data Engineering :

- 4+ years of hands-on experience in SQL.

- 4+ years of experience in Python and/or PySpark.

- Strong expertise in building scalable ETL/ELT pipelines.

- Experience with data warehousing concepts and performance optimization.

2. Google Cloud Platform (GCP) 3+ Years :

- Strong hands-on experience with BigQuery, Dataflow, Dataproc, Pub/Sub, Scheduled Queries, Cloud Functions, and Monitoring Dashboards.

3. Streaming & Messaging :

- Experience with Apache Kafka for real-time data processing.

4. Infrastructure & Automation :

- 2+ years of experience in Terraform scripting.

- Expertise in implementing Infrastructure as Code (IaC).

5. Workflow Orchestration :

- 2+ years of hands-on experience with Apache Airflow, Astronomer, and Cloud Composer.

Good-to-Have Skills :

- Google Data Fusion

- Tekton Pipelines

- CI/CD Pipelines

- Google Cloud Secret Manager

- Prompt Engineering

- Generative AI (GenAI)

- Large Language Models (LLMs)

- LangGraph, CrewAI

- OpenAI SDK and Google ADK (Agent Development Kit).

Preferred Qualifications :

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

- Experience working in enterprise-scale cloud and data modernization projects.

- Exposure to AI-powered workflows, autonomous agents, and LLM orchestration frameworks.

- Strong problem-solving and analytical skills.

- Ability to work in Agile/Scrum environments.

Communication & Other Requirements :

- Excellent verbal and written communication skills.

- Strong stakeholder management and collaboration abilities.

- Demonstrated hands-on project experience matching the required skill set.

- Candidates should be aligned with the approved budget.

- Notice period should meet project onboarding timelines.