Data Engineer - Python/Scala

We are looking for an experienced GCP Data Engineer with a minimum of 5+ years of professional experience in data engineering, cloud-based data solutions, and large-scale distributed systems. This role is fully remote and requires a hands-on professional who can design, build, and optimize data pipelines and solutions on Google Cloud Platform (GCP).

Key Responsibilities :

- Architect, design, and implement highly scalable data pipelines and ETL workflows leveraging GCP services.

- Develop and optimize data ingestion, transformation, and storage frameworks to support analytical and operational workloads.

- Work extensively with BigQuery, Dataflow, Pub/Sub, Dataproc, Data Fusion, Cloud Composer, and Cloud Storage to design robust data solutions.

- Create and maintain efficient data models and schemas for analytical reporting, machine

learning pipelines, and real-time processing.

- Collaborate closely with data scientists, analysts, and business stakeholders to understand requirements and convert them into technical data solutions.

- Implement best practices for data governance, security, privacy, and compliance across the entire data lifecycle.

- Monitor, debug, and optimize pipeline performance ensuring minimal latency and high

throughput.

- Design and maintain APIs and microservices for data integration across platforms.

- Perform advanced data quality checks, anomaly detection, and validation to ensure data

accuracy and consistency.

- Implement CI/CD for data engineering projects using GCP-native DevOps tools.

- Stay updated with emerging GCP services and industry trends to continuously improve existing solutions.

- Create detailed documentation for data processes, workflows, and standards to enable smooth knowledge transfer.

- Support the migration of on-premise data systems to GCP, ensuring zero downtime and efficient cutover.

- Automate repetitive workflows, deployment processes, and monitoring systems using Python, Shell scripting, or Terraform.

- Provide mentoring and technical guidance to junior data engineers in the team.

Required Skills & Experience :

- 5+ years of experience in data engineering with a strong focus on cloud-based data solutions.

- Hands-on expertise with Google Cloud Platform (GCP) and services including BigQuery, Dataflow, Pub/Sub, Dataproc, Data Fusion, Cloud Composer, and Cloud Storage.

- Strong proficiency in SQL, including query optimization, performance tuning, and working with large datasets.

- Advanced programming skills in Python, Java, or Scala for building data pipelines.

- Experience with real-time data streaming frameworks such as Apache Kafka or Google Pub/Sub.

- Solid knowledge of ETL/ELT processes, data modeling (star/snowflake), and schema design for both batch and streaming use cases.

- Proven track record of building data lakes, warehouses, and pipelines that can scale with

enterprise-level workloads.

- Experience integrating diverse data sources including APIs, relational databases, flat files, and

unstructured data.

- Knowledge of Terraform, Infrastructure as Code (IaC), and automation practices in cloud environments.

- Understanding of CI/CD pipelines for data engineering workflows and integration with Git,

Jenkins, or Cloud Build.

- Strong background in data governance, lineage, and cataloging tools.

- Familiarity with machine learning workflows and enabling ML pipelines using GCP services is

an advantage.

- Good grasp of Linux/Unix environments and shell scripting.

- Exposure to DevOps practices and monitoring tools such as Stackdriver or Cloud

Logging/Monitoring.

- Excellent problem-solving, debugging, and analytical skills with the ability to handle complex

technical challenges.

- Strong communication skills with the ability to work independently in a remote-first team

environment.

Nice-to-Have Skills :

- Experience with multi-cloud or hybrid environments (AWS/Azure alongside GCP).

- Familiarity with data visualization platforms such as Looker, Tableau, or Power BI.

- Exposure to containerization technologies such as Docker and Kubernetes.

- Understanding of big data processing frameworks like Spark, Hadoop, or Flink.

- Prior experience in industries with high data volume such as finance, retail, healthcare, or

telecom.

Educational Background :

- Bachelors or Masters degree in Computer Science, Information Technology, Data Engineering, or a related field.

- Relevant GCP certifications (e.g., Professional Data Engineer, Professional Cloud Architect) will be highly preferred.

Why Join Us?

- Opportunity to work on cutting-edge cloud data projects at scale.

- Fully remote working environment with flexible schedules.

- Exposure to innovative data engineering practices and advanced GCP tools.

- Collaborative team culture that values continuous learning, innovation, and career growth.