As a GCP Data Engineer, you will be responsible for architecting and maintaining robust data pipelines that serve as the backbone for our clients' analytical capabilities. You will work closely with cross-functional teams, including data scientists and product managers, to translate business requirements into high-performance data solutions. Your work will directly influence the efficiency of data processing, ensuring that stakeholders have reliable, real-time access to critical business intelligence, ultimately driving data-informed decision-making across the organization.

Key Responsibilities :

- Design and implement scalable data pipelines using Apache Airflow to automate complex ETL workflows for diverse business units.

- Optimize data storage and query performance within BigQuery to support large-scale analytical reporting and dashboarding requirements.

- Develop high-performance data processing applications using Apache Spark to handle massive datasets with high velocity and accuracy.

- Implement CI/CD best practices to ensure seamless deployment and version control of data infrastructure code across development and production environments.

- Collaborate with engineering teams to troubleshoot data quality issues and improve the overall reliability of the data ecosystem.

- Integrate Python-based services with Google Cloud Platform components to enhance automation and system interoperability.

Required Skillset :

- Demonstrated expertise in building and managing end-to-end data pipelines within the Google Cloud Platform ecosystem.

- Proficiency in writing clean, maintainable Python code for data transformation and automation tasks.

- Strong command over BigQuery for complex data modeling and performance tuning of large-scale analytical queries.

- Proven ability to orchestrate workflows using Apache Airflow and process distributed data using Apache Spark.

- Experience in setting up and managing CI/CD pipelines to streamline deployment cycles in a cloud-native environment.

- Excellent communication skills with the ability to articulate technical concepts to non-technical stakeholders and work effectively in a hybrid team setup.

- Candidates must possess 4 to 7 years of relevant experience in data engineering roles.

- A degree in Computer Science, Information Technology, or a related quantitative field is preferred.