Role Overview :

As a Data Engineer specializing in Google Cloud Platform (GCP), you will be instrumental in designing, building, and maintaining scalable and reliable data pipelines.

Your daily tasks will involve extracting, transforming, and loading (ETL) large datasets, ensuring data quality, and optimizing data storage and processing solutions.

You will collaborate closely with data scientists, analysts, and other engineers to understand their data needs and provide them with the infrastructure and tools necessary to derive valuable insights.

Your work will directly impact the business by enabling data-driven decision-making, improving operational efficiency, and driving innovation through advanced analytics.

Key Responsibilities :

- Design and implement robust and scalable data pipelines using GCP services such as BigQuery, Dataproc, and Dataflow to ingest, process, and store large volumes of data.

- Develop and maintain data models and schemas in BigQuery to ensure data consistency and accessibility for analytical purposes.

- Optimize data processing workflows using Apache Spark and PySpark to improve performance and reduce costs.

- Automate data pipeline orchestration and monitoring using Apache Airflow to ensure data reliability and timely delivery.

- Collaborate with data scientists and analysts to understand their data requirements and provide them with the necessary data infrastructure and tools.

- Implement data quality checks and monitoring to ensure data accuracy and reliability.

- Troubleshoot and resolve data pipeline issues to minimize downtime and ensure data availability.

Required Skillset :

- Demonstrated ability to design, build, and maintain data pipelines using Google Cloud Platform (GCP) services, including BigQuery, Dataproc, and Dataflow.

- Proven expertise in using Apache Spark and PySpark for large-scale data processing and transformation.

- Strong proficiency in SQL and experience with relational databases.

- Experience with Apache Airflow for data pipeline orchestration and automation.

- Solid understanding of data modeling principles and data warehousing concepts.

- Excellent problem-solving and analytical skills with the ability to troubleshoot and resolve data-related issues.

- Strong communication and collaboration skills with the ability to work effectively in a team environment.

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Ability to work independently and remotely, demonstrating self-motivation and effective time management.