Data Engineer - ETL/BigQuery

Description :

Key Responsibilities :

- Design, implement, and optimize data pipelines for high-volume data ingestion, transformation, and storage on Google Cloud Platform (GCP).

- Implement and manage automated workflows using orchestration tools (e.g., Apache Airflow, Prefect) for scalable data processing.

- Work with GCP services like BigQuery, Dataflow, Pub/Sub, and Composer to manage cloud infrastructure for data storage and analytics.

- Design and implement API integrations (REST, SOAP, GraphQL) for smooth data flow between systems, handling both structured and unstructured data.

- Collaborate with data scientists, analysts, and engineers to meet data needs for analytics, machine learning, and business intelligence.

- Monitor data pipeline performance, troubleshoot issues, and apply optimization techniques to improve efficiency and scalability.

- Work in an agile environment, participating in sprint planning, code reviews, and CI/CD pipeline management for data deployment.

- Write clear and concise documentation for data pipelines and workflows to ensure maintainability and knowledge sharing.

- Ensure data pipelines and infrastructure follow security best practices and comply with relevant regulations (e.g., GDPR, HIPAA).

Required Skills & Qualifications :

- Strong proficiency in Python for data engineering and workflow automation.

- Experience with workflow orchestration tools (e.g., Apache Airflow, Prefect, or similar).

- Hands-on experience with Google Cloud Platform (GCP) or Google BigQuery (GBQ).

- Expertise in big data processing frameworks, such as Apache Spark.

- Experience with API integrations (REST, SOAP, GraphQL) and handling structured/unstructured data.

- Strong problem-solving skills and ability to optimize data pipelines for performance.

- Experience working in an agile environment with CI/CD processes.

- Strong communication and collaboration skills.