Roles and Responsibilities :

1. Data Pipeline Development :

- Design, develop, and maintain robust batch and streaming data pipelines using GCP services like BigQuery, Dataflow, Pub/Sub, and DataProc.

- Develop reusable and modular workflows using Cloud Composer (Apache Airflow).

2. Data Transformation and Integration :

- Work with business teams to understand data requirements and build transformation logic using Python, PySpark, and SQL.

- Implement complex business rules and data quality checks to ensure accurate and reliable outputs.

3. Performance Optimization :

- Optimize BigQuery queries for cost and performance using partitioning, clustering, and materialized views.

- Tune pipelines and processing logic for large-scale datasets and real-time streaming use cases.

4. Data Management & Infrastructure :

- Manage and monitor data storage in Google Cloud Storage (GCS).

- Support schema evolution, version control, and metadata handling for structured and semi-structured data.

5. Collaboration & Deployment :

- Collaborate with QA, DevOps, Product, and Compliance teams to align data solutions with business KPIs and regulatory standards (e.g., HIPAA, PCI DSS).

- Use Google Cloud SDK, Git, and CI/CD tools to automate deployments and reduce manual efforts.

6. Compliance & Monitoring :

- Build alerts, retries, and SLAs in Composer DAGs to monitor data pipeline health.

- Ensure data compliance and governance practices are followed throughout the lifecycle.