About the Role:

We are looking for an experienced Senior Data Engineer who can quickly onboard and contribute to the design, development, optimization and support of large-scale data platforms. The ideal candidate should be hands-on, comfortable working with complex data pipelines and distributed systems, and capable of independently troubleshooting and delivering solutions in a fast-paced environment.

Key Responsibilities:

- Design, develop and optimize large-scale data pipelines using Apache Spark and Scala.

- Work with high-volume batch and streaming workloads across multiple data sources.

- Develop and support data processing solutions using Hive, Kafka and GCP services.

- Troubleshoot complex data, performance and production issues with minimal supervision.

- Contribute to cloud migration and platform modernization initiatives.

- Build and maintain ETL workflows and data orchestration pipelines.

- Perform performance tuning and optimization of Spark jobs and large datasets.

- Develop reusable, maintainable and production-ready data solutions.

- Implement and maintain CI/CD pipelines and engineering automation.

- Collaborate with engineering, product, analytics and platform teams to deliver end-to-end solutions.

- Participate in code reviews, testing, production deployments and operational support.

Technical Requirements:

- Strong hands-on expertise in Apache Spark and Scala for building and optimizing large-scale data pipelines.

- Experience working with high-volume batch and streaming workloads across multiple data sources.

- Strong experience with Hive and distributed data processing.

- Hands-on experience with Kafka and event/streaming-based architectures.

- Strong programming experience in Scala and/or Python.

- Experience designing, developing and troubleshooting large-scale ETL/data pipelines.

- Strong understanding of data modelling, data quality, performance tuning and distributed systems.

- Experience working with at least one major cloud platform - GCP, AWS or Azure.

- Experience with BigQuery and GCP services is preferred.

- Experience with ETL workflows and orchestration tools such as Oozie or equivalent.

- Hands-on experience with Git, Jenkins and CI/CD practices.

Preferred Qualifications:

- Strong Google Cloud Platform (GCP) experience.

- Exposure to BigQuery, Google Cloud Storage (GCS), Dataproc and Pub/Sub.

- Experience migrating on-premise Spark/Hive workloads to cloud platforms.

- Experience with large-scale data migration and reconciliation.

- Knowledge of data quality and observability frameworks.

- Experience working with multiple upstream and downstream integrations.

- Exposure to multiple cloud environments.