We are seeking a highly motivated and experienced Data Engineer to join our dynamic team in Hyderabad. In this role, you will be instrumental in designing, building, and maintaining our data infrastructure, ensuring the reliable and efficient delivery of data for critical business insights. You will collaborate closely with data scientists, analysts, and other engineers to understand data requirements and translate them into robust and scalable solutions. Your work will directly impact our ability to make data-driven decisions, improve customer experiences, and drive business growth.

Key Responsibilities :

- Design and develop ETL pipelines to ingest, transform, and load data from various sources into our data warehouse using Spark and PySpark, ensuring data quality and consistency for accurate reporting and analysis.

- Build and maintain data infrastructure on Google Cloud Platform (GCP), including data lakes, data warehouses, and data pipelines, to support the growing data needs of the organization.

- Optimize data storage and processing solutions for performance and scalability, ensuring efficient data access for data scientists and analysts.

- Monitor data pipelines and troubleshoot data quality issues, ensuring data integrity and reliability for business-critical applications.

- Collaborate with data scientists and analysts to understand their data requirements and provide them with the necessary data infrastructure and tools to perform their analyses effectively.

- Implement data governance policies and procedures to ensure data security, privacy, and compliance with regulatory requirements.

Required Skillset :

- Demonstrated ability to design, develop, and maintain ETL pipelines using SQL, Spark, and PySpark for large-scale data processing.

- Proven experience building and managing data infrastructure on Google Cloud Platform (GCP), including data lakes, data warehouses, and data pipelines.

- Strong proficiency in Unix/Linux environments and scripting languages such as Python.

- Excellent understanding of data warehousing concepts, data modeling techniques, and data governance principles.

- Ability to communicate effectively with both technical and non-technical audiences, explaining complex data concepts in a clear and concise manner.

- A Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Ability to work effectively in a fast-paced, collaborative environment, adapting to changing priorities and requirements.