Role Overview:

We are seeking a seasoned Data Engineer to join our Noida-based engineering team, playing a pivotal role in architecting and scaling our data infrastructure. In this capacity, you will lead the design and implementation of robust data pipelines, ensuring seamless integration and high-performance processing for our global business units.

You will collaborate closely with cross-functional teams, including Data Scientists, Product Managers, and DevOps engineers, to translate complex business requirements into scalable technical solutions. Your work will directly influence our data-driven decision-making capabilities, enabling the organization to derive actionable insights from massive datasets and maintain a competitive edge in the market.

Key Responsibilities:

- Architect and deploy high-throughput data pipelines using Spark and PySpark to process large-scale datasets, ensuring data integrity and availability for downstream analytics.

- Design and maintain real-time streaming architectures using Kafka to facilitate low-latency data ingestion and event-driven processing across our ecosystem.

- Lead the migration and optimization of data workloads on AWS, leveraging cloud-native services to improve cost-efficiency and system performance.

- Containerize data applications using Docker to streamline deployment processes and ensure environment consistency across development, staging, and production.

- Mentor junior engineers and conduct technical design reviews to uphold high engineering standards and foster a culture of continuous improvement within the team.

- Collaborate with stakeholders to identify data bottlenecks and implement automated solutions that enhance overall data quality and operational efficiency.

Required Skillset:

- Demonstrated expertise in building and managing complex data ecosystems with 9 - 13 years of professional experience in data engineering.

- Proven ability to design scalable distributed systems using Spark, PySpark, and Kafka, with a deep understanding of performance tuning and troubleshooting.

- Strong proficiency in AWS cloud services, with the ability to architect secure, reliable, and cost-effective data solutions in a cloud-native environment.

- Hands-on experience with Docker and container orchestration, enabling efficient CI/CD workflows and modular application development.

- Exceptional communication skills, with the ability to articulate complex technical concepts to non-technical stakeholders and lead cross-functional initiatives effectively.

- A proactive mindset with the adaptability to thrive in a hybrid work environment, balancing independent problem-solving with collaborative team efforts.

- A Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field, reflecting a strong foundation in data structures and algorithms.