Key Responsibilities :

- Design and implement scalable and robust data pipelines using Apache Spark and Kafka to ingest, process, and transform large datasets.

- Build and maintain our data platform infrastructure on AWS, leveraging Kubernetes for container orchestration and management.

- Develop and implement MLOps best practices to streamline the deployment and monitoring of machine learning models built with TensorFlow and PyTorch.

- Establish and enforce data governance policies and procedures to ensure data quality, integrity, and compliance.

- Implement data security measures to protect sensitive data and prevent unauthorized access.

- Collaborate with data scientists and analysts to understand their data needs and provide them with the tools and infrastructure they need to be successful.

- Mentor and guide junior engineers on best practices for data platform development and operations.

- Proactively identify and resolve performance bottlenecks and scalability issues in the data platform.

Required Skillset :

- Demonstrated ability to design, develop, and maintain large-scale data platforms using technologies such as Apache Spark, Kafka, and Kubernetes.

- Proven expertise in building and deploying data infrastructure on AWS, including experience with services such as EC2, S3, EMR, and RDS.

- Strong understanding of MLOps principles and experience with deploying and monitoring machine learning models using TensorFlow and PyTorch.

- Deep knowledge of data governance principles and practices, including data quality, data security, and data compliance.

- Excellent communication and collaboration skills, with the ability to effectively communicate technical concepts to both technical and non-technical audiences.

- Bachelor's or Master's degree in Computer Science or a related field.

- Ability to work independently and as part of a team in a fast-paced, agile environment.

- 8 -15 years of relevant experience in data platform engineering.