Description :

Role Overview :

We are seeking a highly skilled and experienced AWS Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining scalable and reliable data pipelines and infrastructure on the AWS cloud. You will collaborate closely with data scientists, analysts, and other engineers to deliver high-quality data solutions that drive business insights and improve decision-making. Your work will directly impact our ability to leverage data for innovation, optimization, and enhanced customer experiences.

Key Responsibilities :

- Design and implement robust and scalable data pipelines using AWS services such as S3, Glue, Lambda, and other relevant technologies to ingest, process, and transform large datasets for various analytical and machine learning purposes.

- Develop and maintain data models and schemas in Snowflake DB to ensure data quality, consistency, and accessibility for downstream users and applications.

- Build and optimize data processing workflows using Databricks to perform complex data transformations, feature engineering, and model training at scale.

- Implement data governance and security best practices to ensure data privacy, compliance, and integrity across all data systems and pipelines.

- Collaborate with data scientists and analysts to understand their data requirements and provide them with the necessary data infrastructure and tools to perform their analyses and build machine learning models.

- Monitor and troubleshoot data pipelines and systems to ensure their reliability, performance, and availability, and implement proactive measures to prevent data incidents.

- Automate data pipeline deployment and management using infrastructure-as-code principles and DevOps practices to improve efficiency and reduce manual effort.

- Evaluate and recommend new AWS services and technologies to improve our data infrastructure and capabilities, and stay up-to-date with the latest trends and best practices in the data engineering field.

Required Skillset :

- Demonstrated ability to design, build, and maintain scalable and reliable data pipelines on AWS using services such as S3, Glue, Lambda, and other relevant technologies.

- Proven expertise in data modeling and schema design in Snowflake DB, with a strong understanding of data warehousing principles and best practices.

- Hands-on experience in building and optimizing data processing workflows using Databricks, including Spark SQL, Python, and other relevant tools.

- Solid understanding of data governance and security best practices, with the ability to implement data privacy, compliance, and integrity measures.

- Excellent communication and collaboration skills, with the ability to work effectively with data scientists, analysts, and other engineers.

- Strong problem-solving and analytical skills, with the ability to troubleshoot data pipeline issues and identify opportunities for improvement.

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Adaptable to work in a remote/hybrid environment and collaborate effectively with distributed teams.