Infocepts - Data Engineer - AWS Cloud

Description :

Position : Data Engineer AWS.

Purpose of the Position :

You will be a critical member of the InfoCepts Cloud Data Architect Team.

This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, Spark-Scala, PySpark, S3, AWS Lambda etc.

Strong foundation on database concepts and SQL is also required.

Key Result Areas and Activities :

- Study existing technology landscape and understand current data integration framework and do impact assessment for the requirements.

- Develop spark jobs using Scala for new project requirements.

- Enhance existing spark jobs for any ongoing product enhancement.

- Performance tuning of spark jobs, stress testing etc.

- Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache Airflow.

- Responsible for database design process, logical design, physical design, star schema, snowflake schema etc.

- Analyze data processing, integration, modelling and reporting requirements & Define data loading strategies considering volume, data types, frequency and analytics specifications.

- Ensure optimal balance between cost and performance.

- Project documentation, Adheres to Quality guidelines & Schedules.

- Works hand in hand with PM for successful delivery of project and provide Estimation, scoping, scheduling assistance.

- Manage build phase and quality assure code to ensure fulfilling requirements and adhering to Cloud architecture.

- Resolve difficult design and develop issues.

Work and Technical Experience :

- 5+ years of IT experience, with at least 3 years on cloud-related projects.

- Strong SQL skills and database fundamentals.

- Hands-on experience with AWS services : S3, EC2, EMR, Athena, Glue, Lambda.

- Big Data technologies : Hadoop, Hive, Spark, YARN.

- Programming languages : Scala, Python, Shell Scripts.

- Experience in developing ETL workflows with complex transformations.

- Quick learner with adaptability to new AWS and Big Data technologies.

Qualifications :

- Bachelor's degree in computer science, Engineering, or related field.

- Technical certifications or evidence of continued learning.

- 5+ years of IT experience, with at least 3 years on cloud-related projects.

Qualities :

- Hold strong technical knowledge and experience.

- Should have the capability to deep dive and research in various technical related fields.

- Self-motivated and focused on delivering outcomes for a fast-growing team and firm.

- Able to communicate persuasively through speaking, writing, and client presentations.