Role Overview :

As a Principal AWS Data Engineer, you will be instrumental in architecting, developing, and optimizing our next-generation data platforms on Amazon Web Services. This role demands a seasoned professional capable of tackling complex data challenges, from designing scalable data lakes and warehouses to implementing robust ETL/ELT pipelines using PySpark and AWS Glue. You will collaborate closely with data scientists, business analysts, and product managers to translate intricate business requirements into high-performance, cost-effective data solutions. Your contributions will directly empower data-driven decision-making across the organization, enhancing product capabilities and driving significant business outcomes.

Key Responsibilities :

- Architect and implement highly scalable, reliable, and secure data pipelines and data solutions on AWS, leveraging services like S3, Glue, Redshift, Athena, Lambda, and Kinesis to support diverse analytical and operational needs.

- Develop, optimize, and maintain complex ETL/ELT processes using PySpark to ingest, transform, and load large volumes of structured and unstructured data from various internal and external sources.

- Lead the design and implementation of data models for data warehouses and data lakes, ensuring data quality, consistency, and accessibility for downstream consumption by analytics and machine learning teams.

- Collaborate proactively with cross-functional teams, including data scientists, product owners, and software engineers, to understand data requirements and deliver innovative data products that drive business value.

- Establish and enforce best practices for data governance, data security, data privacy, and compliance within the AWS data ecosystem, ensuring adherence to organizational and regulatory standards.

- Mentor and guide junior data engineers, fostering a culture of technical excellence, continuous learning, and innovation within the data engineering team.

- Troubleshoot and resolve complex data-related issues, optimize data pipeline performance, and ensure the high availability and reliability of critical data infrastructure.

Required Skillset :

- Demonstrated mastery in designing, building, and managing large-scale data platforms and pipelines on Amazon Web Services (AWS), with deep expertise in AWS Glue, S3, Redshift, Athena, EMR, Lambda, and Kinesis.

- Exceptional proficiency in PySpark for developing efficient and scalable data processing applications, coupled with strong programming skills in Python.

- A profound understanding of data warehousing principles, dimensional modeling, ETL/ELT methodologies, and modern data architecture patterns.

- Solid command of SQL for data manipulation, querying, and optimization across various database systems, including relational and NoSQL databases.

- Proven ability to lead technical initiatives, drive architectural decisions, and provide technical guidance to a team of engineers.

- Outstanding problem-solving capabilities and analytical acumen, with a track record of delivering robust solutions to complex data challenges.

- Excellent verbal and written communication skills, enabling effective collaboration with technical and non-technical stakeholders.

- A Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field from a reputable institution.