Data Engineer - ETL/Data Pipeline

Job Responsibilities :

- Implement data ingestion from structured and unstructured data sources into AWS Data Lake or data warehouse systems.

- Work with large-scale datasets to ensure efficient data transformation and loading across various AWS storage layers (S3, Redshift, RDS).

- Write complex SQL queries for data validation, transformation, and reporting.

- Develop and maintain metadata, data lineage, and logging for data quality and traceability.

- Optimize data workflows for performance, scalability, and cost efficiency.

- Collaborate with Data Scientists, Analysts, and Business teams to understand data needs and deliver robust solutions.

- Ensure data governance, security, and compliance in cloud data pipelines.

- Perform unit and integration testing, and support deployment in lower and production environments.

Technical Skills :

- Strong hands-on experience with AWS Glue (Jobs, Crawlers, Workflows, Dynamic Frames)

- Experience with AWS services : S3, Redshift, Lambda, Athena, Step Functions, CloudWatch, IAM

- Strong experience in Apache Spark (PySpark preferred) for distributed data processing

- Advanced skills in Python (data manipulation, scripting, exception handling, performance tuning)

- Strong in SQL writing complex queries, stored procedures, and query optimization

- Experience in building and orchestrating ETL/ELT pipelines

- Familiarity with schema design, data partitioning, compression formats (Parquet, ORC, Avro)

- Hands-on with data cataloging, metadata management, and data warehousing concepts

Additional Skills (Nice to Have) :

- Experience with Airflow, AWS Step Functions, or other orchestration tools

- Exposure to data warehousing concepts and experience with Redshift or Snowflake

- Experience working in Agile Scrum environments

- Understanding of data security, privacy, and compliance standards (GDPR, HIPAA, etc.)

Personal Qualities :

- Strong communication and collaboration skills, with experience in stakeholder interaction

- Ability to work independently and within a cross-functional team

- Detail-oriented, with a commitment to delivering high-quality, reliable solutions