Data Engineer - ETL/Python

Responsibilities :

- Design, build, and maintain scalable ETL pipelines and data workflows using AWS Glue, Spark, and Python

- Implement data ingestion from structured and unstructured data sources into AWS Data Lake or data warehouse systems

- Work with large-scale datasets to ensure efficient data transformation and loading across various AWS storage layers (S3, Redshift, RDS)

- Write complex SQL queries for data validation, transformation, and reporting

- Develop and maintain metadata, data lineage, and logging for data quality and traceability

- Optimize data workflows for performance, scalability, and cost efficiency

- Collaborate with Data Scientists, Analysts, and Business teams to understand data needs and deliver robust solutions

- Ensure data governance, security, and compliance in cloud data pipelines

- Perform unit and integration testing, and support deployment in lower and production environments

- Contribute to best practices in cloud data architecture, DevOps, and CI/CD automation for data pipelines

Required Technical Skills :

Cloud & Data Engineering :

- Strong hands-on experience with AWS Glue (Jobs, Crawlers, Workflows, Dynamic Frames)

- Experience with AWS services: S3, Redshift, Lambda, Athena, Step Functions, CloudWatch, IAM

- Strong experience in Apache Spark (PySpark preferred) for distributed data processing

Programming & Scripting :

- Strong in SQL writing complex queries, stored procedures, and query optimization

Data Management & Workflow :

- Experience in building and orchestrating ETL/ELT pipelines

- Familiarity with schema design, data partitioning, compression formats (Parquet, ORC, Avro)

- Hands-on with data cataloging, metadata management, and data profiling

Preferred Skills (Nice to Have) :

- Knowledge of DevOps for Data CI/CD pipelines using tools like Git, Jenkins, CodePipeline

- Exposure to data warehousing concepts and experience with Redshift or Snowflake

- Experience working in Agile Scrum environments

- Understanding of data security, privacy, and compliance standards (GDPR, HIPAA, etc.)

Soft Skills :

- Excellent analytical thinking and problem-solving abilities

- Ability to work independently and within a cross-functional team

- Detail-oriented, with a commitment to delivering high-quality, reliable solutions