Key Responsibilities :

- Design and develop inbound and outbound file processing using AWS Glue, Python, and PySpark.

- Develop reusable Python modules, utilities, and frameworks for data ingestion, validation, transformation, and error handling.

- Build batch inbound and outbound file processing using Amazon S3 and AWS Glue.

- Configure Amazon EventBridge rules for scheduling and event-based processing.

- Develop AWS Lambda functions for lightweight processing, automation, validation, and integration activities.

- Implement messaging and notification solutions using Amazon SNS and Amazon SQS.

- Configure Amazon CloudWatch logs, metrics, dashboards, and alarms for operational monitoring.

- Develop workflows using AWS Glue Workflows, AWS Step Functions, or equivalent orchestration services.

- Integrate AWS services with relational databases, APIs, file systems, and external applications.

- Implement data ingestion from CSV, XML, JSON, Excel, fixed-length, variable-length, and delimited files.

- Develop data validation, reconciliation, audit, retry, and exception-handling processes.

- Optimize AWS Glue jobs for performance, scalability, and cost efficiency.

- Implement secure access using AWS IAM roles, policies, encryption, and secrets management.

- Support production deployments, incident resolution, root-cause analysis, and performance tuning.

- Collaborate with DevOps teams to implement CI/CD pipelines for AWS Glue jobs and related infrastructure.

- Prepare technical documentation, data flow diagrams, operational procedures, and support documentation.

Qualifications :

Required Skills and Experience :

- 8 - 12 years of experience in Python development and data engineering.

- Strong hands-on experience with AWS Glue and PySpark.

- Excellent knowledge of Python programming, object-oriented programming, and exception handling.

- Experience designing ETL pipelines and data processing frameworks.

- Strong experience with Amazon S3, including bucket structures, lifecycle policies, partitioning, and file formats.

- Experience with Amazon EventBridge for scheduled and event-driven execution.

- Experience developing and deploying AWS Lambda functions using Python.

- Experience with Amazon SNS and SQS for notifications, messaging, and asynchronous processing.

- Experience with Amazon CloudWatch for logging, monitoring, alerting, and troubleshooting.

- Understanding of AWS IAM, security policies, encryption, and least-privilege access.

- Experience working with relational databases such as Oracle, PostgreSQL, MySQL, or Amazon RDS.

- Strong knowledge of SQL and database connectivity from Python or AWS Glue.

- Experience working with REST/SOAP APIs and external system integrations.

- Understanding of JSON, XML, CSV, Excel, and fixed-width file formats.

- Experience with Git and software development best practices.