Apache Airflow Developer - Autosys/Markit EDM

About the Role :

We are looking for an experienced Apache Airflow Developer with strong expertise in workflow orchestration, AWS cloud services, and Python development.

The role involves migrating legacy Autosys batch jobs to Apache Airflow and ensuring seamless integration with cloud and downstream systems.

Key Responsibilities :

- Analyse existing Autosys job definitions, workflows, and dependencies.

- Design, develop, and implement equivalent Airflow DAGs for robust orchestration.

- Integrate Airflow with MarkitEDM and other internal/downstream systems.

- Ensure scheduling, monitoring, logging, and alerting meet defined SLAs.

- Collaborate with cloud engineering and application teams for smooth deployment on AWS.

- Document migration logic, implementation details, and operational procedures.

- Provide support during UAT, performance testing, and production rollout.

- Troubleshoot issues and optimize workflows for stability and performance.

Required Skills & Experience :

- Strong hands-on experience with Apache Airflow (DAG development, scheduling, operators, hooks).

- Good understanding of Autosys job scheduling and batch processing.

- Experience working with MarkitEDM or similar enterprise data management tools.

- Proficiency in Python scripting for orchestration, automation, and data workflows.

- Solid working experience with AWS services such as ECS, S3, Lambda, CloudWatch, IAM, etc.

- Good understanding of CI/CD pipelines and DevOps best practices.

- Strong analytical, debugging, and problem-solving skills.

- Excellent communication and documentation skills.

Desirable Skills:

- Experience in financial services, capital markets, or large-scale enterprise data environments.

- Knowledge of Airflow deployment in containerized environments (Docker, Kubernetes).

- Familiarity with audit, logging, compliance, and operational controls in batch processing