About the job :

We are seeking an experienced Senior Data Platform Engineer with strong expertisein Apache Airflow, AWS Cloud, Database Administration, and Data PipelineOperations.

The ideal candidate will be responsible for deploying, administering,maintaining, and troubleshooting enterprise-scale data platforms hosted across bothon-premises infrastructure and AWS Cloud environments.

The role requires hands-on expertise in Airflow platform management, PostgreSQL, Amazon Redshift,Snowflake, Linux administration, and data pipeline support.

Key Responsibilities :

- Apache Airflow Administration & Platform Engineering Build, deploy, configure, and maintain Apache Airflow environments across on-premises infrastructure and AWS Cloud.

- Perform Airflow platform administration activities including start/stop operations,upgrades, patching, configuration management, and capacity planning.

- Design, deploy, and manage Apache Airflow and Amazon Managed Workflowsfor Apache Airflow (MWAA) environments.

- Provision and manage AWS infrastructure required for Airflow using AWSCloudFormation templates and Infrastructure as Code (IaC) practices.

- Automate Airflow installation, configuration, and maintenance using Shellscripting on Linux/UNIX platforms.

- Manage Airflow DAG deployments through Jenkins CI/CD pipelines and releasemanagement processes.

- Monitor Airflow schedulers, workers, executors, metadata databases, and DAGperformance to ensure platform stability and reliability.

- Troubleshoot and resolve Airflow platform, scheduler, executor, and DAG-relatedissues across development, UAT, and production environments.

- Manage Airflow metadata databases and perform performance tuning activities.

- Administer Airflow hosted on Linux/UNIX servers, ensuring platform security,performance, and availability.

- Ability to develop, maintain, and troubleshoot Python-based Airflow DAGs andworkflow dependencies.

- Support High Availability (HA), Disaster Recovery (DR), backup, restore, andfailover testing activities

- AWS Cloud & Data Platform Management Design and manage MWAA clusters, including environment configuration,scaling, networking, security, and performance optimization.

- Manage AWS services supporting Airflow and data platforms, including EC2,RDS PostgreSQL, ELS, Lambda, Redshift, S3, IAM, CloudWatch, VPC, LoadBalancers, and Auto Scaling services.

- Optimize cloud infrastructure utilization while ensuring high availability, security,and operational efficiency.

- Data Pipeline Operations & Support Monitor, support, and maintain business-critical ETL and data pipelines runningon Airflow.

- Investigate and resolve failed DAGs, performance bottlenecks, and dataprocessing issues.

- Perform root cause analysis for recurring pipeline failures and implementpreventive measures.

- Collaborate with development and data engineering teams to improve pipelinereliability and performance.

- Ensure data platform availability and adherence to SLA requirements.

- Database Administration & Support Administer and support PostgreSQL databases hosted on AWS.

- Manage Amazon Redshift clusters, including monitoring, maintenance,performance tuning, and capacity management.

- Support Snowflake platform operations and coordinate with application teams forissue resolution.

- Perform database query analysis, optimization, and performance tuning.

- Administer and support Amazon RDS databases, primarily PostgreSQL, inproduction, DR, UAT, and development environments.

- Monitor database health, storage utilization, and system performance.

- CI/CD & Automation Design and maintain Jenkins pipelines for Airflow DAG deployment and platformautomation.

- Develop and maintain Shell scripts for Airflow installation, maintenance,monitoring, and operational tasks.

- Implement automation solutions for platform provisioning, applicationdeployment, patching, and operational support.

- Security & Compliance Renew, install, and manage SSL/TLS certificates for Airflow platforms andassociated services.

- Support security vulnerability remediation and compliance initiatives.

- Ensure secure authentication, access management, and encryption standardsare maintained across data platforms.

- Troubleshooting & Incident Management Analyze and resolve complex issues related to Airflow, databases, Linux servers,and data pipelines.

- Participate in incident management, problem management, and changemanagement processes.

- Develop and maintain SOPs, operational runbooks, and technicaldocumentation.

- Provide on-call support for critical production incidents and platform outages.

- Linux/UNIX Administration Manage Linux/UNIX servers hosting Airflow and related platform components.

- Perform system monitoring, log analysis, troubleshooting, performance tuning,and maintenance activities.

- Support platform patching, system upgrades, and infrastructure hardening.

- Required Skills Strong expertise in Apache Airflow administration and troubleshooting.

- Experience deploying and managing Airflow on AWS and on-premises servers.

- Hands-on experience with PostgreSQL and Amazon Redshift administration.

- Working knowledge of Snowflake platform operations and support.

- Strong SQL development, query analysis, and performance tuning skills.

- Experience supporting enterprise ETL/Data Pipeline environments.

- Strong Linux/UNIX system administration experience.

- Knowledge of SSL/TLS certificate management and renewal processes.

- Experience with AWS services such as EC2, RDS, S3, IAM, CloudWatch, andnetworking fundamentals.

- Strong troubleshooting, problem-solving, and analytical skills.

- Preferred Skills Experience with Python scripting and Airflow DAG development.

- Knowledge of CI/CD tools such as Jenkins, GitHub, andCloudFormation/Terraform.

- Experience with monitoring tools such as Splunk, Grafana, Dynatrace, orCloudWatch.

- Familiarity with Data Engineering, Data Warehousing, and Cloud PlatformOperations.

- Experience supporting production environments with stringent SLA requirements.

- Experience 5+ years of overall IT experience.

- 3+ years of hands-on experience with Apache Airflow administration.

- Experience supporting mission-critical data platforms in cloud and hybridenvironments.

- Education Bachelor's or Master's degree in Computer Science, Information Technology,Engineering, or related field