Posted on: 19/11/2026



ROLE SUMMARY:
We are looking for a hands-on Senior Associate Data Engineer with strong expertise in Python, PySpark, SQL, and Apache Airflow. The role will design and build scalable data pipelines, transform large datasets, orchestrate production workflows, enforce data quality, and independently own technical deliverables from design through deployment and support.
KEY RESPONSIBILITIES:
- Design, develop, and maintain scalable batch and near-real-time data pipelines using Python and PySpark.
- Build and manage Apache Airflow DAGs for scheduling, orchestration, monitoring, retries, and dependency management.
- Develop optimized SQL for data ingestion, transformation, reconciliation, validation, and analytical workloads.
- Process high-volume datasets and implement reusable data-transformation frameworks and utilities.
- Implement data-quality controls, audit checks, lineage, logging, alerting, and operational dashboards.
- Optimize Spark jobs, SQL queries, partitioning, caching, joins, and resource utilization for performance and cost.
- Own technical deliverables across estimation, design, development, testing, deployment, and production support.
- Perform code reviews, debugging, root-cause analysis, documentation, and mentoring of junior engineers.
REQUIRED SKILLS:
- 7 - 10 years of data engineering or software engineering experience with strong hands-on delivery capability.
- Strong Python programming, including OOP, modular design, exception handling, testing, and data-processing libraries.
- Hands-on PySpark experience with DataFrames, Spark SQL, transformations, actions, joins, partitioning, and tuning.
- Advanced SQL including CTEs, window functions, complex joins, aggregations, data validation, and query optimization.
- Strong Apache Airflow experience covering DAG development, operators, sensors, dependencies, scheduling, and monitoring.
- Experience with relational databases, data lakes or warehouses, file formats such as Parquet, and schema evolution.
- Experience with Git, CI/CD pipelines, Linux, test automation, and production support practices.
- Strong Agile/Scrum, problem-solving, communication, collaboration, and stakeholder-management skills.
PREFERRED SKILLS:
- Financial Services / Capital Markets experience, including reconciliation, controls, or regulatory data.
- Azure or AWS data services, Databricks, Snowflake, Docker, Kubernetes, or Kafka exposure.
- Knowledge of data governance, metadata management, security, observability, and cloud cost optimization.
IDEAL CANDIDATE PROFILE:
Hands-on Senior Associate who can independently build and operate reliable data pipelines, optimize distributed processing, ensure data quality, own production outcomes, and mentor team members.
ROLE POSITIONING:
Data Engineering + Pipeline Orchestration + Technical Ownership
Did you find something suspicious?
Posted by
Posted in
Data Engineering
Functional Area
Data Engineering
Job Code
1684551