AgileEngine - Senior Data Engineer - Python

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Python, Pandas, and PySpark.

- Implement and optimize ETL workflows to ensure high data reliability and performance.

- Work extensively with Delta Tables and Parquet file formats for data storage and processing.

- Integrate and automate data workflows using tools like Databricks, DBT, and Airflow.

- Collaborate with data analysts, scientists, and stakeholders to translate business requirements into efficient data models and processes.

- Apply best practices for data governance, testing, and workflow optimization.

- Conduct performance tuning and data quality assurance to ensure smooth data operations.

- Take end-to-end ownership of assigned modules, ensuring timely and high-quality delivery.

- Work in an Agile environment and contribute to continuous process improvement initiatives.

Required Skills & Qualifications :

Must-Have :

- Strong proficiency in Python, including hands-on experience with Pandas and PySpark.

- Solid background in data engineering, ETL pipeline development, and workflow optimization.

- Expertise in handling Delta Tables and Parquet formats.

Good-to-Have :

- Experience with Databricks, Apache Spark, DBT, and Airflow.

- Familiarity with PyTest or DBT testing frameworks.

- Knowledge of advanced Pandas optimizations for large-scale data processing.

- Exposure to ESG or financial data systems.

Soft Skills :

- Strong analytical, problem-solving, and decision-making abilities.

- Excellent written and verbal communication skills for collaboration with global teams.

- Proven ability to take end-to-end ownership of deliverables.

- Agile mindset with a proactive approach to innovation and process improvement.

Educational Qualification :

- Bachelors or Masters degree in Computer Science, Data Engineering, Information Technology, or a related field.