Data Engineer - Python Programming

Key Responsibilities :

Data Pipeline Development :

- Design, build, and maintain highly efficient and scalable ETL/ELT pipelines to process large volumes of structured and unstructured data

- Optimize data workflows for performance, reliability, and cost-effectiveness

- Monitor pipeline health and implement automated alerting and recovery mechanisms

Data Architecture & Infrastructure :

- Build and maintain cloud-based data infrastructure (Azure or AWS, GCP)

- Implement data governance frameworks and ensure compliance with data privacy regulations

- Design schema and data models for optimal query performance and scalability

Data Quality and Integration :

- Develop data validation frameworks and implement quality checks throughout pipelines

- Integrate data from multiple sources including APIs, databases, streaming platforms, and third-party services

- Create and maintain data documentation, lineage tracking, and metadata management

- Troubleshoot data quality issues and implement corrective measures

Collaboration & Support :

- Partner with data scientists, analysts, and business stakeholders to understand data requirements

- Support analytics teams with data access, modeling, and optimization

- Participate in code reviews and maintain high standards for data engineering practices

- Mentor junior team members and contribute to team knowledge sharing

Skills :

- Knowledge of machine learning workflows and MLOps practices

- Familiarity with data visualization tools (Tableau, Looker, Power BI)

- Experience with stream processing and real-time analytics

- Experience with data governance and compliance frameworks (GDPR, CCPA)

- Contributions to open-source data engineering projects

- Relevant Cloud certifications (e.g., Microsoft Certified : Azure Data Engineer Associate, AWS

Certified Data Engineer, Google Cloud Professional Data Engineer).

- Specific experience or certifications in Microsoft Fabric, or Dataiku, Snowflake.

Required Qualifications :

Technical Skills :

- Programming Languages : Proficiency in Python