Master Card - Lead Data Engineer - SQL/Python

Job Description :

Responsibilities :

- Design, develop, and maintain scalable and efficient data pipelines, ETL/ELT workflows, and real-time streaming solutions.

- Architect and implement data models, data warehouses, and data lakes to support analytics and business intelligence.

- Lead the integration of data from multiple sources ensuring data quality, consistency, and availability.

- Optimize data pipelines for performance, scalability, and cost efficiency.

- Lead and mentor a team of data engineers, ensuring best practices in coding, architecture, and documentation.

- Collaborate closely with data scientists, analysts, product managers, and business stakeholders to understand data needs.

- Drive engineering excellence through code reviews, technical guidance, and process improvements.

- Work with cloud platforms (AWS/Azure/GCP) to build and maintain reliable and secure data infrastructure.

- Manage orchestration and workflow tools (Airflow, DBT, Luigi, Prefect, etc.).

- Implement monitoring, logging, and alerting for data systems.

- Establish and enforce data quality, data governance, and security standards.

- Ensure compliance with regulatory requirements (e.g., GDPR, HIPAA, SOC 2) where applicable.

- Develop automated testing frameworks to validate data pipelines and datasets.

Required Skills & Qualifications :

- 5- 8 years of experience in data engineering or related roles.

- Strong proficiency in Python, SQL, and distributed data processing frameworks (e.g., Spark, Flink, Hive).

- Hands-on experience with cloud data platforms such as AWS (Glue, Redshift, EMR), Azure (Data Factory, Synapse), or GCP (BigQuery, Dataflow).

- Expertise in ETL/ELT pipeline development, workflow orchestration, and data modeling.

- Strong experience with data warehouse/lake architectures (Snowflake, Databricks, Delta Lake, etc.).

- Solid understanding of CI/CD, Git, containerization (Docker), and DevOps best practices.

- Experience leading engineering efforts or small teams.

Preferred Qualifications :

- Experience with real-time data streaming (Kafka, Kinesis, Pub/Sub).

- Knowledge of machine learning data pipelines and MLOps.

- Exposure to data governance tools and frameworks (e.g., Collibra, Apache Atlas).

- Previous experience in a high-growth or product-oriented environment.