Lead Data Engineer - Apache Spark

Description :

Responsibilities :

- Lead the design and development of end-to-end data pipelines using Apache Spark (Batch and Streaming).

- Architect and implement real-time data ingestion frameworks using Kafka.

- Build scalable ETL/ELT workflows to support analytics, reporting, and data science initiatives.

- Develop and maintain data models (conceptual, logical, physical) for enterprise data platforms.

- Optimize Spark jobs for performance, reliability, and scalability.

- Ensure data quality, governance, and security across all data flows.

- Drive best practices for coding standards, CI/CD, and cloud-based data architecture.

- Mentor junior engineers and collaborate with cross-functional teams (Data Science, DevOps, Product).

- Troubleshot complex data processing issues and provided technical leadership during incidents.

Requirements :

- 7+ years of hands-on experience in Data Engineering.

- Strong working knowledge of Spark, Python, SQL, and API Integration frameworks is a must.

- Working experience in Modern data architecture and modeling concepts, including Cloud data lakes, data warehouses, and data marts.

- Familiarity with dimensional modeling, star schemas, and real-time/batch ETL pipelining, including experience with data streaming (Kafka).

- In-depth experience with Kafka for real-time data ingestion and streaming.

- Strong proficiency in SQL (analytical, performance tuning).

- Solid understanding of data modeling principles (OLTP, OLAP, dimensional modeling, star/snowflake schemas).

- Experience building large-scale distributed data processing systems.

- Hands-on experience with cloud platforms such as AWS / Azure / GCP (any).

- Knowledge of CI/CD, containerization (Docker), and orchestration tools (Airflow, Jenkins, etc. ).

- Strong problem-solving, debugging, and leadership skills.

- Bachelor's or Master's degree in Computer Science, Engineering, or related field.

Preferred Qualifications :

- Experience with Delta Lake, Lakehouse architecture, or cloud-native data platforms.

- Exposure to NoSQL databases (Cassandra, MongoDB, DynamoDB).

- Knowledge of data governance, metadata management, and cataloging tools.

- Prior experience leading a technical team or project.