Lead Data Engineer/Data Architect

Job Title : Lead Data Engineer / Data Architect (AWS | Databricks)

Location : Bangalore / Chandigarh

Work Mode : Hybrid / Work from Office

Experience : 8 - 12 Years

Role Overview :

We are looking for a Lead Data Engineer / Data Architect with 8+ years of experience in building and optimizing large-scale data platforms, preferably in banking, lending, or financial services.

This is a high-impact leadership role where you will drive the design and development of end-to-end data platforms, modernize BI/AI/ML pipelines, and lead teams to deliver production-grade, scalable, and audit-ready data systems. You will collaborate with global stakeholders and play a key role in transforming data into actionable business insights.

Key Responsibilities :

- Lead the architecture and development of end-to-end data and analytics platforms

- Design and implement scalable data pipelines for BI, analytics, and reporting

- Define and drive data architecture standards, best practices, and governance frameworks

- Set up and manage AWS Glue, data lakes, and cloud-based data pipelines

- Lead Databricks platform setup, governance, and optimization

- Refactor legacy systems (e.g., SAS) into modern Spark-based architectures

- Build and optimize PySpark / Spark SQL ELT pipelines using Delta Lake (ACID compliance)

- Orchestrate workflows using Databricks Workflows / Airflow (if applicable)

- Drive data quality, lineage, validation, and audit frameworks

- Lead and mentor data engineering and BI teams

- Collaborate with business stakeholders to translate requirements into data solutions

- Oversee dashboard development and secure deployment (row-level security)

- Ensure scalability, performance, and production readiness of all data systems

Required Skills :

- Strong expertise in Python, PySpark, and Spark SQL

- Advanced proficiency in SQL and large-scale data processing

- Hands-on experience with Databricks (must-have)

- Deep expertise in AWS Glue and cloud-based data engineering

- Experience with Tableau or similar BI tools

- Knowledge of SAS (migration/refactoring preferred)

- Strong understanding of data governance, data quality, and lineage frameworks

- Proven experience in building scalable data platforms and pipelines

- Exposure to AI/ML data pipelines and modern data architectures

Leadership Expectations :

- Lead end-to-end ownership of data platforms

- Drive technical decision-making and architecture design

- Mentor and guide teams with hands-on involvement

- Ensure alignment of data solutions with business goals and growth

- Work closely with global clients and stakeholders

Preferred Qualifications :

- AWS Certified Data Engineer - Associate

- Experience in banking, lending, or credit risk domains

- Proven experience leading data engineering / BI teams

Why Join :

- Work on high-impact financial data platforms

- Exposure to AI/ML and modern data technologies

- Opportunity to work with global stakeholders

- Strong learning, growth, and leadership opportunities

- Collaborative and innovation-driven culture

Benefits :

- Comprehensive health insurance

- Paid time off and flexible leave policies

- Learning & development programs

- High ownership and growth-driven environment