Location : Bangalore

Experience : 7+ Years

Role Type : Full-time

Role Overview :

The Principal Data Engineer / Solution Architect will be responsible for designing, architecting, and leading large-scale, cloud-native data platforms. This role combines deep hands-on data engineering expertise with solution architecture, technical leadership, and stakeholder management to deliver scalable, secure, and high-performance data solutions on Azure.

Key Responsibilities :

Architecture & Solution Design :

- Own end-to-end data architecture for batch, streaming, analytics, and AI/ML platforms on Azure

- Translate business requirements into scalable, secure, and cost-effective data solutions

- Define architecture standards, design patterns, and best practices across data platforms

- Lead architecture reviews, design governance, and technical decision-making

- Collaborate with business stakeholders, product owners, and enterprise architects

- Data Engineering & Platform Development

- Design and implement scalable batch and streaming pipelines using Apache Spark, Databricks, and Azure Data Factory

- Lead real-time data processing solutions using Apache Kafka / Azure Event Hubs

- Architect and optimize ETL/ELT workflows for structured and unstructured data

- Build and maintain distributed data systems using Delta Lake, Cassandra, and cloud storage

- Implement and manage Delta Live Tables (DLT) for reliable and testable pipelines

- Design and optimize data models for cloud data warehouses such as Databricks and Snowflake

Cloud, DevOps & MLOps :

- Design and manage CI/CD pipelines using Azure DevOps

- Implement infrastructure automation using Terraform (IaC)

- Integrate Databricks with Azure Synapse, Azure Machine Learning, and other Azure services

- Enable ML model deployment and lifecycle management using MLflow and MLOps practices

- Define disaster recovery, backup, and high-availability strategies

- Performance, Governance & Security

- Apply performance optimization techniques (partitioning, indexing, caching, compression)

- Ensure data quality, governance, security, and compliance across platforms

- Troubleshoot and resolve production issues related to performance, data, and infrastructure

Leadership & Collaboration :

- Lead and mentor junior and mid-level engineers

- Conduct code reviews, design walkthroughs, and technical knowledge sessions

- Drive engineering excellence, scalability, and best practices

- Collaborate with global Agile teams to deliver enterprise-grade solutions

Required Qualifications :

- 7+ years of experience in Data Engineering and cloud-native data platforms

- Strong expertise in Azure, Databricks, Apache Spark, and Kafka/Event Hubs

- Proven experience in solution architecture and technical leadership roles

- Hands-on experience with Delta Lake, Delta Live Tables (DLT), and distributed systems

- Strong programming skills in Python and SQL (Scala is a plus)

- Solid understanding of data modeling, data warehousing, and cloud architectures

- Experience with CI/CD pipelines, Azure DevOps, and Terraform

- Strong understanding of data governance, security, and performance tuning

- Excellent communication and stakeholder management skills

Preferred Qualifications :

- Experience with Snowflake or similar cloud data warehouses

- Exposure to Cassandra or other NoSQL databases

- Azure and/or Databricks certifications