Happiest Minds Technologies - Lead Data Engineer - Azure Databricks

Role : Senior Data Engineer (Databricks + CosmosDB)

Location : Bangalore (Hybrid - 3 days from client office)

Experience : 8-10 Years

Job Summary :

We are seeking a highly skilled Senior Data Engineer with deep expertise in Azure Databricks and Cosmos DB. The ideal candidate will be responsible for designing, building, and optimizing scalable, cloud-native data platforms and data pipelines.

This role requires a hands-on approach to developing high-performance data solutions that support analytics, AI/ML initiatives, real-time processing, and enterprise reporting requirements while ensuring data quality, security, and operational excellence.

Responsibilities :

1. Data Engineering & Pipeline Development :

- Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and Apache Spark.

- Build robust ETL/ELT processes to ingest, transform, and curate large volumes of structured and unstructured data.

- Develop reusable data engineering frameworks and accelerators.

- Optimize data processing jobs for performance, scalability, and cost efficiency.

2. Databricks Platform Development :

- Build and maintain data solutions using Azure Databricks.

- Develop notebooks and workflows using Apache Spark RDD, Apache Spark SQL, and Python.

- Implement Delta Lake architectures for reliable and efficient data management.

- Configure and optimize Databricks clusters and workloads.

- Support CI/CD implementation for Databricks deployments.

3. Cosmos DB Development & Optimization :

- Design and implement scalable solutions using Azure Cosmos DB (NoSQL).

- Model data effectively for NoSQL workloads.

- Optimize partition strategies, indexing policies, and query performance.

- Manage data replication, consistency models, and throughput provisioning.

- Integrate Cosmos DB with analytics and downstream systems.

4. Azure Data Platform Integration :

- Develop and integrate end-to-end solutions using Azure services such as Azure Data Factory, Azure Functions, and Azure Logic Apps.

5. Real-Time & Streaming Data Processing :

- Build streaming data pipelines using Spark Structured Streaming.

- Process event-driven workloads from Azure Event Hubs, Kafka Stream, or IoT sources.

- Implement near real-time analytics solutions integrated with Cosmos DB and Databricks.

6. DevOps & CI/CD :

- Implement Devsecops practices for Data Engineering solutions.

- Automate deployment pipelines using Azure DevOps or GitHub Actions.

- Manage source control and release processes.

Mandatory Skills :

- Azure Databricks

- Apache Spark RDD

- Apache Spark SQL

- Delta Lake

- Azure Cosmos DB (NoSQL)

Preferred Skills :

- Delta Live Tables (DLT)

- Azure Databricks

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 8-10 years of experience in data engineering or related roles.

- Strong understanding of cloud engineering principles and modern data architectures.

- Excellent problem-solving skills and ability to work in a fast-paced environment.