Hyderabad & Bengaluru - (1 days at office required in a week).

We are looking for an experienced, hands-on Databricks Architect and Data Engineering Manager with 10+ years of experience in designing and delivering scalable data engineering solutions.

The ideal candidate should be a strong individual contributor and technical lead, capable of driving solution architecture, developing reusable Python-based frameworks, designing complex data models, and leading a team of Data Engineers.

Key Technical Skills & Responsibilities :

Azure Databricks Architecture :

- Strong hands-on experience with Azure Databricks Lakehouse Architecture, Delta Lake, Unity Catalog, Databricks Workflows/Jobs, and Databricks Asset Bundles (DAB).

- Experience designing scalable, secure, and production-ready Databricks solutions.

Advanced Python & PySpark :

- Expert-level proficiency in Python, PySpark, and SQL, with the ability to independently develop, review, troubleshoot, and optimize complex data engineering solutions.

- Must be comfortable contributing directly to code development.

Reusable Python Framework Development :

- Proven experience architecting and developing modular, reusable, configuration-driven Python frameworks for data ingestion, transformation, validation, orchestration, and automation.

- Strong understanding of metadata-driven design, configurable business rules, extensible components, and reusable execution engines.

Data Modeling & Architecture :

- Strong experience in conceptual, logical, and physical data modeling, relational and dimensional modeling, schema design, source-to-target mappings, CDC, SCD Type 2, and historical data processing.

- Ability to independently design complex data models and develop SQL DDL scripts.

Lakehouse Data Engineering :

- Experience designing and implementing Bronze, Silver, and Gold data architectures, Delta tables, scalable ETL/ELT pipelines, incremental processing, and enterprise data transformation frameworks.

Real-Time & Batch Data Processing :

- Hands-on experience building batch and streaming ingestion pipelines using Spark Structured Streaming, Auto Loader, Azure Event Hubs, Delta Change Data Feed (CDF), and related technologies.

Pipeline Reliability & Data Quality :

- Ability to architect production-grade pipelines with robust exception handling, retries, replay, reconciliation, audit logging, data quality validations, and failure recovery mechanisms.

Performance & Cost Optimization :

- Strong experience optimizing Spark workloads, SQL queries, Delta tables, cluster configurations, job execution, and overall Databricks compute costs.

DevOps & Engineering Best Practices :

- Experience with Git, CI/CD, Databricks Asset Bundles, automated deployments, unit testing, code quality standards, and reusable engineering practices.

Pharmaceutical / Life Sciences Domain :

- Good understanding of pharmaceutical industry data and business processes, preferably across Commercial, Sales, Marketing, Patient Services, Clinical, or R&D domains.

- Exposure to pharma datasets, data integration challenges, industry-specific business rules, and data governance requirements.

Technical Leadership & Team Management :

- Ability to lead and mentor Data Engineers, provide technical direction, conduct code reviews, define engineering standards, resolve complex technical challenges, and ensure high-quality project delivery.

Solution Design & Stakeholder Collaboration :

- Ability to translate functional and business requirements into scalable technical architectures, prepare solution design documents, define integration patterns, and collaborate with business stakeholders, architects, DevOps, QA, and engineering teams.

Mandatory Qualifications :

- 10+ years of overall IT experience, with significant experience in Data Engineering and Data Architecture.

- Strong hands-on experience in Azure Databricks, Python, PySpark, and SQL.

- Proven experience architecting and developing reusable, metadata-driven Python frameworks.

- Strong expertise in enterprise data modeling and complex data pipeline design.

- Experience leading Data Engineering teams while remaining actively involved in hands-on development.

- Working knowledge of Pharmaceutical / Life Sciences data and business processes.