Lead Data Engineer

Description :

Lead Data Engineer

Experience Required : 10+ years

Location : India (Hyderabad)

Job Summary :

The Lead Data Engineer will coordinate the technical implementation of Databricks data pipelines and manage the data engineering team. This role requires hands-on expertise in building scalable data solutions, optimizing performance, and ensuring quality delivery. You will work closely with the Principal Architect to translate designs into robust implementations while mentoring junior engineers.

Key Responsibilities :

- Lead development of data pipelines for Bronze, Silver, and Gold layers in medallion architecture

- Implement Unity Catalog configurations, schemas, and table migrations

- Coordinate and mentor team of 3-5 data engineers, assign tasks, and conduct code reviews

- Optimize Delta Lake performance including Z-ordering, data skipping, and liquid clustering

- Design and implement complex data transformations using PySpark and Spark SQL

- Implement access control policies, row-level security, and column masking

- Develop automated data quality checks and validation frameworks

- Create and maintain comprehensive technical documentation and code repositories

- Troubleshoot production issues and implement performance tuning solutions

- Coordinate with client teams for requirements clarification and solution delivery

Required Technical Skills :

- 10+ years of hands-on data engineering experience with large-scale data platforms

- Strong expertise in Databricks, Apache Spark, PySpark, and Spark SQL

- Deep knowledge of Delta Lake, Unity Catalog, and Databricks Workflows

- Proficiency in Python for data engineering, automation, and scripting

- Experience with SQL optimization, query tuning, and performance analysis

- Strong understanding of data modeling, schema design, and ETL/ELT patterns

- Experience with cloud data services (AWS S3/Glue/Athena or Azure ADLS/Synapse/Data Factory)

- Knowledge of version control (Git), CI/CD pipelines, and agile methodologies

- Experience with data quality frameworks and monitoring tools

- Strong leadership, communication, and team coordination skills

Preferred Qualifications :

- Databricks Certified Data Engineer Associate or Professional certification

- Experience with DBT (Data Build Tool) for analytics engineering

- Knowledge of streaming data processing (Spark Structured Streaming, Kafka)

- Experience with data governance tools and frameworks

- Previous experience leading offshore delivery teams