Lead Data Engineer - Azure Databricks

Posting title : Lead Data Engineer (Databricks)

Experience : 7+ Years

Location : Pune/ Ahmedabad

Work mode : Hybrid

Primary skills : Databricks, Data Modelling/ Data Lakehouse, Gen AI, SQL, Python/Scala, Azure Data Factory (ADF)/AWS Glue, Docker, Team Lead

Qualification : Any Technical Degree

Roles & Responsibilities :

- Lead the design, development, and optimization of data solutions using Databricks, ensuring they are scalable, efficient, and secure.

- Collaborate with cross-functional teams to gather and analyse data requirements, translating them into robust data architectures and solutions.

- Develop and maintain ETL pipelines, leveraging Databricks and integrating with Azure Data Factory as needed.

- Implement machine learning models and advanced analytics solutions, incorporating Generative AI to drive innovation.

- Ensure data quality, governance, and security practices are adhered to, maintaining the integrity and reliability of data solutions.

- Provide technical leadership and mentorship to junior engineers, fostering an environment of learning and growth.

- Stay updated on the latest trends and advancements in data engineering, Databricks, Generative AI, and Azure Data Factory to continually enhance team capabilities.

Required Qualifications :

- Bachelors or masters degree in computer science, Information Technology, or a related field.

- 7+ to 10 years of experience in data engineering, with a focus on Databricks.

- Proven expertise in building and optimizing data solutions using Databricks and integrating with Azure Data Factory/AWS Glue.

- Proficiency in SQL and programming languages such as Python or Scala.

- Strong understanding of data modelling, ETL processes, and Data Warehousing/Data Lakehouse concepts.

- Familiarity with cloud platforms, particularly Azure, and containerization technologies such as Docker.

- Excellent analytical, problem-solving, and communication skills.

- Demonstrated leadership ability with experience mentoring and guiding junior team members.

Preferred Skills :

- Experience with Generative AI technologies and their applications.

- Familiarity with other cloud platforms, such as AWS or GCP.

- Knowledge of data governance frameworks and tools.