Databricks Engineer - Python/SQL

Urgent Requirement Databricks Engineer AWS

Location : Hyderabad

Experience : 8+ Years

Role Overview :

- Lead the design, development, and optimization of scalable and secure data pipelines using AWS services such as Glue, S3, Lambda, EMR, and Databricks Notebooks, Jobs, and Workflows.

- Oversee the development and maintenance of data lakes on AWS Databricks, ensuring performance and scalability.

- Build and manage robust ETL/ELT workflows using Python and SQL, handling both structured and semi-structured data.

- Implement distributed data processing solutions using Apache Spark/PySpark for large-scale data transformation.

- Collaborate with cross-functional teams including data scientists, analysts, and product managers to ensure data is accurate, accessible, and well-structured.

- Enforce best practices for data quality, governance, security, and compliance across the entire data ecosystem.

- Monitor system performance, troubleshoot issues, and drive continuous improvements in data infrastructure.

- Conduct code reviews, define coding standards, and promote engineering excellence across the team.

- Mentor and guide junior data engineers, fostering a culture of technical growth and innovation.

Requirements :

- 8+ years of experience in data engineering with proven leadership in managing data projects and teams.

- Expertise in Python, SQL, Spark (PySpark), and experience with AWS and Databricks in production environments.

- Strong understanding of modern data architecture, distributed systems, and cloud-native solutions.

- Excellent problem-solving, communication, and collaboration skills.

- Prior experience mentoring team members and contributing to strategic technical decisions is highly desirable.

- 8+ years of experience in data engineering with proven leadership in managing data projects and teams.

- Expertise in Python, SQL, Spark (PySpark), and experience with AWS and Databricks in production environments.

- Strong understanding of modern data architecture, distributed systems, and cloud-native solutions.

- Excellent problem-solving, communication, and collaboration skills.

- Prior experience mentoring team members and contributing to strategic technical decisions is highly desirable.