Position Title : Databricks Engineer

Location : Noida/Hyderabad

P&L Responsibility : No

Key Stakeholders : Data Scientists; Business Analysts; Machine Learning Engineers; Project Sponsors

Reports To : Director of Data Engineering

Direct Reports : None (Individual Contributor)

Nature of Work : Hybrid design, develop and optimise scalable big data solutions on the Databricks Unified Analytics Platform, enabling advanced AI/ML workflows

About the Client :

This is a high-growth data & AI solutions firm partnering with mid-market and large enterprises in finance, healthcare and retail. With a lean, agile team of data engineers and machine learning specialists, they deliver end-to-end analytics - from data ingestion and lakehouse architecture to production-grade AI deployments. A strong focus on continuous upskilling, cross-functional innovation workshops and mentorship drives a culture of growth and thought leadership.

Why Join This Organisation :

- Lead the design of enterprise-scale data lakehouse and AI/ML solutions for marquee clients

- Thrive in a lean, agile environment that prioritises innovation, learning and collaboration

- Shape the next generation of predictive analytics and streaming data pipelines

- Accelerate your career through mentorship programs and exposure to cutting-edge tooling

Position Overview :

As a Databricks Engineer, youll be the technical cornerstone for building high-performance, cost-efficient data architectures on Databricks. Collaborating with data scientists, ML engineers and business stakeholders, youll architect robust ETL workflows and real-time analytics solutions that drive strategic decision-making.

You will tackle challenges across distributed data processing, performance tuning and governance - promoting best practices in Apache Spark, security and compliance. Your contributions will ensure data integrity, reliability and scalability across large-scale enterprise environments.

Key Responsibilities :

- Design, implement and optimize batch and streaming pipelines in Databricks

- Develop scalable ETL workflows using Apache Spark for large datasets

- Architect and tune Delta Lake-based data lakehouse solutions for performance and cost efficiency

- Enforce data governance, security policies and compliance standards

- Integrate advanced AI/ML models into production pipelines alongside data scientists

- Automate deployments via CI/CD tools and infrastructure-as-code (e.g., Terraform)

- Monitor and troubleshoot cluster performance, job failures and data quality issues

Education & Experience Requirements :

- Bachelors or Masters in Computer Science, Data Engineering or related field

- 5+ years hands-on experience with Databricks and Apache Spark in enterprise settings

- Proven expertise in SQL and Python or Scala for data processing

- Experience with cloud platforms (AWS, Azure or GCP) and Delta Lake architecture

- Familiarity with CI/CD frameworks, DevOps best practices and infrastructure-as-code

- Solid understanding of data security, compliance and governance

Essential Skills & Competencies :

- Technical/Functional : Advanced knowledge of Databricks, Apache Spark, Delta Lake, cloud data services

- Leadership : Cross-functional collaboration, influencing architectural decisions, mentoring peers

- Behavioural : Proactive problem-solver, detail-oriented, adaptable in fast-paced agile teams, strong communicator