Databricks Engineer - Unified Data Analytics Platform

We are seeking a Databricks Unified Data Analytics Platform Engineer to design, build, optimize, and maintain end-to-end data analytics solutions on the Databricks Lakehouse Platform.

The role requires expertise in data engineering, analytics pipelines, cloud infrastructure, and big data technologies, enabling teams to perform advanced analytics, machine learning, and business intelligence.

The ideal candidate is passionate about data architecture, scalable analytics pipelines, and cloud integration, and has experience translating business requirements into technical solutions on Databricks.

Key Roles & Responsibilities :

- Databricks Platform Engineering Design, develop, and manage end-to-end data pipelines on Databricks using Apache Spark (PySpark, Spark SQL, or Scala).

- Build and maintain Lakehouse architectures integrating structured and unstructured data sources. Configure, optimize, and monitor Databricks clusters, jobs, and workflows for performance, reliability, and cost efficiency.

- Implement robust data ingestion, transformation, and orchestration pipelines. Data Engineering & Analytics

- Enablement Develop ETL/ELT pipelines to process high-volume and high-velocity data.

- Support BI tools, dashboards, and reporting by ensuring high-quality curated datasets (bronze, silver, gold layers).

- Collaborate with data scientists and analytics teams to enable ML/AI workloads on Databricks.

Cloud & Integration :

- Integrate Databricks with cloud storage solutions (AWS S3, Azure ADLS, GCP) and relational/non-relational databases.

- Implement secure access, data governance, and compliance using Databricks features like Unity Catalog. Ensure seamless integration with other cloud platforms, APIs, and third-party tools.

- Performance & Reliability Optimize Spark jobs and queries for performance, scalability, and cost-efficiency.

- Implement data quality checks, error handling, and monitoring mechanisms to ensure reliable pipeline execution.

- Troubleshoot and resolve data pipeline and system performance issues.

Collaboration & Stakeholder :

- Engagement Work closely with business analysts, product managers, and data teams to translate business requirements into technical solutions.

- Participate in architecture discussions and provide recommendations for platform and design improvements.

- Mentor junior engineers and foster a culture of best practices in data engineering and analytics.

Professional & Technical Skills :

- Must-Have Skills Strong hands-on experience with Databricks Unified Data Analytics Platform

- Expertise in Apache Spark (PySpark, Spark SQL, Scala)

- Experience building large-scale, production-grade data pipelines Strong SQL skills for analytics and transformations

Cloud & Platform Skills :

- Experience with AWS, Azure, or GCP Databricks Knowledge of Delta Lake, data versioning, and Lakehouse concepts

- Familiarity with cloud storage, data formats (Parquet, Delta), and security/access controls

Data & Analytics Skills :

- Knowledge of data modeling, warehousing, and analytics architecture Experience enabling BI dashboards, reporting, and advanced analytics use cases

- Understanding of data governance, lineage, and compliance requirements

Tools & Methodologies :

- Familiarity with Agile/Scrum development methodologies

- Experience with CI/CD, orchestration, and scheduling tools

- Knowledge of monitoring, alerting, and observability tools

Good-to-Have Skills :

- Experience with Unity Catalog and role-based access controls

- Knowledge of streaming data pipelines (Kafka, Kinesis, Event Hubs)

- Exposure to ML workflows and MLflow