Description :

The Databricks Engineer will collaborate closely with the architecture leader to build, scale, and manage the Human Capital Data Lake environment on Databricks. This role is pivotal in designing secure, high-performance data pipelines and implementing advanced data engineering practices to support enterprise analytics and reporting initiatives.

Key Responsibilities (KRA) :

- Administer and maintain Databricks workspaces, clusters, environments, and configurations

- Design, develop, and optimize scalable data pipelines and ingestion frameworks for new data sources

- Implement and manage Attribute-Based Access Control (ABAC) models for secure data access

- Build reusable, maintainable data transformation logic using PySpark and Spark-based frameworks

- Ensure performance tuning, reliability, and fault tolerance of data engineering workflows

- Collaborate with architecture, analytics, and governance teams to align with enterprise data standards

- Implement Delta Lake best practices for incremental loads, schema evolution, and ACID compliance

- Drive automation for deployment, monitoring, and operational efficiencies

- Troubleshoot cluster, pipeline, and data quality issues to ensure uninterrupted data availability

- Research, evaluate, and integrate innovative approaches to improve scalability and efficiency

Key Skillsets Required :

- Strong hands-on expertise in Databricks platform administration and cluster management

- Proficiency in PySpark, Spark SQL, and distributed data processing

- Experience designing ETL/ELT data pipelines with Delta Lake architectures

- Strong understanding of ABAC, data security, and data governance practices

- Expertise in SQL, data modeling, and large-scale data transformations

- Experience with Cloud platforms (Azure / AWS / GCP) for data engineering workloads

- Knowledge of data orchestration and automation frameworks

- Strong problem-solving skills for debugging performance, cluster, or pipeline issues

- Familiarity with CI/CD for data engineering deployments