Role Overview :

We are seeking an experienced Lead Data Engineer to design, build, and optimize scalable data platforms within the Azure Databricks ecosystem. The ideal candidate will have strong expertise in PySpark/Scala, Delta Lake, and data pipeline architecture, along with experience leading complex data initiatives and mentoring team members.

Key Responsibilities :

- Lead and contribute to the development of a self-service data platform for reporting and analytics.

- Design and build data solutions using Databricks, SQL, Python, Spark, and Delta Lake within the Azure ecosystem (Blob Storage, Data Factory, Event Hubs).

- Implement best practices for ETL/ELT processes, focusing on data quality management, partitioning, maintainability, and reusability.

- Collaborate with engineers, product managers, and business leaders to ensure seamless integration of the data platform with systems such as Tableau, Power BI, APIs, and other applications.

- Establish CI/CD pipelines and use infrastructure-as-code tools for deployment and monitoring (Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, Azure Monitor/Datadog).

- Ensure adherence to organizational compliance standards and Code of Conduct as defined in LogixGarden.

Required Skills & Experience :

- 8+ years of data engineering experience.

- 3+ years in a Senior or Principal Engineer role, with proven experience leading projects.

- Strong hands-on experience with Databricks and related integrations (Delta Live Tables, Auto Loader, Databricks Workflows / Apache Airflow, Unity Catalog).

- Expertise in PySpark and/or Scala.

- Strong DataFrame programming experience (Spark, Pandas, NumPy).

- Proven ability to design scalable, secure, and maintainable data pipelines.

- Excellent communication and leadership skills, with a passion for mentoring.

- Solid understanding of data security and governance frameworks, with exposure to HIPAA or SOC 2 compliance preferred.

Preferred Skills :

- Azure cloud experience.

- Azure Databricks implementation expertise.

- Experience designing and implementing data security and governance frameworks.