Lead Data Engineer - Azure Databricks

Description :

The Lead Data Engineer will take ownership of engineering, enhancement, and continuous evolution initiatives for the modernized Data Analytics and Reporting framework. The role involves driving end-to-end delivery for data engineering projects, coordinating cross-functional teams, ensuring robust data platform performance, and enabling scalable, reliable analytics capabilities.

The position demands strong technical expertise, solutioning capabilities, leadership, and the ability to collaborate effectively with business users, onshore stakeholders, and global teams.

Primary Skills : Azure, Databricks, PySpark, Azure DevOps

Role Responsibility :

- Oversee the day-to-day operations of the data engineering support team.

- Ensure consistent monitoring, stability, and performance of data pipelines and environments.

- Serve as the first line of escalation for production incidents and critical issues.

- Lead the triage, resolution, and root cause analysis of incidents.

- Coordinate with system, infrastructure, and application teams to resolve high-impact issues.

- Implement proactive monitoring and preventive measures.

- Ensure environment consistency and version control across all stages.

- Support release cycles, deployments, and configuration management.

- Validate platform readiness post-release and coordinate UAT/production validations.

- Maintain service-level commitments and ensure adherence to defined operational KPIs.

- Document operational processes, known issues, and best practices.

- Continuously identify areas for automation and process optimization.

- Act as the onshore liaison between business users, data engineers, and infrastructure teams.

- Communicate operational updates, risk assessments, and improvement plans clearly to stakeholders.

- Participate in governance reviews, service audits, and performance reporting.

- Mentor and guide junior support engineers.

- Conduct regular knowledge transfer sessions and maintain updated operational documentation.

Role Requirement :

- Strong understanding of data processing principles, ETL/ELT workflows, and modern data architecture concepts.

- Proficiency in incident triage, root cause analysis, and service restoration.

- Good awareness of data quality, validation, and reconciliation methodologies.

- Experience handling release planning, migration, and deployment coordination.

- Understanding of data governance, metadata, and operational compliance aspects.

- Strong ownership mindset with the ability to independently manage onshore operational responsibilities.

- Excellent coordination and stakeholder engagement skills.

- Ability to prioritize critical incidents and maintain composure under pressure.

- Strategic thinking identifying improvement areas and driving process excellence initiatives.

- Collaborative leadership style fostering teamwork across global and cross-functional teams.

Behavioral Attributes :

- Proactive, dependable, and detail-oriented.

- Excellent communication, documentation, and presentation skills.

- Demonstrates accountability, initiative, and customer-centric thinking.

- Open to learning and adapting to evolving data and support technologies.

Additional Requirement :

- Overall 7+ years of experience with a minimum of 3- 4 years in Databricks and 2- 3 years of experience leading a data engineering team.

- Hands-on experience with Databricks, Azure DevOps, and data pipeline engineering & monitoring.

- Good understanding of the Azure Data Engineering ecosystem (ADF, Synapse, Storage).

- Exposure to Python or PySpark for the development and debugging of data jobs.

- Experience with orchestration tools (e.g., Airflow, ADF pipelines).

- Strong SQL, PL/SQL, and advanced data troubleshooting skills.

- Familiarity with Data Quality frameworks and validation processes.

- Experience managing end-to-end engineering lifecyclefrom requirement analysis and design to deployment and user support.

- Experience in incident, change, and release management processes.

- Knowledge of modern data platform practices and observability tools.

- Excellent analytical, communication, and problem-solving abilities.

- Flexibility to work in overlapping shifts to support On-site (internal, client, and business) teams.