Data Engineer will be responsible for development and deployment of end-to-end data ingestion pipelines using Azure Databricks, Apache Spark, and related technologies, ensuring scalability, performance, and efficiency.

Key responsibilities and tasks :

- Design & Architecture : Design high-performance, resilient, and scalable data architectures for data ingestion and processing using best practices for Azure Databricks and Spark.

- Team Leadership : Provide technical guidance and mentorship to a team of data engineers, fostering a culture of collaboration, continuous learning, and innovation.

- Collaboration : Work closely with data scientists, business analysts, and other stakeholders to understand data requirements and ensure smooth integration of various data sources into the data lake/warehouse.

- Optimization & Performance Tuning : Ensure data pipelines are optimized for speed, reliability, and cost efficiency in an Azure environment. Conduct performance tuning, troubleshooting, and debugging of Spark jobs and Databricks clusters.

- Code Quality & Best Practices : Enforce and advocate for best practices in coding standards, version control, testing, and documentation.

- Integration with Azure Services : Work with other Azure services such as Azure Data Lake Storage, Azure SQL Data Warehouse, Azure Synapse Analytics, and Azure Blob Storage to integrate data seamlessly.

- Continuous Improvement : Stay current with industry trends and emerging technologies in the field of data engineering and make recommendations for improvements to the teams tools and processes.

- Ensure Data Quality : Implement data validation and data quality checks as part of the data ingestion process to ensure consistency, accuracy, and integrity of ingested data.

- Risk & Issue Management : Proactively identify risks and blockers and resolve complex technical issues in a timely and effective manner.

Required skills and qualifications :

- Bachelors or masters degree in computer science, Information Technology, Engineering, or a related field