Job description :

- Lead the data engineering team by driving the design, build, test, and launch of new data pipelines and models on the production data platform

- Define and implement the processes needed to achieve operational excellence in all areas, including agile development, and data solutions

- Oversee the design, development and maintenance of data infrastructure, including data warehouses, data lakes, data integration components supporting optimal extraction, transformation, and loading of data

- Leading the communication with vendors, clients, the leadership team, and other stakeholders to facilitate effective project management and provide ongoing support

- Collaborate with business owners for roadmap planning and prioritization, to deliver robust cloud-based data solutions for our customers

- Work closely with cross-functional teams to propel and execute necessary solution enhancements and provide support for existing solutions

- Define and enforce best practices for data engineering, data modeling, and data quality to ensure accuracy, reliability, reusability, security and consistency of data

- Evaluate and implement new technologies, tools, and frameworks to improve the performance and efficiency of the data engineering team

- Lead the planning, prioritization, and execution and tracking of data engineering projects, ensuring timely delivery and alignment with business objectives

- Establish and monitor key performance indicators (KPIs) to measure the performance and effectiveness of the data engineering team

- Stay current with industry trends and advancements in data engineering, Azure Cloud, and provide strategic guidance for adopting new technologies or methodologies

Qualifications :

- 8-10 years of demonstrable experience in data engineering, analytics, data warehousing, data management, Data governance and Compliance Requirements.

- Experience managing data engineers and guiding a team of engineers through project planning, execution, tracking and monitoring, and quality control stages

- Solid experience with cloud-based data tools and platforms

- Proficient in implementing efficient cost management strategies, particularly about storage and computational expenses.

- Experience building processes supporting data transformation, data structures, metadata, security, governance, and workload management.

- Experience supporting and working with cross-functional teams in a dynamic environment

- Expertise in designing and optimizing data models, RDBMS, NoSQL DBs, and data warehousing solutions

- Excellent leadership, communication, and interpersonal skills, with the ability to effectively collaborate with cross-functional teams

- Proven ability to drive innovation, make strategic decisions and lead complex data engineering initiatives to successful completion

Skill Required :

Must Haves :

- SQL, Python, Py Spark, Spark SQL, Spark, Distributed Systems

- Databricks, ADLS Gen 2 Blob Storage, Azure DevOps, Azure Data Factory

- ETL, Building Data Pipelines, Data Warehousing, Datamart, Data Modelling and Governance

- Agile Practices, SDLC, DevOps practices, and CI/CD pipelines for data engineering workflows

- Solid understanding of the Microsoft Azure stack for large-scale data engineering developments and deployments is highly preferred

- Hands-on experience with Azure Databricks, including data ingestion, data transformation, workflow management and optimization, monitoring and troubleshooting Spark Jobs

- Ability to set up and manage Azure Data Lake Storage (ADLS) Gen 2 accounts, and familiarity with data lake architecture and best practices

- Familiarity with big data frameworks (e.g., Apache Spark).

- Knowledge of Azure Key Vaults for securely storing and managing cryptographic keys, secrets, and certificates

Good to Have :

- Event Hubs for log management

- Workflow Orchestration

- Cosmos DB

- Power BI

- Professional Services Background

- Scala, Java