Description :

Relevant Experience : 7-10 years.

Job description :

- Lead the data engineering team by driving the design, build, test, and launch of new data pipelines and models on the production data platform.

- Define and implement the processes needed to achieve operational excellence in all areas, including agile development, and data solutions.

- Oversee the design, development and maintenance of data infrastructure, including data warehouses, data lakes, data integration components supporting optimal extraction, transformation, and loading of data.

- Leading the communication with vendors, clients, the leadership team, and other stakeholders to facilitate effective project management and provide ongoing support.

- Collaborate with business owners for roadmap planning and prioritization, to deliver robust cloud-based data solutions for our customers.

- Work closely with cross-functional teams to propel and execute necessary solution enhancements and provide support for existing solutions.

- Define and enforce best practices for data engineering, data modeling, and data quality to ensure accuracy, reliability, reusability, security and consistency of data.

- Evaluate and implement new technologies, tools, and frameworks to improve the performance and efficiency of the data engineering team.

- Lead the planning, prioritization, and execution and tracking of data engineering projects, ensuring timely delivery and alignment with business objectives.

- Establish and monitor key performance indicators (KPIs) to measure the performance and effectiveness of the data engineering team.

- Stay current with industry trends and advancements in data engineering, Azure Cloud, and provide strategic guidance for adopting new technologies or methodologies.

Qualifications :

- 8-12 years of demonstrable experience in data engineering, analytics, data warehousing, data management, Data governance and Compliance Requirements.

- Experience managing data engineers and guiding a team of engineers through project planning, execution, tracking and monitoring, and quality control stages.

- Solid experience with cloud-based data tools and platforms.

- Proficient in implementing efficient cost management strategies, particularly about storage and computational expenses.

- Experience building processes supporting data transformation, data structures, metadata, security, governance, and workload management.

- Experience supporting and working with cross-functional teams in a dynamic environment.

- Expertise in designing and optimizing data models, RDBMS, NoSQL DBs, and data warehousing solutions.

- Excellent leadership, communication, and interpersonal skills, with the ability to effectively collaborate with cross-functional teams.

- Proven ability to drive innovation, make strategic decisions and lead complex data engineering initiatives to successful completion.

Skill Required :

Must Haves :

- SQL, Python, Py Spark, Spark SQL, Spark, Distributed Systems.

- Databricks, ADLS Gen 2 Blob Storage, Azure DevOps, Azure Data Factory.

- ETL, Building Data Pipelines, Data Warehousing, Datamart, Data Modelling and Governance.

- Agile Practices, SDLC, DevOps practices, and CI/CD pipelines for data engineering workflows.

- Solid understanding of the Microsoft Azure stack for large-scale data engineering developments and deployments is highly preferred.

- Hands-on experience with Azure Databricks, including data ingestion, data transformation, workflow management and optimization, monitoring and troubleshooting Spark Jobs.

- Ability to set up and manage Azure Data Lake Storage (ADLS) Gen 2 accounts, and familiarity with data lake architecture and best practices.

- Familiarity with big data frameworks (e.g., Apache Spark).

- Knowledge of Azure Key Vaults for securely storing and managing cryptographic keys, secrets, and certificates.

Good to Have :

- Event Hubs for log management.

- Workflow Orchestration.

- Cosmos DB.

- Power BI.

- Professional Services Background.

- Scala, Java.