Role Overview :

As a Senior Databricks Architect, you will be at the forefront of designing and implementing cutting-edge data solutions for our clients, leveraging the power of Azure Databricks and related technologies. You will collaborate closely with data engineers, data scientists, and business stakeholders to understand their needs and translate them into scalable, robust, and efficient data architectures. Your expertise will be instrumental in enabling data-driven decision-making and driving significant business outcomes for our clients.

Key Responsibilities :

- Design and implement scalable and robust data architectures on Azure Databricks to meet the evolving needs of our clients.

- Develop and maintain ETL pipelines using Spark, PySpark, and Informatica/Informatica IDMC to ingest, transform, and load data from various sources into the data lake.

- Optimize Spark jobs and Databricks clusters for performance and cost-efficiency to ensure optimal resource utilization.

- Collaborate with data scientists and business analysts to understand their data requirements and provide them with the necessary data infrastructure and tools.

- Provide technical leadership and mentorship to junior team members, fostering a culture of innovation and continuous learning.

- Ensure data quality and security by implementing appropriate data governance policies and procedures.

- Evaluate and recommend new technologies and tools to improve our data processing capabilities and stay ahead of the curve.

- Troubleshoot and resolve complex data-related issues to minimize downtime and ensure data availability.

Required Skillset :

- Demonstrated ability to design and implement large-scale data solutions using Azure Databricks, Apache Spark, and related technologies.

- Proven expertise in developing ETL pipelines using Spark, PySpark, Informatica, and Informatica IDMC.

- Strong understanding of data warehousing concepts, data modeling techniques, and data governance principles.

- Excellent programming skills in Python and experience with Spark Streaming.

- Ability to communicate complex technical concepts effectively to both technical and non-technical audiences.

- Bachelor's or Master's degree in Computer Science or a related field.

- Adaptable to work in a remote/hybrid environment and collaborate effectively with distributed teams.

- 8-12 years of relevant experience.