Role Overview :

We are seeking a seasoned Azure Databricks Engineer to join our high-performing data engineering team in Bangalore/Chennai. In this role, you will architect and maintain robust data pipelines, transforming complex raw data into actionable insights that drive strategic decision-making. You will collaborate closely with cross-functional teams, including data scientists, business analysts, and product stakeholders, to ensure data integrity and performance at scale. By leveraging your deep technical expertise, you will play a pivotal role in optimizing our cloud data infrastructure, directly impacting the efficiency of our analytics platforms and the quality of data-driven outcomes for our global clients.

Key Responsibilities :

- Design and implement scalable end-to-end data pipelines using Azure Data Factory and Azure Databricks to support high-volume data processing requirements.

- Develop complex data transformation logic using PySpark and Apache Spark to ensure high-performance data ingestion and cleaning for downstream consumption.

- Optimize SQL queries and Spark jobs to improve system performance, reduce latency, and minimize cloud infrastructure costs for the organization.

- Partner with data architects and business stakeholders to translate complex business requirements into technical data solutions that meet evolving organizational needs.

- Maintain rigorous standards for data quality, security, and governance across all Azure-based data platforms to ensure compliance and reliability.

- Mentor junior team members and conduct code reviews to foster a culture of technical excellence and continuous improvement within the engineering team.

Required Skillset :

- Demonstrated expertise in building and managing large-scale data solutions within the Azure ecosystem, with a deep understanding of Azure Databricks and Azure Data Factory.

- Advanced proficiency in PySpark and Apache Spark for distributed computing, coupled with the ability to write highly optimized SQL for complex data manipulation.

- Strong analytical and problem-solving abilities, with a proven track record of troubleshooting performance bottlenecks in distributed data environments.

- Excellent communication skills, with the ability to articulate technical concepts clearly to non-technical stakeholders and collaborate effectively in a hybrid work environment.

- A proactive mindset toward learning new technologies and adapting to changing business requirements in a fast-paced, professional setting.

- A Bachelors or Masters degree in Computer Science, Information Technology, or a related quantitative field, supported by 6-12 years of relevant industry experience.