Senior Data Engineer

Key Responsibilities :

- Design, develop, and maintain enterprise-scale data engineering solutions using Azure Databricks and PySpark.

- Analyze, modernize, and migrate complex T-SQL transformation logic into optimized Databricks implementations.

- Design and develop scalable data ingestion pipelines using Azure Data Factory (ADF) to ingest data from multiple enterprise source systems into RAW and curated data layers.

- Build, maintain, and optimize enterprise ETL/ELT pipelines ensuring high availability, scalability, and performance.

- Monitor pipeline execution, analyze performance metrics, troubleshoot bottlenecks, and implement optimization strategies.

- Optimize Databricks workloads using partitioning, caching, Delta Lake optimization, file format tuning, and Spark performance best practices.

- Collaborate with Solution Architects, Business Analysts, Product Owners, and cross-functional engineering teams.

- Participate in technical solution design, architecture discussions, and engineering best practices.

- Lead technical initiatives and provide guidance to junior and mid-level Data Engineers.

- Conduct code reviews and ensure adherence to coding standards and data engineering best practices.

- Prepare technical documentation and support production deployments.

- Work closely with global teams on enterprise-scale cloud data transformation initiatives.

Required Skills :

- Minimum 6+ years of hands-on experience with Azure Databricks.

- Strong expertise in Core SQL, Complex SQL, and T-SQL (Mandatory).

- Strong hands-on experience with PySpark.

- Experience designing enterprise-scale data ingestion pipelines using Azure Data Factory (ADF).

- Strong understanding of ETL/ELT architecture and enterprise data engineering principles.

- Experience migrating complex SQL transformation logic into Databricks using PySpark.

- Expertise in Databricks performance optimization and Spark tuning.

- Experience working with enterprise cloud data platforms.

- Strong analytical, debugging, troubleshooting, and problem-solving skills.

- Excellent communication and stakeholder management skills.

- Ability to mentor engineering teams and provide technical leadership.

Good to Have :

- .NET knowledge

- Delta Lake

- Lakehouse Architecture

- Azure DevOps

- CI/CD Pipelines

- Agile/Scrum methodology

Educational Qualification :

- Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related discipline.