- Build scalable ETL/ELT processes to extract, transform, and load data into Azure Data Lake, Azure SQL Data Warehouse / Synapse, and other cloud storage systems.

- Implement reusable pipeline components and parameterization for optimized workflows.

2. Data Lake & Data Warehouse Engineering :

- Manage and optimize Azure Data Lake Storage (ADLS) for high-volume data ingestion and processing.

- Design and develop data models, schemas, and storage layers for enterprise Data Warehouse and analytical workloads.

- Implement best practices for partitioning, indexing, and performance optimization.

3. ETL Optimization & Performance :

- Fine-tune ETL processes to maximize efficiency, minimize latency, and enhance reliability.

- Ensure scalable architecture that supports both batch and real-time data processing. Use monitoring tools to analyze pipeline performance and troubleshoot bottlenecks.

4. Data Quality, Testing & Governance:

- Implement data validation, testing, and data quality checks throughout the pipeline lifecycle.

- Ensure accuracy, completeness, and consistency of data across systems. Support data governance initiatives, including metadata management using Azure Data Catalog.

5. Advanced Processing & Architecture :

- Work with Apache Spark, Databricks, or similar frameworks for large-scale data processing.

- Contribute to data architecture decisions, design reviews, and cloud migration strategies.

- Collaborate with architects to implement secure, governed data environments.

6. Documentation & Collaboration :

- Create and maintain comprehensive documentation including pipeline designs, data dictionaries, mapping documents, and process flows.

- Collaborate with cross-functional teams including BI, analytics, product owners, and business stakeholders.

- Provide support for data-related inquiries, enhancements, and issue resolution.

7. Monitoring & Troubleshooting :

- Monitor scheduled jobs, diagnose data pipeline failures, and perform timely resolutions.

- Implement alerting, logging, and automated recovery mechanisms to ensure system reliability.

- Perform root cause analysis and continuous improvement on recurring issues.

Skills & Experience :

Required Technical Skills :

- Strong hands-on experience with Azure Data Factory (ADF) Expertise in Azure Data Lake Storage (ADLS)

- Strong knowledge of Data Warehouse concepts and architecture Hands-on experience with Apache Spark, Databricks (preferred), and distributed data processing

- Proficiency in ETL/ELT development and orchestration Knowledge of Azure Data Catalog, metadata management, and data governance

- Strong SQL skills for data manipulation, analysis, and optimization

- Experience with version control (Git), CI/CD for data pipelines (preferred) Soft Skills Strong analytical and problem-solving mindset

- Excellent communication and documentation skills Ability to collaborate with stakeholders, business units, and vendors

- Detail-oriented with a commitment to quality and consistency Ability to work independently and in agile teams