Data Engineer - ETL Tools

Description :

Key Responsibilities :

- Design, develop, and maintain end-to-end data pipelines using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics

- Build and manage data solutions using Azure Data Lake and Azure SQL Database

- Implement batch and incremental data ingestion processes from multiple data sources

- Develop scalable data processing solutions using PySpark and Spark SQL

- Develop and optimize ETL workflows using Azure Data Factory

- Work with SSIS packages (fair knowledge required) for data integration and migration

- Ensure data quality, validation, and reconciliation across data pipelines

- Design and implement ERD and Dimensional Data Models (Star/Snowflake schemas)

- Optimize data structures for reporting, analytics, and performance

- Support analytics workloads in Azure Synapse Analytics

- Perform query optimization and performance tuning using T-SQL, Spark SQL

- Identify and resolve data pipeline failures and performance bottlenecks

- Monitor and improve system reliability and scalability

- Use Git and Azure DevOps for version control, CI/CD, and release management

- Collaborate with data analysts, BI developers, data scientists, and business stakeholders

- Participate in design reviews, code reviews, and Agile ceremonies

Required Skills & Qualifications :

Technical Skills :

- Strong hands-on experience with Azure SQL Database, Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake

- Proficiency in T-SQL, PySpark, Spark SQL

- Fair knowledge of KQL (Kusto Query Language)

- Strong understanding of data modeling concepts (ERD, Dimensional Modeling)

- Experience in performance tuning and troubleshooting large-scale data systems

- Fair knowledge of Hadoop and Apache Spark

- Experience with Git and Azure DevOps

Professional Skills :

- Strong analytical and problem-solving abilities

- Excellent communication and collaboration skills

- Ability to work independently and mentor junior engineers

- Experience working in Agile/Scrum environments