Job Overview :

We are seeking an experienced Senior Data Engineer to join our dynamic team and drive our data engineering initiatives. The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimizing data workflows, and enabling data-driven decision making across the organization.

Key Responsibilities :

- Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services

- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark

- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation

- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes

- Create and maintain data models, schemas, and documentation to ensure data quality and consistency

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions

- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed

- Ensure data security, privacy, and compliance with industry standards and regulations

- Mentor junior team members and provide technical guidance on data engineering best practices

- Stay current with emerging technologies and industry trends in data engineering and cloud computing

- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables

- Design and implement data governance frameworks and data quality monitoring solutions

Required Qualifications :

- 8+ years of total professional experience in software development, data engineering, or related fields

- 7+ years of hands-on experience in data engineering roles with proven expertise in designing and implementing data solutions

- Extensive experience with Azure Data Factory (ADF) for building and orchestrating data pipelines

- Strong proficiency in Azure Databricks and Apache Spark for large-scale data processing

- Hands-on experience with Lake flow Declarative Pipelines and modern data pipeline architectures

- Advanced SQL skills with experience in query optimization and database design

- Proficient in PySpark and Python for data processing and transformation tasks

- Experience with Azure cloud services including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics

- Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT processes

- Knowledge of data governance, data quality, and data security best practices

- Experience with version control systems (Git) and CI/CD pipelines

- Excellent problem-solving skills and attention to detail

- Strong communication skills and ability to work effectively in cross-functional teams

Preferred Qualifications :

- Bachelor's degree in Computer Science, Engineering, Mathematics, or related technical field

- Azure certifications such as Azure Data Engineer Associate (DP-203) or Azure Solutions Architect Expert

- Experience with additional Azure services like Azure Stream Analytics, Azure Event Hubs, or Azure Cosmos DB

- Knowledge of machine learning pipelines and MLOps practices

- Experience with Infrastructure as Code (IaC) tools like ARM templates or Terraform

- Familiarity with containerization technologies (Docker, Kubernetes)

- Experience with real-time data processing and streaming technologies

- Knowledge of other programming languages such as Scala or Java

- Experience with data visualization tools and business intelligence platforms

- Previous experience in mentoring and leading technical teams

Technical Skills :

- Cloud Platforms : Microsoft Azure (primary focus)

- Data Processing : Azure Databricks, Apache Spark, PySpark

- Data Orchestration : Azure Data Factory (ADF), Lake flow Declarative Pipelines

- Databases : SQL Server, Azure SQL Database, Azure Synapse Analytics, Azure Data Lake

- Programming Languages : Python, SQL, PySpark

- Development Tools : Git, Azure DevOps, Visual Studio Code

- Data Formats : JSON, Parquet, Avro, CSV, XML

- Monitoring & Logging : Azure Monitor, Application Insights