Egen - Senior Data Engineer - Azure

Description :

Job Overview :

We are seeking a highly skilled Senior Data Engineer with deep expertise in Microsoft Azure data ecosystem to design, develop, and maintain scalable data pipelines and architectures. The ideal candidate will play a key role in building robust data solutions that support advanced analytics, BI, and AI workloads across the organization. This role involves working with cross-functional teams data scientists, analysts, and business stakeholders to ensure high-quality, secure, and performant data delivery.

Total experience : 8 to 10 years

Key Responsibilities :

1. Data Architecture & Pipeline Development :

- Design, develop, and maintain data ingestion, transformation, and storage pipelines using Azure Data Factory, Azure Databricks, and Synapse Analytics.

- Build end-to-end ETL/ELT workflows from diverse sources such as APIs, on-prem databases, SaaS applications, and data lakes.

- Implement data modeling (star schema, snowflake, data vault, etc.) for analytical and operational data stores.

- Manage data ingestion frameworks for both batch and streaming (real-time) use cases using Azure Event Hubs / Azure Stream Analytics / Kafka.

2. Data Management & Governance :

- Ensure data quality, consistency, and integrity across all environments.

- Implement and enforce data governance standards, including metadata management, lineage tracking, and data cataloging (e.g., Azure Purview).

- Manage security, compliance, and access controls using Azure Active Directory (AAD) and RBAC principles.

- Automate data validation, auditing, and monitoring workflows using modern DevOps practices.

3. Cloud & DevOps Integration :

- Manage and optimize Azure Data Lake Storage (ADLS Gen2) for cost and performance.

- Leverage Infrastructure as Code (IaC) tools such as Terraform, Bicep, or ARM templates for environment provisioning.

- Implement CI/CD pipelines for data workflows using Azure DevOps / GitHub Actions.

- Optimize compute and storage costs while maintaining scalability and resilience.

4. Collaboration & Leadership :

- Partner with data scientists, BI developers, and product teams to deliver reliable data solutions.

- Mentor junior data engineers on best practices, coding standards, and Azure services.

- Participate in architectural reviews and contribute to data platform design decisions.

- Communicate technical insights and trade-offs to non-technical stakeholders effectively.

Core Azure Services :

- Azure Data Factory (ADF) pipeline orchestration, triggers, linked services, integration runtime.

- Azure Databricks PySpark, Delta Lake, notebooks, job orchestration.

- Azure Synapse Analytics dedicated and serverless SQL pools, data modeling, query optimization.

- Azure Data Lake Storage (ADLS Gen2) hierarchical namespace, lifecycle management.

- Azure Functions / Logic Apps event-driven data workflows.

- Azure Event Hubs / Kafka / Stream Analytics real-time data ingestion.

Programming & Tools :

- Strong in Python or Scala for data transformations.

- Proficient in SQL (T-SQL, Spark SQL, or Synapse SQL).

- Experience with PySpark for distributed data processing.

- Knowledge of Git, CI/CD, and DevOps best practices.

- Familiarity with Power BI integration or other reporting tools is a plus.

Data Architecture & Modeling :

- Expertise in data warehouse design, data lakehouse architecture, and data pipelines.

- Understanding of data partitioning, indexing, and query optimization techniques.

- Experience with dimensional modeling, slowly changing dimensions, and fact tables.

Other Skills :

- Experience with API-based data ingestion and RESTful integrations.

- Exposure to machine learning data pipelines or feature stores is a plus.

- Familiarity with cost optimization and performance tuning in Azure.

Soft Skills :

- Excellent analytical and problem-solving abilities.

- Strong communication and documentation skills.

- Ability to work in an agile, fast-paced environment.

- Proven experience collaborating in cross-functional teams.

- Self-driven with a passion for data engineering and cloud technologies.

Salary : 10-24lpa