Senior Data Engineer - ETL/Data Warehousing

Experience : 5-8 years

Location : Bengaluru

Role Overview :

The Senior Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support analytics, reporting, and machine learning initiatives. The role involves hands-on experience with Databricks and Azure data platforms to ensure high-quality, reliable, and efficient data delivery across the organization.

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain robust, scalable ETL/ELT pipelines using Databricks, PySpark, and cloud-native tools.

- Optimize data ingestion and transformation workflows for performance and cost efficiency.

Data Architecture & Modeling :

- Design and implement data lake/data warehouse architectures leveraging Databricks Lakehouse, Snowflake, Redshift, or BigQuery.

- Develop and maintain dimensional and semantic models to support advanced analytics and BI use cases.

Data Quality & Governance :

- Implement automated data validation, quality checks, and monitoring frameworks within Databricks workflows.

- Contribute to metadata management, lineage tracking, and governance processes.

Collaboration & Stakeholder Management :

- Partner with data scientists, analysts, and business stakeholders to understand requirements and deliver curated, production-grade datasets.

- Support deployment and operationalization of ML models within Databricks or integrated MLOps frameworks.

Automation & CI/CD :

- Automate workflows using orchestration tools such as Azure tools, or Databricks Workflows.

- Implement CI/CD and infrastructure-as-code practices for data pipelines and jobs.

Cloud & Infrastructure :

- Manage and optimize cloud data infrastructure (Azure).

- Leverage Databricks for scalable compute, collaborative development, and cost-efficient data processing.