Data Engineer - Azure Databricks

Job Title : Data Engineer (AWS + Databricks)

Experience : 58 Years

Location : Bangalore (Candidates must be local)

Notice Period : Immediate to 30 days

Job Overview :

We are looking for an experienced Data Engineer with strong expertise in Databricks and AWS to design, develop, and optimize large-scale data pipelines. The ideal candidate should have hands-on experience in building efficient ETL/ELT workflows, implementing governance frameworks, and integrating CI/CD within modern data platforms.

Key Responsibilities :

- Design and build scalable data pipelines using Databricks (PySpark, Auto Loader, Delta Live Tables, Structured Streaming).

- Implement Medallion Architecture and optimize data workflows for performance and reliability.

- Develop and maintain ETL/ELT processes using PySpark and Databricks SQL Endpoints.

- Work extensively with AWS services such as Glue, Lambda, IAM, CloudWatch, and CloudTrail.

- Implement Unity Catalog, data governance policies, and security best practices.

- Integrate AI/ML models and workflows into the data ecosystem.

- Collaborate with cross-functional teams to understand data requirements and deliver high-quality solutions.

- Set up and manage CI/CD pipelines for Databricks deployment.

Mandatory Skills :

Databricks (3+ years) :

- PySpark-based ETL/ELT development

- Auto Loader, Structured Streaming

- Delta Live Tables & Delta Lake

- Medallion Architecture

- Databricks SQL Endpoints

- CI/CD integrations (Azure DevOps/GitHub/GitLab)

AWS (5+ years) :

- AWS Glue

- AWS Lambda

- CloudWatch & CloudTrail

- IAM & security best practices

Governance & Advanced :

- Unity Catalog

- Policy enforcement using Immuta or Privacera

- AI/ML pipeline integration

Preferred Qualifications :

- Experience with orchestration tools (Airflow, Step Functions).

- Strong knowledge of data modeling and performance tuning.

- Exposure to data quality frameworks (DQ expectations).