Data Engineer

Description :

Role Overview :

We are looking for an experienced Data Engineer with strong expertise in Databricks, AWS, and modern data engineering practices. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and cloud-native data platforms. This role requires hands-on experience in Databricks, PySpark, Delta Lake, Airflow, and AWS services, along with strong SQL and data modeling skills.

Experience with governance, observability, and telemetry pipelines is essential. Exposure to AI/LLM-based automation and metadata enrichment concepts will be considered an added advantage.

Key Responsibilities :

Technical Design & Development :

- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Airflow.

- Build reusable frameworks and templates for telemetry and enterprise data pipelines.

- Develop and optimize data workflows using PySpark, Spark SQL, and Delta Lake.

- Work with Databricks Delta Live Tables and Unity Catalog for data governance and orchestration.

- Ensure cloud-native architecture aligned with AWS best practices.

- Integrate and process data from multiple structured and unstructured sources.

Governance, Security & Optimization :

- Ensure data quality, integrity, governance, and compliance across data platforms.

- Optimize SQL queries, Spark jobs, and data models for scalability and performance.

- Implement monitoring, logging, and observability mechanisms for data workflows.

- Support security and access management strategies within Databricks and AWS environments.

Cross-Functional Collaboration :

- Collaborate with architects, technical leads, and business stakeholders to understand data requirements and deliver scalable solutions.

- Work closely with analytics and BI teams to enable reporting and dashboarding solutions.

- Participate in design discussions, code reviews, and technical documentation.

AI/LLM Enablement (Good to Have) :

Familiarity with Agentic AI concepts for :

- Workflow automation

- Data quality validation

- Metadata enrichment

- Intelligent operational monitoring

Required Skills & Experience :

Experience :

- 8+ years of overall experience in Data Engineering.

- Minimum 3+ years of hands-on experience in Databricks.

Technical Skills :

Strong expertise in :

- Databricks

- PySpark

- Spark SQL

- Delta Lake

- Unity Catalog

- Delta Live Tables (DLT)

Strong SQL skills for :

- Data modeling

- Query optimization

- Performance tuning

Hands-on experience with AWS services :

- S3

- Glue

- Kinesis

- Redshift

- API Gateway

- SNS

- Lambda

- Experience with Airflow for workflow orchestration.

Good to Have Skills :

- DBT

- Splunk

- Atlan

- Java

- API integrations

- Power BI