We are seeking a Databricks Engineer responsible for designing and implementing scalable data ingestion and transformation pipelines within a cloud based data platform.

The role focuses on building high performance data pipelines, integrating multiple source systems, and ensuring reliable data availability for analytics and downstream applications.

Key Responsibilities :

- Design and develop scalable data ingestion pipelines using Databricks and Apache Spark.

- Build and maintain ETL/ELT workflows to ingest structured and semi structured data from various source systems.

- Implement data transformation logic using PySpark, SQL, and Delta Lake.

- Integrate data from APIs, databases, file systems, and streaming platforms.

- Optimize data processing performance and manage large-scale data workloads.

- Collaborate with Data Architects, Business Analysts, and QA teams to ensure accurate data delivery.

- Implement monitoring, logging, and error handling mechanisms for ingestion pipelines.

- Support CI/CD deployment processes for data engineering solutions.

- Ensure adherence to data governance, security, and data quality standards.

Required Qualifications

- 4+ years of experience in Data Engineering or Big Data development.

- Strong experience with Databricks and Apache Spark.

- Proficiency in Python (PySpark) and SQL.

- Experience building data pipelines in cloud environments such as AWS, Azure, or GCP.

- Knowledge of data ingestion frameworks and ETL tools.

- Experience working with structured and semi-structured data formats (JSON, Parquet, CSV, Avro).

- Familiarity with version control tools such as Git.

Preferred Qualifications

- Experience with Delta Lake architecture.

- Knowledge of workflow orchestration tools such as Airflow or Azure Data Factory.

- Experience with streaming platforms such as Kafka or Event Hub.

- Exposure to CI/CD pipelines and DevOps practices for data platforms.