Job Summary :

We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, Apache Airflow, SQL, and Python to design and optimize data pipelines, ETL workflows, and cloud-based data solutions. The ideal candidate will have deep experience in big data processing, cloud-based analytics, and automation to drive efficient and scalable data engineering solutions.

Key Responsibilities :

- Design, develop, and optimize data pipelines using Databricks and Apache Airflow.

- Implement PySpark-based transformations and processing in Databricks for handling large-scale data.

- Develop and maintain SQL-based data pipelines, ensuring performance tuning and optimization.

- Create Python scripts for automation, data transformation, and API-based data ingestion.

- Work with Airflow DAGs to schedule and orchestrate data workflows efficiently.

- Optimize data lake and data warehouse performance for scalability and reliability.

- Integrate data pipelines with cloud platforms (AWS, Azure, or GCP) and various data storage solutions.

- Ensure adherence to data security, governance, and compliance standards.

Required Skills & Qualifications :

- 8-9 years of experience in Data Engineering or related fields.

- Strong expertise in Databricks (PySpark, Delta Lake, DBSQL).

- Proficiency in Apache Airflow for scheduling and orchestrating workflows.

- Advanced SQL skills for data extraction, transformation, and performance tuning.

- Strong programming skills in Python (pandas, NumPy, PySpark, APIs).

- Experience with big data technologies and distributed computing.

- Hands-on experience with cloud platforms (AWS / Azure / GCP).

- Expertise in data warehousing and data modeling concepts.

- Understanding of CI/CD pipelines, version control (Git)

- Experience with data governance, security, and compliance best practices.

- Excellent troubleshooting, debugging, and performance optimization skills.