Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

- Build and optimize ETL/ELT workflows for processing structured and unstructured data.

- Develop data transformation and processing logic using Python, PySpark, and SQL.

- Work with Delta Lake to manage reliable, scalable, and high-performance data storage.

- Implement data ingestion solutions from multiple sources, including databases, APIs, files, and cloud storage.

- Optimize Spark jobs, queries, and data pipelines for performance, scalability, and cost efficiency.

- Implement data quality checks, validation rules, and monitoring mechanisms.

- Collaborate with Data Architects, Data Scientists, Business Analysts, and other engineering teams to understand data requirements.

- Work with cloud-based data platforms and services to build integrated data solutions.

- Follow best practices for data security, governance, version control, and deployment.

- Troubleshoot data pipeline failures and ensure data availability, accuracy, and reliability.

Required Skills & Qualifications :

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

- Hands-on experience with Azure Databricks or Databricks.

- Strong programming skills in Python and PySpark.

- Strong proficiency in SQL and experience working with relational databases.

- Good understanding of Apache Spark architecture and distributed data processing.

- Experience developing and managing ETL/ELT data pipelines.

- Hands-on experience with Delta Lake and Delta tables.

- Understanding of data warehousing concepts, data modelling, and data integration.

- Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP.

- Familiarity with Git and CI/CD practices for data engineering workflows.

- Strong analytical, problem-solving, and debugging skills.