About the Role:

We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, SQL, and Delta Lake to build scalable data platforms and modern Lakehouse solutions. The ideal candidate should have experience in Hadoop modernization, ETL/ELT development, data pipeline optimization, and cloud-based data engineering.

Key Responsibilities:

- Design and implement scalable Lakehouse architectures using Databricks and Delta Lake.

- Migrate Hadoop-based workloads to modern Databricks platforms.

- Develop and optimize ETL/ELT pipelines using PySpark and SQL.

- Build Medallion Architecture (Bronze, Silver, Gold) for enterprise analytics.

- Implement workflow orchestration using Airflow, Databricks Workflows, or NiFi.

- Develop real-time data pipelines using Kafka and streaming technologies.

- Ensure data governance, security, lineage, and access management using Unity Catalog.

- Collaborate with business stakeholders and global clients to deliver data solutions.

Required Skills:

- Strong experience in Databricks, PySpark, SQL, Delta Lake, and Apache Spark.

- Hands-on experience with Hadoop (HDFS, Hive, YARN) and large-scale data processing.

- Expertise in ETL/ELT development, data modeling, and distributed systems.

- Experience with Airflow, NiFi, Kafka, and streaming data pipelines.

- Knowledge of Lakehouse Architecture, Medallion Architecture, and Unity Catalog.

- Experience with AWS, Azure, or GCP cloud platforms.

- Strong understanding of data governance, security, and performance optimization.

Good to Have:

- Experience with Generative AI, NLP, LLMs, and Conversational AI platforms.

- Knowledge of Vector Databases, Semantic Search, and RAG architectures.

- Exposure to Banking or Financial Services domain.

- Understanding of GDPR, CCPA, and AI compliance standards.