About the Role :

We are looking for a highly skilled Data Engineer to join our team. You will play a key role in building scalable data pipelines, enabling efficient data landing into Delta Lake, and driving insights through modern data platforms like Databricks, AWS, Snowflake, BI tools, and emerging GenAI technologies.

This role is ideal for someone who enjoys solving complex data challenges, optimizing performance, and working in a fast-paced, startup-style environment.

Key Responsibilities :

- Design and implement robust data pipelines for ingestion and transformation.

- Perform data landing into Delta Lake with proper partitioning, schema evolution, and optimization.

- Design and implement Medallion Architecture (Bronze, Silver, Gold layers).

- Develop and manage data workflows using Databricks (PySpark-based).

- Build and maintain data pipelines orchestrated via Apache Airflow.

- Work with AWS services (S3, Redshift, EMR, Glue, Lambda) for data ingestion, storage, and processing.

- Work with Snowflake for data warehousing, including :

1. Managing virtual warehouses and workload optimization

2. Analyzing query history and performance tuning

3. Leveraging information schema & account usage/stats schema

- Working with AWS CUR (Cost & Usage Report) logs for cost analysis and optimization

- Build scalable Lakehouse architectures for analytics and reporting.

- Optimize data processing jobs for performance, cost, and reliability.

- Collaborate with analytics teams and support BI tools (Looker, Tableau, Power BI).

- Develop and maintain LookML models for business reporting.

- Implement data governance, cataloging, and security best practices.

- Support and enable GenAI and ML use cases by building curated, high-quality datasets.

- Work with teams to integrate LLM-powered solutions into data workflows (where applicable).

- Monitor pipelines using observability tools and ensure high availability.

- Troubleshoot production issues and continuously improve system performance.

Required Skills & Experience :

- Bachelors degree in Computer Science or equivalent practical experience

- 7+ years of strong experience in Data Engineering.

- Advanced proficiency in Python and PySpark.

- Hands-on experience with Databricks platform.

- Strong experience with Apache Airflow (workflow scheduling & orchestration).

- Hands-on experience with AWS services :

1. S3, Redshift, EMR, Glue, Lambda

- Hands-on experience with Snowflake, including :

1. Warehouse management and performance tuning

2. Query optimization and cost monitoring

3. Working with information schema / account usage views

- Strong understanding of :

1. Delta Lake

2. Lakehouse Architecture

3. Medallion Architecture (Bronze, Silver, Gold layers)

4. Data modeling and ETL/ELT design

- Working knowledge of :

1. LLM orchestration frameworks

2. RAG (Retrieval-Augmented Generation)

3. MCP (Model Context Protocol)

4. AWS Bedrock or equivalent AI platforms

- Experience with AWS Nova Pro

- Experience in performance tuning of Spark and SQL workloads.

- Experience with BI tools (Looker preferred; Tableau/Power BI acceptable).

- Strong understanding of data governance, cataloging, and security practices.

- Solid understanding of distributed systems, networking, and cloud security.

- Strong problem-solving and debugging skills.

Good to Have :

- Experience working with AWS CUR logs / cost optimization frameworks.

- Experience supporting MLOps / ML pipelines.

- Exposure to GenAI technologies (LLMs, embeddings, vector databases, prompt engineering).

- Familiarity with monitoring & observability tools :

1. Prometheus, Grafana, Datadog, ELK stack

- Databricks, Snowflake, or AWS certifications.

- Experience working with large-scale data (TBs/PBs).

- Exposure to CI/CD pipelines for data workflows.

- Familiarity with streaming pipelines (Kafka/Event Hubs).

What Were Looking For :

- Strong hands-on engineer with end-to-end ownership mindset.

- Ability to work in a fast-paced, startup-style environment.

- Excellent collaboration and communication skills.

- Passion for building scalable data platforms and enabling AI/GenAI use cases.

Why Join Us :

- Exposure to modern data ecosystem (Databricks + Delta Lake + AWS + Snowflake + Airflow + BI + GenAI).

- Opportunity to work on next-gen data + AI platforms.

- Flexible and collaborative work culture.

- Opportunity to influence architecture and design decisions.