Capco - Data Engineer - Python/Snowflake DB

Description : Data Engineer with Python and Snowflake (AI/Cortex Development Focus)

Job Summary :

- Lead the development and deployment of advanced data engineering and data science solutions, leveraging deep expertise in Python and the Snowflake Data Cloud platform within the AWS ecosystem.

- Drive innovation by building applications on the Snowflake AI/Cortex platform, focusing on generative AI capabilities, context enrichment, and integration with Large Language Models (LLMs).

- Responsible for end-to-end data lifecycle management, from complex ETL processes and data modeling to ensuring robust data governance, security, and deployment via CI/CD pipelines.

- Expected to work the 12 :30 PM - 9 :30 PM shift with a mandatory 3 days WFO (Tues, Wed, Thurs) in Hinjewadi, Pune.

Core Responsibilities :

Cloud Data Architecture and Snowflake Development :

- Utilize 9+ years of total experience, with 7+ years dedicated to developing data engineering and data science projects on Snowflake within the AWS cloud environment.

- Design, implement, and optimize data ingestion pipelines using Snowpipe and develop data processing workflows using Snowpark, specifically handling bulk and unstructured data from cloud sources like AWS S3 and data lakes.

- Implement effective data modeling techniques (e.g., Dimensional, Data Vault) optimized for Snowflake's architecture.

- Architect and manage Snowflake's governance framework, including account hierarchy models and a robust Account-role-permissions strategy.

- Implement advanced Data Governance/Security concepts using Snowflake Tags for Row/Column-level dynamic data masking.

- Establish and manage Data Sharing mechanisms, leveraging the Internal Data Marketplace and Data Exchanges for external and internal listings.

Python Engineering and Deployment :

- Leverage 7+ years of hands-on Python development experience, adhering to Object-Oriented Programming (OOP) principles for clean, maintainable, and high-quality engineering code.

- Implement sophisticated data structures and manipulation logic using Pandas and custom Python modules, demonstrating a deep understanding of server-side concurrency and multi-threading concepts.

- Utilize Object-Relational Mapping (ORM) frameworks like SQLAlchemy or equivalents for robust database interaction.

- Develop and deploy containerized Python web applications (using Docker or Kubernetes) on the AWS Cloud, utilizing CI/CD pipelines for automated deployment.

- Develop and deploy scalable serverless Python applications, such as AWS Lambda functions.

AI/Cortex and LLM Integration :

- Apply 1+ years of experience specifically building applications using the Snowflake AI/Cortex platform.

- Implement Cortex Agents and Cortex Search, focusing on integrating Cortex LLMs with external data sources.

- Deep understanding of context enrichment techniques using effective Prompts or Retrieval-Augmented-Generation (RAG) methods to enhance LLM query responses.

- Focus on building custom interaction applications that integrate with LLMs, including implementing Relevance Search functionality.

- Continuously improve application functionality through performance tuning, experimentation, and integrating customer feedback.

Required Technical Skills and Experience :

- Minimum 9+ years total experience, with a core focus (7+ years) on Data Engineering, Python, and cloud data platforms.

- Expert-level proficiency in Python development, including OOP, multi-threading, Pandas, and professional IDE usage (VS Code or Anaconda).

- Mandatory experience with Snowflake architecture, data modeling, performance tuning, and governance features (e.g., dynamic masking, role management).

- Hands-on experience with AWS services for data storage, compute, and deployment (S3, Lambda, EC2, Kubernetes).

- Proven experience with version control systems, specifically Git or Bitbucket, and using Python unit testing frameworks.

- Strong experience utilizing Agile and SCRUM methodologies, preferably managed through JIRA.

- Excellent analytical skills, advanced problem-solving capabilities, and the ability to communicate complex technical concepts clearly.

Preferred Skills and Experience :

- Direct development experience with Snowpark for advanced data transformations.

- Experience building Snowflake native applications using Streamlit and deploying them onto AWS instances (EC2 or Docker containers).

- Experience implementing application-level Cache strategies within Python web applications for performance optimization.

- Familiarity with duckdb in conjunction with Apache Arrow for in-memory analytical processing.

- Experience implementing CI/CD pipelines specifically for deploying and versioning Snowflake applications (DDL/DML).