Senior Data Architect - Databricks Platform

Job Description :

Key Responsibilities :

- Design and implement robust, scalable, and optimized data architectures leveraging the Databricks platform and Snowflake ecosystem.

- Architect data pipelines and frameworks that are efficient, fault-tolerant, and capable of handling large-scale, complex data processing.

- Define data modeling standards, data lakehouse architecture, and best practices for structured and unstructured data management.

- Collaborate with enterprise architects to align data platform strategy with overall organizational goals.

- Evaluate new technologies, frameworks, and tools to enhance scalability, reliability, and performance.

- Develop and manage end-to-end ETL/ELT pipelines using Databricks notebooks, Delta Lake, Snowflake, and Azure Data Factory (ADF).

- Ensure seamless integration across diverse data sources on-premises and cloud-based.

- Implement efficient data transformation, ingestion, and cleansing frameworks for analytics and AI workloads.

- Standardize data pipeline templates for reusability and consistency across teams.

- Work extensively with Azure, AWS, or GCP platforms to integrate Databricks and Snowflake with storage layers such as ADLS, Amazon S3, and Google Cloud Storage.

- Architect hybrid data solutions enabling smooth data exchange across cloud, on-premise, and third-party applications.

- Collaborate with cloud engineering teams to define networking, storage, and compute strategies supporting large-scale data workloads.

- Optimize performance of data workflows and clusters by fine-tuning Databricks configurations, SQL queries, caching, and partitioning strategies.

- Identify and resolve performance bottlenecks in data ingestion and transformation processes.

- Implement monitoring tools and alerts to proactively detect and address performance issues.

- Define and enforce data governance frameworks, ensuring compliance with organizational and regulatory standards.

- Implement role-based access control (RBAC), encryption, and data masking techniques for sensitive datasets.

- Establish data lineage, metadata management, and auditing mechanisms for transparency and traceability.

- Ensure compliance with GDPR, HIPAA, and other data protection standards as applicable.

- Automate data pipeline deployments using CI/CD tools such as Azure DevOps, Jenkins, or GitHub Actions.

- Create monitoring dashboards to track data pipeline performance, latency, and data quality metrics.

- Collaborate closely with data scientists, analysts, and business stakeholders to translate analytical requirements into technical deliverables.

- Work with product and engineering teams to ensure seamless integration of data solutions with enterprise systems.

- Lead and mentor data engineering teams, conduct technical reviews, and ensure adherence to architectural best practices.

- Stay current with the latest Databricks, Snowflake, and Azure Data Services advancements.

- Evaluate emerging trends in data lakehouse, AI/ML integration, and automation tools.

- Drive initiatives to modernize legacy systems and adopt next-generation data architectures (e.g., medallion architecture, data mesh).