The role will be responsible for defining technology and data architecture for large-scale, cloud-native data platforms and leading the design of highly scalable, reliable, secure, and high-performance data solutions.

The Principal Architect will work closely with engineering leadership, technology teams, business stakeholders, and clients to define architecture, establish technical standards, drive modernization initiatives, and ensure successful delivery of complex data engineering programs.

The ideal candidate should have extensive hands-on experience with distributed data processing, AWS data services, Spark/Scala, Hadoop ecosystems, data pipelines, ETL/ELT architectures, and cloud-native solutions, along with proven experience providing technical leadership across large enterprise programs.

Key Responsibilities :

Architecture & Technical Leadership :

- Define and own the overall architecture for enterprise-scale data engineering and analytics platforms.

- Design scalable, highly available, secure, and resilient cloud-based data architectures.

- Lead architecture decisions covering data ingestion, processing, transformation, storage, integration, and consumption layers.

- Define technical standards, architecture principles, design patterns, and best practices for data engineering solutions.

- Evaluate existing platforms and recommend modernization, migration, and optimization strategies.

- Drive architectural transformation from legacy Hadoop and on-premise platforms to cloud-native AWS environments.

- Review solution architecture, technical designs, proof of concepts, and implementation approaches.

- Identify architectural risks and define appropriate mitigation strategies.

- Ensure solutions meet scalability, performance, security, availability, and maintainability requirements.

- Provide technical direction to senior engineers, architects, and development teams.

Data Engineering & Platform Architecture :

- Architect large-scale data pipelines using batch and distributed processing frameworks.

- Design robust ETL and ELT architectures for structured, semi-structured, and unstructured data.

- Define data ingestion strategies across APIs, databases, files, event streams, and enterprise applications.

- Design data lake and data warehouse architectures using AWS and modern data technologies.

- Architect high-volume data processing solutions using Apache Spark and Hadoop.

- Define data processing patterns using Spark Core, Spark SQL, RDDs, and DataFrames.

- Establish best practices for data partitioning, caching, serialization, parallelism, and distributed processing.

- Design scalable workflows for complex data transformation and processing requirements.

- Ensure data pipelines are reliable, reusable, observable, and production-ready.

AWS Cloud Architecture :

- Define cloud-native data architectures using AWS services.

- Provide architectural direction for AWS Glue, Amazon S3, Amazon Redshift, Redshift Spectrum, Athena, API Gateway, AWS Lambda, Step Functions, and Amazon EMR.

- Design secure and scalable data storage solutions using Amazon S3.

- Architect ETL and data integration solutions using AWS Glue.

- Design analytical workloads using Amazon Redshift, Redshift Spectrum, and Athena.

- Architect serverless processing solutions using Lambda and Step Functions.

- Design API-based data integration using API Gateway and related AWS services.

- Define appropriate AWS services based on scalability, performance, cost, security, and business requirements.

- Drive cloud optimization and ensure efficient utilization of AWS infrastructure and services.

- Establish cloud architecture standards covering security, networking, monitoring, governance, and cost optimization.

Spark & Scala Architecture :

- Provide technical leadership for large-scale Apache Spark applications.

- Design and optimize Spark jobs running on distributed Hadoop and AWS environments.

- Apply advanced Spark optimization techniques and best practices.

- Architect solutions using Spark Core, Spark SQL, RDDs, DataFrames, and distributed processing concepts.

- Guide teams on memory management, partitioning, shuffling, serialization, parallelism, and query optimization.

- Establish development standards for Scala-based Spark applications.

- Provide guidance on Scala functional programming concepts including Try, Option, Future, and Collections.

- Define best practices around Scala OOP concepts including Classes, Traits, Objects, Singleton and Companion Objects, and Case Classes.

- Guide teams on advanced Scala language features including Type Systems and Implicits/Givens.

- Review complex Spark/Scala implementations and recommend performance and architectural improvements.

Hadoop & Big Data :

- Provide architectural direction for Hadoop-based data platforms.

- Work with HDFS, Hive, Spark, and AWS EMR environments.

- Define strategies for migration and modernization of Hadoop workloads.

- Architect solutions for processing very large datasets across distributed environments.

- Optimize data storage and processing approaches for performance and scalability.

- Define appropriate file formats and storage strategies for large-scale data platforms.

- Work with Apache Avro, Parquet, JSON, and other data formats.

- Exposure to Protocol Buffers and geospatial data processing is an advantage.

Data Pipeline & Workflow Architecture :

- Architect end-to-end data pipelines covering ingestion, validation, transformation, processing, and delivery.

- Define workflow orchestration strategies using Apache Airflow, Oozie, AWS Step Functions, or similar tools.

- Design dependency management, retry, failure handling, and recovery mechanisms for critical data workflows.

- Establish monitoring and operational standards for enterprise data pipelines.

- Ensure data pipelines meet defined SLAs for performance, availability, and data freshness.

- Design reusable frameworks and components for data ingestion and processing.

Programming & Application Development :

- Provide technical leadership across Python, Scala, and object-oriented programming.

- Strong understanding of data structures, algorithms, design patterns, and software engineering principles.

- Review complex application and data processing code for quality, performance, scalability, and maintainability.

- Guide teams in developing reusable and modular components.

- Design and review API integrations and service-based data applications.

- Ensure implementation follows secure coding and enterprise development standards.

Testing & Quality Engineering :

- Define testing strategies for data engineering and Spark applications.

- Establish standards for unit, integration, functional, and performance testing.

- Review test coverage and ensure critical data processing logic is adequately validated.

- Experience with Scala testing frameworks such as ScalaTest.

- Define quality gates for production deployment.

- Drive automated testing and continuous quality improvement across data platforms.

DevOps & Engineering Practices :

- Establish engineering standards around Git-based source control and branching strategies.

- Provide guidance on CI/CD practices for data engineering applications.

- Work with build tools such as SBT and Gradle.

- Review deployment and release strategies for cloud-based data applications.

- Ensure effective version control, code review, release management, and production deployment practices.

- Drive automation across development, testing, deployment, and operational processes.

Performance & Optimization :

- Identify and resolve performance bottlenecks across data pipelines and distributed applications.

- Optimize Spark jobs, SQL queries, data storage, and processing workflows.

- Evaluate application and infrastructure performance against business requirements.

- Define performance benchmarks and capacity planning strategies.

- Recommend architectural improvements to improve processing efficiency and reduce cloud costs.

Security, Governance & Reliability :

- Ensure data platforms comply with enterprise security and governance standards.

- Define appropriate access control, encryption, authentication, and authorization mechanisms.

- Work with security teams to address vulnerabilities and compliance requirements.

- Ensure data architectures support availability, disaster recovery, and business continuity requirements.

- Incorporate data quality, lineage, governance, and auditability into solution architecture.

- Establish appropriate monitoring, logging, alerting, and operational controls.

Client & Stakeholder Management :

- Act as a senior technical advisor for business and technology stakeholders.

- Engage with senior leadership and clients to understand strategic technology requirements.

- Translate business objectives into scalable technology and data architecture solutions.

- Present architecture proposals, technical recommendations, and modernization roadmaps to senior stakeholders.

- Participate in technical workshops, architecture reviews, and solution discussions.

- Manage technical dependencies and drive alignment across multiple engineering teams.

Must-Have Skills :

- 22 - 25 years of overall technology/software engineering experience.

- Extensive experience in enterprise data engineering and architecture.

- Strong hands-on expertise in Apache Spark and Scala.

- Strong experience with AWS data services including AWS Glue, S3, Redshift/Redshift Spectrum, Athena, EMR, Lambda, Step Functions, and API Gateway.

- Strong experience designing and implementing ETL/ELT data integration solutions.

- Extensive experience designing large-scale data pipelines.

- Strong experience with Hadoop, HDFS, Hive, and distributed data processing.

- Strong Python programming skills.

- Deep knowledge of Spark Core, Spark SQL, RDDs, DataFrames, and Spark optimization.

- Strong understanding of Scala functional programming and OOP concepts.

- Strong understanding of data structures and algorithms.

- Experience with workflow orchestration tools such as Airflow or Oozie.

- Strong knowledge of data formats including Avro, Parquet, and JSON.

- Strong experience with Git, branching strategies, and code management.

- Experience with build tools such as SBT and Gradle.

- Strong analytical, problem-solving, communication, and technical leadership skills.