Data Architect - ETL/Data Warehousing

Description :

The core responsibilities for the job include the following :

Data Architecture and Infrastructure :

- Design and build robust, scalable, and high-performance data architectures that enable seamless data access and analysis.

- Develop, optimize, and manage data pipelines and ETL processes for data ingestion from diverse data sources.

- Lead the design and implementation of cloud-based data warehousing and big data solutions (AWS, GCP, Azure).

Data Pipeline Optimization :

- Develop, maintain, and optimize data pipelines that efficiently move, transform, and process large volumes of structured and unstructured data.

- Implement best practices for data pipeline automation, ensuring real-time data availability and minimal downtime.

Advanced Data Solutions :

- Work on the integration of advanced data processing frameworks such as Spark, Kafka, and Hadoop for batch and real-time data streaming.

- Collaborate with data science teams to deploy machine learning models and pipelines into production.

Mentorship and Leadership :

- Act as a technical mentor to junior and mid-level data engineers, guiding best practices, code reviews, and technical problem-solving.

- Lead by example in coding standards, technology decisions, and system architecture.

Collaboration :

- Work closely with data scientists, customer success, product teams, and other stakeholders to translate business needs into scalable data solutions.

- Ensure seamless data flow and availability for different teams to drive insights, reporting, and decision-making.

Performance and Security :

- Ensure data integrity, security, and compliance with company policies and industry standards.

- Optimize data storage, retrieval, and scalability to improve performance and reduce costs.

Innovation and Continuous Improvement :

- Stay up to date with emerging technologies and data engineering methodologies, continuously improving data infrastructure.

- Proactively propose innovative solutions to improve the efficiency and reliability of data systems.

Requirements :

- 12+ years of experience in data engineering or related fields.

- Strong experience in designing and building scalable data pipelines and architectures.

- Extensive hands-on experience with cloud platforms such as AWS, GCP, or Azure.

- Proven experience with big data technologies (e.g., Hadoop, Spark, Kafka, Hive).

- Strong experience with data warehousing solutions (e.g., Redshift, Snowflake, BigQuery, etc.) and data modeling.

- Must have designed and built at least two scalable systems.

Technical Skills :

- Proficiency in programming languages like Python, PySpark, or Scala.

- Strong experience in designing robust APIs and microservices.

- Experience with data pipeline orchestration tools (e.g., Apache Airflow, Kubeflow, etc.)

- Deep understanding of ETL/ELT processes and data modeling.

Problem Solving and Innovation :

- Strong analytical skills with the ability to solve complex technical challenges.

- Experience in troubleshooting data systems and optimizing their performance.

- Ability to evaluate and recommend new technologies and tools to improve data systems.

Collaboration and Communication :

- Excellent communication skills, with the ability to explain complex technical issues to both technical and non-technical stakeholders.

- Ability to work in a collaborative, cross-functional environment with multiple stakeholders.

- Strong interpersonal skills and experience mentoring other engineers.

Good To Have :

- Hands-on experience with machine learning pipelines and working with data science teams.

- Experience with data governance frameworks and compliance (e.g., GDPR, CCPA).

- Experience in real-time data streaming and processing technologies (e.g., Kafka, Kinesis, Flink).

- Good knowledge of SQL and database technologies.

- Familiarity with containerization (e.g., Docker) and orchestration (e.g., Kubernetes) is a plus.

- Experience building data pipelines for CPG/Retail industries.