Role Overview :

We are looking for a Lead Data Architect to design, build, and scale our data pipelines and entity resolution systems. This role combines deep technical expertise in data engineering with hands-on experience in AI-assisted tooling, entity matching, and data integration from diverse sources. You will lead architectural decisions for our data platform, mentor engineers, and ensure our pipelines are reliable, scalable, and production-grade.

Key Responsibilities :

- Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple internal and external sources.

- Own the end-to-end orchestration of data workflows using tools like Dagster, ensuring observability, reliability, and maintainability of pipelines.

- Design and implement entity resolution workflows - including matching, merging, and survivorship logic - using tools such as Splink, to produce clean, deduplicated, golden records.

- Build and maintain web scrapers to source data from external providers, ensuring resilience to source changes, rate limits, and data quality issues.

- Integrate and reconcile data coming from multiple, often inconsistent, sources into unified, trustworthy datasets.

- Design and maintain data models and schemas across transactional and analytical systems, ensuring consistency, scalability, and performance.

- Leverage AI/LLM-based tools and techniques to enhance data pipeline capabilities - e.g., intelligent data extraction, automated data quality checks, or AI-assisted entity matching.

- Define and enforce best practices around pipeline design, testing, monitoring, and documentation.

- Collaborate closely with data engineers, product managers, and other stakeholders to translate business requirements into scalable data architecture.

- Provide technical leadership and mentorship to the data engineering team.

Required Skills & Experience :

- Strong hands-on experience building and maintaining production-grade data pipelines at scale.

- Practical experience with Dagster (or similar orchestration tools like Airflow/Prefect) for pipeline orchestration.

- Experience with Splink or similar probabilistic/deterministic record linkage tools for entity matching, merging, and survivorship.

- Strong proficiency in Python, including experience writing and maintaining web scrapers.

- Proven experience integrating and maintaining data pipelines that pull from multiple, heterogeneous data sources.

- Experience applying AI/ML tools within data engineering workflows (e.g., LLM-assisted data cleaning, extraction, or matching).

- Hands-on experience with relational and distributed databases such as PostgreSQL and Google Cloud Spanner.

- Strong understanding of data modeling principles (normalization, dimensional modeling, schema design) across OLTP and OLAP systems.

- Experience with cloud data warehousing platforms such as BigQuery, Redshift, and cloud platforms (GCP/AWS/Azure).

- Strong communication skills and experience working cross-functionally with engineering and product teams.

- Experience with distributed data processing frameworks (e.g., Spark, Dask).

- Familiarity with data governance, lineage, and cataloging tools.

- Prior experience in a lead or architect-level role guiding a data engineering team.

What We're Looking For :

A technically strong, hands-on leader who can balance architectural thinking with the practical grit of debugging a flaky scraper or tuning a matching algorithm - someone who's comfortable owning both the big picture and the messy details of real-world data.