About the Role:

Our teams are strong at building and scaling products. Every service has an owner, every feature has engineers behind it, and every database has deep expertise. What we need now is someone who can look across all of themsomeone who understands how the entire distributed system works together and can identify risks that may not be visible within an individual service. A retry policy in one service, combined with a queue configuration several hops downstream, can become a cascading failure under peak load. Your role is to identify those risks before they reach production.

You'll start with Workflows, HighLevel's automation engine and one of the largest systems in the company. It handles 3.1 billion enrollments and 21.5 billion action executions every month, with traffic peaking at 28,000+ requests per second. The platform runs across thousands of pods on GCP and uses Pub/Sub, Cloud Tasks, Redis, and multiple database engines. From there, your scope will expand into Conversations, which processes 2.6 billion messages every month, as well as our new Ticketing platform. With Ticketing, you'll have the opportunity to influence the architecture early, building resilience and scalability into the system from the start rather than addressing them later. This is not a traditional architect role where you design systems and hand them over for implementation. It is also not a feature-focused engineering role with a defined product backlog.

This role sits across systems and teams, with a clear mandate: understand how the platform behaves as a whole, identify systemic risks, and make sure our architecture can continue to scale reliably.

Responsibilities:

- Own the architecture health of a billion-scale distributed system, its failure modes, capacity limits, consistency guarantees, and the interactions between 50+ deployments that no single team can see.

- Approve critical-path designs. Changes that touch the system's core go through you, not as bureaucracy, but as the person accountable for the whole staying sound. When there's a disagreement, you make your case on merit.

- Hunt gaps proactively, single points of failure, unbounded queues, missing idempotency, thundering herds, quiet data-loss windows, and drive the fixes before they become incidents.

- Build the parts nobody else can. No sprint tickets. You prototype the risky architectural bets yourself, ship the remediation after serious incidents, and pair into the gnarliest cross-team bugs, roughly a quarter to a third of your time in code, all of it on the hardest problems.

- Make resilience a property of the system, not a heroic act: degradation strategies, backpressure, isolation boundaries, capacity models that survive 10%+ month-over-month growth.

- Raise the teams around you. Design reviews that teach, post-mortems that change architecture (not just add alerts), and patterns that 80+ engineers build on.

- Set the standard for how AI-assisted engineering works safely on systems this critical, where a bad merge doesn't cost a demo; it costs real businesses their revenue.

The Terrain:

- Runtime: Node.js (TypeScript), Go, thousands of pods on GKE.

- Messaging and async: GCP Pub/Sub, Cloud Tasks, Redis.

- Storage: MongoDB, Firestore, ClickHouse, Elasticsearch.

- Scale: 21.5B automation actions/month, 2.6B messages/month, 28.5K req/s peaks, ~226B async events across the org.

Requirements:

- 10+ years of engineering experience with deep, hands-on ownership of large-scale distributed systems, comparable scale strongly preferred: hundreds of services or thousands of instances, billions of daily events.

- You've carried sole accountability for a production system through real failures, not adjacent to it, not advising on it. Owned it.

- Deep command of queueing and async architectures, delivery semantics, ordering, backpressure, idempotency, exactly-once myths and at-least-once realities.

- Strong with multiple storage engines (SQL and NoSQL); you reason about consistency models, indexing at scale, and when each engine is the wrong choice.

- Expert-level depth in Redis or comparable in-memory systems, including their failure modes under memory pressure and network partition.

- Production experience on Kubernetes at scale, resource limits, autoscaling behaviour, and what actually happens when a node pool dies.

- Exceptional design communication, docs, diagrams, and RCAs that drive decisions across multiple teams.

- Fluent in Node.js and/or Go, enough to prototype your own proposals and ship fixes on the critical path.

Bonus Points:

- You've made AI agents genuinely productive on complex systems, and know how to keep AI-generated code from becoming AI-generated incidents.

- GCP-native experience: Pub/Sub, Cloud Tasks, GKE, Firestore.

- You've done this job before under another name: "the systems person," principal engineer, architect-who-still-codes.

- Experience taking a 0-1 system to production alongside hardening mature ones.