Role : Senior Software Architect / Principal Engineer - Threat Intelligence Platform

Experience : 10+ Years (4+ Years in Architect / Technical Lead Capacity)

Employment Type : Full-Time

Location : Bengaluru

Reporting To : CTO / VP Engineering

Domain : Cyber Threat Intelligence (CTI), OSINT, Dark Web Monitoring, Telegram Intelligence

About the Role :

We are building a next-generation Cyber Threat Intelligence platform that continuously ingests intelligence from OSINT sources, Telegram channels, dark-web ecosystems, and other external threat feeds. Our platform leverages custom AI/LLM pipelines to enrich, correlate, classify, and deliver actionable intelligence to enterprise security teams.

We are seeking an experienced Senior Software Architect / Principal Engineer to lead the end-to-end architecture of our platformfrom large-scale data collection and AI-driven enrichment to multi-tenant intelligence delivery. This role requires a hands-on technical leader who can define architecture, drive engineering excellence, mentor teams, and transform rapidly developed systems into highly scalable, resilient, secure, and cost-efficient platforms.

Key Responsibilities :

1. Platform Architecture & Engineering :

- Own and drive the architecture of a large-scale, event-driven threat intelligence platform.

- Design scalable ingestion, processing, enrichment, storage, and presentation pipelines.

- Build resilient distributed systems capable of handling high-volume intelligence feeds.

- Define architectural standards, technical roadmaps, and engineering best practices.

2. Distributed Systems & Scalability :

- Design asynchronous, queue-driven processing architectures using Celery, Redis, Pub/Sub, or equivalent technologies.

- Implement checkpointed crawling, fault-tolerant workflows, retry mechanisms, circuit breakers, and backoff strategies.

- Architect deduplication and near-duplicate detection systems using techniques such as hashing, SimHash, and similarity matching.

- Ensure horizontal scalability and high availability across cloud and on-premise infrastructure.

3. AI & LLM Architecture :

- Design and oversee AI-driven enrichment pipelines powered by multiple LLM providers.

- Build provider-agnostic architectures supporting Gemini, Groq, Cerebras, SambaNova, and future AI providers.

- Implement intelligent routing, fallback strategies, API key rotation, rate-limit handling, and cost optimization.

- Define schema-constrained extraction, entity resolution, classification, summarization, and intelligence correlation workflows.

- Drive adoption of custom-trained LLM models and AI-powered threat detection capabilities.

4. Cloud & Infrastructure Leadership :

- Lead cloud architecture and infrastructure decisions across GCP and hybrid environments.

- Optimize workloads using Cloud Run, Compute Engine, Cloud SQL, BigQuery, Dataflow, GCS, Memorystore, and related services.

- Develop capacity planning and infrastructure cost optimization strategies.

- Define Infrastructure-as-Code and deployment automation standards.

5. Security, Reliability & Operations :

- Establish platform-wide security standards and secure-by-design principles.

- Implement secrets management, service authentication, least-privilege access controls, and auditability.

- Build comprehensive observability using logging, monitoring, tracing, alerting, and SLO frameworks.

- Drive CI/CD maturity and operational excellence.

6. Technical Leadership :

- Mentor engineering teams and conduct architecture reviews.

- Create Architecture Decision Records (ADRs) and technical design documentation.

- Define coding standards, development workflows, and engineering best practices.

- Collaborate with Product, Threat Intelligence, and Leadership teams to translate business requirements into scalable technical solutions.

Required Qualifications :

1. Experience :

- 10+ years of experience building and operating production-grade backend or distributed systems.

- Minimum 4+ years serving as a Software Architect, Principal Engineer, or Technical Lead.

2. Backend Engineering :

- Expert-level proficiency in Python.

- Strong experience with : 1. FastAPI 2. Flask 3. SQLAlchemy 4. Async Python 5. Celery 6. Redis.

- Strong proficiency in TypeScript and Node.js.

3. Distributed Systems :

- Extensive experience designing event-driven and queue-based architectures.

- Proven expertise with asynchronous processing, worker orchestration, and distributed workflows.

4. Cloud Platforms :

- Strong hands-on experience with GCP and/or AWS/Azure services, including : 1. Compute Engine 2. Cloud Run 3. Cloud Functions 4. Cloud SQL (PostgreSQL) 5. Google Cloud Storage (GCS) 6. BigQuery 7. Dataflow 8. Secret Manager 9. Networking and Security Services.

5. Data Engineering :

- Deep understanding of PostgreSQL performance tuning and data modeling.

- Experience building multi-storage architectures combining : 1. OLTP Databases 2. Analytics Platforms 3. Object Storage Systems.

- Expertise in deduplication, similarity detection, and large-scale data processing.

6. AI & LLM Engineering :

- Proven experience integrating LLMs into production systems.

- Experience with : 1. Prompt Engineering 2. Structured Output Extraction 3. Multi-Provider AI Routing 4. AI Cost Governance 5. Throughput Optimization 6. Rate-Limit Management.

7. DevOps & Security :

- Strong experience with : 1. Docker 2. Docker Compose 3. CI/CD Pipelines 4. Infrastructure Automation 5. Secrets Management.

- Solid understanding of : 1. Zero Trust Principles 2. Service Authentication 3. IAM 4. Least Privilege Access Models.

8. Cost Optimization :

- Demonstrated ability to optimize infrastructure spend and scale systems efficiently based on measurable business requirements.

Preferred Qualifications :

1. Threat Intelligence & Security :

- Cyber Threat Intelligence (CTI), OSINT Platforms, Dark Web Intelligence, Telegram Intelligence Collection, SOC Platforms, Threat Hunting Systems.

2. Data Collection & Scraping :

- Large-scale web scraping architectures, Telethon, Headless Browser Automation, Tor-based Collection Systems, Anti-Bot Evasion Techniques, Rate-Limit Management.

3. Standards & Integrations :

- TAXII, STIX, REST API Design, SIEM Integrations.

4. Machine Learning & NLP :

- spaCy, DeBERTa, Entity Extraction, Classification Pipelines, Zero-Shot Learning, Self-hosted Inference Platforms (1. vLLM 2. Ollama).

5. Frontend Technologies :

- React, Next.js, TypeScript, Ant Design.

6. SaaS Architecture :

- Multi-Tenant SaaS Platforms, Tenant Isolation, Role-Based Access Control (RBAC).

What Success Looks Like (First 6 Months) :

- A consolidated, version-controlled platform eliminating infrastructure drift and ad-hoc deployments.

- Reliable AI enrichment pipelines capable of processing intelligence feeds without backlog accumulation.

- Comprehensive observability, monitoring, alerting, and operational dashboards.

- Secure secrets management through centralized vault systems.

- A documented target-state architecture and phased migration roadmap.

- Measurable reductions in infrastructure and AI operational costs while improving platform reliability and scalability.

Technology Stack :

1. Backend & Processing : Python, FastAPI, Flask, Celery, Redis, SQLAlchemy.

2. Frontend : Next.js, React, TypeScript, Ant Design.

3. Databases & Storage : PostgreSQL, Cloud SQL, BigQuery, GCS, MinIO.

4. Cloud & Infrastructure : Google Cloud Platform (GCP), Cloud Run, Compute Engine, Cloud Functions, Dataflow, Secret Manager.

5. AI & Machine Learning : Gemini, Cerebras, Groq, SambaNova, Custom LLM Models, spaCy, DeBERTa.

6. Security & Intelligence : Tor, TAXII, STIX, OSINT Collection Frameworks, Threat Intelligence Integrations.