Senior Python Engineer Web Data Extraction at Scale

We extract live insurance quotes from a dozen carrier portals in parallel, in under 10 seconds, and feed them into a product that trucking companies use to buy coverage. If you've built and maintained large-scale extraction systems the kind where target sites actively resist you and break your parsers every few weeks this is that problem, with a real business on the other end.

Please read this before applying this is not what you may be thinking.

This is NOT RPA or no-code automation. No Zapier, Make, n8n, Power Automate, UiPath, Automation Anywhere, or Blue Prism. You'll be writing Python and reasoning about network behavior and the DOM, not dragging boxes in a builder.

This is NOT QA or test automation.

You will not test our software. You'll extract from other companies' sites in production, at speed, while they try to block you. E2E regression testing against an app you control is a different discipline; we've reviewed nearly 200 resumes and can tell immediately we will ask, and we will check.

The interesting parts :

- Latency as a hard constraint (most extraction is batch and overnight ours is synchronous, user-facing, with a 10-second ceiling, so concurrency design is the whole game)

- Adversarial targets (bot detection, CAPTCHAs, session expiry, rate limits, silent throttling, multi-step authenticated forms)

- Correctness matters (these are insurance quotes a wrong number is a business liability, not a bad data point)

- Self-healing is the goal (today a human fixes each breakage by hand and the fix is lost we want every breakage to produce a permanent fix)

What we need :

- 5+ years production Python

- You have built and operated extraction systems against sites that didn't want to be extracted from Playwright, Selenium, Puppeteer, Scrapy; the tool matters less than the scar tissue

- Deep familiarity with anti-bot countermeasures : proxy rotation, fingerprinting, header and user-agent strategy, session and cookie persistence, CAPTCHA handling

- Async Python and the discipline to make parallel work finish inside a fixed time budget

- Parser resilience you have opinions about why brittle selectors fail and what to do instead

- Pipeline reliability : retries, backoff, circuit breakers, dead-letter queues, meaningful alerting

Strong pluses :

- Orchestration at scale (Airflow, Celery, Temporal)

- Document parsing and OCR (Tesseract, Textract, pdfplumber, PyMuPDF)

- LLM-assisted extraction or field mapping across inconsistent formats

- Regulated domains (insurance, finance, healthcare, compliance, trade)