Job Description:

Web Scraping and Crawling, AWS Bedrock (AI Development For Scraping)

Role Overview:

We are looking for an experienced Senior Web Scraping / Data Extraction Engineer with 6+ years of experience owning scraping platforms end-to-end. The ideal candidate should have strong expertise in web crawling, web scraping, data extraction, and scraping frameworks such as Scrapy, Playwright, and Crawlee.

The role will also involve mentoring engineers, establishing crawling standards, reviewing engineering practices, and leveraging LLM-assisted extraction using AWS Bedrock in a cost-controlled manner.

Key Responsibilities:

- Own and manage the web scraping platform end-to-end, from architecture and development to optimization and maintenance.

- Design, develop, and maintain scalable web crawling and scraping solutions.

- Establish and maintain standards and best practices for web crawling and data extraction.

- Review engineering practices, code, and scraping approaches to ensure quality and scalability.

- Mentor engineers and provide technical guidance on scraping and data extraction practices.

- Develop and optimize scraping solutions using Scrapy, Playwright, and Crawlee.

- Contribute to open-source projects and libraries related to Scrapy, Playwright, or data parsing.

- Implement LLM-assisted data extraction and verification solutions.

- Leverage AWS Bedrock for LLM-based verification while maintaining control over operational costs.

- Identify opportunities to improve scraping efficiency, reliability, and data quality.

- Collaborate with engineering and data teams to build robust data extraction solutions.

Required Skills & Experience:

- 6+ years of experience owning and managing web scraping/crawling platforms end-to-end.

- Strong experience with Web Scraping and Web Crawling.

- Hands-on expertise with:

1. Scrapy

2. Playwright

3. Crawlee

4. Data parsing and extraction

- Experience mentoring engineers and establishing technical standards.

- Strong understanding of scraping architecture, crawling practices, and data extraction.

- Experience with open-source contributions to Scrapy, Playwright, or parsing libraries.

- Experience with LLM-assisted extraction.

- Hands-on experience with AWS Bedrock for LLM-based verification.

- Strong focus on controlling operational costs while implementing LLM-based solutions.

Key Skills:

Scrapy, Playwright, Crawlee, Web Scraping, Web Crawling, Data Scraping, Data Extraction, Data Engineering, AWS Bedrock, LLM, LLM-Assisted Extraction, Data Parsing, Scraping Platform, Open-Source Contributions.