Key Responsibilities :

- Develop and maintain robust web scraping and web crawling solutions using Python.

- Build data extraction pipelines to collect, clean, transform, and store information from multiple web sources.

- Develop and optimize scraping solutions for both static and dynamic websites.

- Work with frameworks and libraries such as Requests, Scrapy, BeautifulSoup, Selenium, Puppeteer, or Playwright.

- Design efficient solutions for handling large-scale data extraction and crawling requirements.

- Integrate scraped data with SQL databases and data warehousing platforms.

- Perform data validation, cleansing, and quality checks to ensure accuracy and consistency.

- Troubleshoot scraping failures, website changes, anti-bot challenges, and performance issues.

- Optimize crawlers for scalability, reliability, and execution speed.

- Collaborate with data, engineering, and business teams to understand data requirements and deliver effective solutions.

Required Skills :

- 5+ years of experience in Python development with strong hands-on programming expertise.

- Proven experience in Web Scraping / Web Crawling.

- Strong knowledge of Python programming and scripting.

- Hands-on experience with SQL and any database or data warehousing technology.

- Experience with one or more of: Requests, Scrapy.

- Experience with one or more of: Puppeteer, Playwright, BeautifulSoup, Selenium.

- Knowledge of PySpark is preferred.

- Strong understanding of HTML, CSS, DOM, HTTP requests, and web technologies.

- Good debugging, analytical, and problem-solving skills.

Preferred Candidate Profile :

- Strong hands-on experience in building production-grade web scraping and crawling solutions.

- Experience working with large datasets and high-volume data extraction.

- Ability to work independently and handle complex scraping requirements.

- Strong understanding of data processing, database integration, and automation.