Netscribes - Web Scraping API Developer - Python

Job Description :

Key Responsibilities :

- Web Scraping & Data Ingestion : Design, build, and maintain sophisticated web scraping applications to ingest competitor data from diverse online retail platforms.

- API Development : Develop scalable RESTful APIs in Python (FastAPI / Flask / Django) to serve extracted data for downstream consumption by internal and external systems.

- Deployment & Orchestration : Containerize and deploy API services using Docker and orchestrate deployments via Azure DevOps / GitHub Actions.

- Azure Cloud Utilization : Leverage a wide array of Azure services including Azure Functions, Azure App Services, Azure Data Lake, Azure Key Vault, and Azure Monitor / Application Insights for end-to-end deployment, monitoring, and observability.

- Robust Pipeline Development : Implement comprehensive logging, exception handling, and retry mechanisms to build highly robust and fault-tolerant scraping and ingestion pipelines.

- Tooling & Libraries : Integrate and utilize open-source libraries such as BeautifulSoup, Scrapy, Playwright, Puppeteer, and Selenium for scalable and efficient web data extraction.

- Data Collaboration : Collaborate closely with Data Scientists and Analysts to ensure the scraped data is queryable, usable, and valuable for analytics.

- Monitoring & Dashboards : Build dashboards or simple frontends (if needed) to effectively monitor scraping coverage, API health, and overall system performance.

- Development Practices : Adhere to best practices for Git-based version control, work within agile methodologies using tools like JIRA/Confluence, and actively participate in code reviews and CI/CD cycles.

Required Skills & Qualifications :

Programming and APIs :

- Strong proficiency in Python (3.x) with significant production experience.

- Proven expertise in REST API development using frameworks like FastAPI, Flask, or Django REST Framework.

- Extensive experience with Web Scraping tools such as BeautifulSoup, Scrapy, Playwright/Selenium.

- Solid understanding of JSON handling, data transformation, and robust exception management.