Lead Python Developer - Backend Technologies

Position Summary :

Job Responsibility :

- Develop and maintain web scraping solutions to extract, process, and manage large-scale data from multiple sources.

- Work with relational and non-relational databases, ensuring high availability, scalability, and performance.

- Implement authentication, authorization, and security best practices across services.

- Write clean, maintainable, and testable code following best practices and coding standards.

- Collaborate with frontend engineers, data engineers, and DevOps teams to deliver robust

solutions and troubleshoot, debug, and upgrade existing applications.

- Stay updated with backend development trends, tools, and frameworks to continuously

improve processes. Utilize core crawling experience to design efficient strategies for scraping

the data from different websites and applications.

- Collaborate with technology teams, data collection teams to build end to end technology-

enabled ecosystems and partner in research projects to analyze the massive data inputs.

- Responsible for the design and development of web crawlers, able to independently solve

various problems encountered in the actual development process.

- Stay updated with the latest web scraping techniques, tools, and industry trends to continuously improve the scraping processes.

Job Requirements :

- Strong understanding of TypeScript-based server-side frameworks (e.g., Node.js, NestJS, Express) and Python frameworks (e.g., FastAPI, Django, Flask).

- Experience with tools and libraries for web scraping (e.g., Scrapy, BeautifulSoup, Selenium, Puppeteer)

- Hands-on experience with Temporal for creating and orchestrating workflows

- Proven hands-on experience in web scraping, including crawling, data extraction,

deduplication, and handling dynamic websites.

- Proficient in implementing proxy solutions and handling bot-detection challenges (e.g., Cloudflare).

- Experience working with Docker, containerized deployments, and cloud environments (GCP or Azure).

- Proficiency with database systems such as MongoDB and ElasticSearch.

- Hands-on experience with designing and maintaining scalable APIs.

- Knowledge of software testing practices (unit, integration, end-to-end).

- Familiarity with CI/CD pipelines and version control systems (Git).

- Strong problem-solving skills, attention to detail, and ability to work in agile environments.

- Great communication skills and ability to navigate in undirected situations.

Job Exposure :

- In depth industry exposure on data harvesting techniques to build, scale the robust and sustainable model, using open-source applications

- Effectively collaboration with IT team to design the tailor-made solutions basis upon clients requirement

- Unique opportunity to research on various agencies, vendors, products as well as technology tools to compose a solution