Role : Senior Member - Data Engineer, Core Tech

Level : L2/L3 (Senior Individual Contributor)

Experience Range : 2- 4 years (2023, 2022, 2021)

Work Location : Hyderabad, Bengaluru, Gurugram (open to candidates from any of these locations)

Education : B.Tech. degree in Computer Science or related discipline, with a strong foundation in data structures and algorithms

Role Overview :

As part of the Data Engineering team within Core Tech, the Senior Member will design, build, and maintain web robots (web spiders) to extract diverse data formats (HTML, text, PDFs, Excel, etc.) from public websites and transform them into structured, usable datasets. The role involves reverse-engineering websites, automating extraction processes, building ETL pipelines, parsing raw data, and storing it efficiently.

You will be responsible for monitoring systems to ensure data quality and availability, managing dynamic website changes, and collaborating closely with business stakeholders to meet data requirements. This position offers an opportunity to work on end-to-end data pipeline automation in a large-scale environment where extracted data directly supports key business decisions.

Must-haves :

- 2- 4 years of website data extraction and scraping experience.

- Strong knowledge of relational databases, complex SQL queries, and ETL operations.

- Proficiency in Python for data operations.

- Expertise with Python frameworks for scraping such as Requests, UrlLib2, Selenium, BeautifulSoup, Scrapy.

- Understanding of HTTP requests/responses, HTML, CSS, XML, JSON, and JavaScript.

- Experience using Chrome DevTools for reverse engineering website behavior and layout changes.

- Solid problem-solving, analytical, and debugging skills.

- Experience building automated data pipelines and ensuring high data integrity in production environments.

Good to have :

- Experience storing and managing data in recommended database systems for high performance.

- Background building scalable, high-performing data extraction systems for production workflows.

- Familiarity with operational monitoring systems to ensure high availability and zero data loss.

- Prior exposure to the financial domain or large-scale tech/data companies.