Job Description :

Title : Lead Market Intelligence & Engineering

Experience : 6 to 10 years

Location : Bangalore

Must Have :

- Data infrastructure roles.

- Deep hands-on expertise with large-scale crawling frameworks (Scrapy, Playwright, Puppeteer, or custom solutions) and techniques to handle dynamic pages, CAPTCHAs, and anti-bot measures.

- Strong proficiency in Python and SQL; experience with data pipeline orchestration tools (Airflow, Dagster, or similar).

- Experience building and operating real-time or near-real-time data pipelines (Kafka, Flink, or equivalent).

- Proven track record of collecting, structuring, and delivering e-commerce or marketplace data at scale.

- Experience with cloud infrastructure (AWS preferred) for deploying and scaling crawling workloads.

- Strong understanding of data quality, monitoring, and observability practices.

Role And Responsibilities :

1. External Data Collection & Crawling Infrastructure :

- Design, build, and maintain scalable crawling systems to collect structured data from marketplaces (Amazon, Flipkart, Shopee, Lazada, Noon, etc.), competitor sites, and public data sources across India, SEA, and GCC.

- Data to be collected includes (but is not limited to) :

1. Product listings, pricing, promotions, and discounts across platforms

2. Category rankings, bestseller lists, search result positions

3. Competitor assortment, new launches, and catalogue changes

4. Customer ratings, reviews, and sentiment signals

5. Ad placement data, sponsored product visibility, and share-of-voice metrics

6. Brand presence and market penetration signals by country

- Handle anti-scraping mechanisms, rate-limiting, geo-restrictions, and platform-specific nuances across 15+ marketplace environments.

- Build real-time and near-real-time data feeds where speed of insight matters (e.g., competitor price changes, flash sale monitoring, ad bid landscapes).

2. Data Structuring, Quality & Enrichment :

- Normalize and structure heterogeneous data from diverse platforms into a unified, queryable data layer that different teams can consume reliably.

- Build entity resolution and matching systems to map products, brands, and categories consistently across marketplaces and regions.

- Implement data quality monitoring, anomaly detection, and freshness tracking to ensure data reliability for downstream decision-making.

- Enrich external data with taxonomy, categorization, and metadata tagging relevant to Opptras category focus (consumer electronics, home & kitchen, babycare, FMCG, etc.).

3. Decision Support Across Business Functions :

The data you collect and organize will directly power decisions across :

- Performance Marketing Competitor ad spend signals, keyword opportunity mapping, category share-of-voice, price position relative to competition feeding real-time bidding and budget allocation decisions.

- Pricing & Promotions Competitive price monitoring, discount depth analysis, marketplace fee structures, and margin-optimal price recommendations across platforms and regions.

- Buying & Inventory Demand signals from search trends, bestseller velocity, seasonal patterns, and category growth indicators to inform what to buy, how much, and when.

- Brand Onboarding Market attractiveness scoring by country and category, whitespace identification, competitive landscape assessment, and brand-market fit analysis to guide which brands to pursue and where.

- Category Intelligence Category size estimation, growth trajectories, consumer preference shifts, and regulatory landscape changes across target markets.

4. Integration with Internal Data & Analytics :

- Work closely with internal data engineering and analytics teams to combine external intelligence with Opptras operational data (sales, inventory, logistics, marketing performance).

- Build unified data products and dashboards that give platform teams, country heads, and category managers a single view combining market context with internal performance.

- Ensure data pipelines are production-grade, well-documented, and serve both batch analytics and real-time operational needs.

5. Scale & Innovation :

- Continuously expand coverage to new platforms, data sources, and geographies as Opptra scales.

- Explore and integrate alternative data sources (social listening, creator/influencer performance data, regulatory databases, trade data).

- Leverage AI/ML to automate data extraction from unstructured sources, improve entity matching, and generate predictive signals from collected data.