Job Summary :

We are seeking an experienced professional to lead Market Intelligence and Data Engineering initiatives focused on collecting and analyzing external marketplace data. This role involves designing scalable data collection systems, building reliable data pipelines, and delivering insights that support strategic business decisions.

Key Responsibilities :

External Data Collection & Crawling Infrastructure :

- Design and maintain scalable data collection systems to gather structured information from online marketplaces and public data sources

- Collect and monitor data such as product listings, pricing, promotions, product rankings, reviews, and competitor activity

- Develop systems capable of operating across multiple e-commerce platforms and geographic markets

- Build real-time or near-real-time data pipelines for monitoring price changes, promotions, and marketplace trends

- Ensure reliable data collection while managing platform constraints such as rate limits and regional restrictions

Data Structuring, Quality & Enrichment :

- Transform and normalize data from multiple sources into a unified and structured data model

- Develop systems to map products, brands, and categories across platforms

- Implement monitoring processes to maintain data quality, reliability, and freshness

- Enhance datasets with taxonomy, categorization, and relevant metadata

Decision Support for Business Functions :

Provide data insights that support teams across areas such as :

Marketing :

- Competitive signals, search performance insights, and category visibility

Pricing :

- Monitoring market prices, discounts, and competitive positioning

Inventory & Buying :

- Demand signals, category trends, and seasonal patterns

Market Expansion :

- Identifying opportunities across regions and categories

Integration with Internal Data Systems :

- Collaborate with internal data and analytics teams to combine external market intelligence with operational data

- Develop data products and dashboards that support business decision-making

- Maintain reliable, production-grade pipelines for both batch analytics and real-time insights

Continuous Improvement & Innovation :

- Expand data coverage across additional platforms and markets

- Explore new data sources such as social insights or market signals

- Apply AI/ML techniques to improve data extraction, matching, and predictive insights

Required Skills :

- 6 - 10 years of experience in data engineering, web crawling, or market data infrastructure

- Strong expertise in Python and SQL

- Experience with web crawling frameworks such as Scrapy, Playwright, Puppeteer, or similar tools

- Experience building data pipelines using tools such as Airflow, Dagster, or similar orchestration frameworks

- Experience working with real-time data processing tools such as Kafka, Flink, or similar platforms

- Experience deploying scalable systems on cloud infrastructure (AWS preferred)

- Strong understanding of data quality, monitoring, and observability practices

Preferred Experience :

- Experience working with e-commerce or marketplace data environments

- Knowledge of large-scale data collection and structured data systems

- Exposure to AI/ML techniques for data extraction and analysis