A user loads the list of URLs into CSV file. The system loads them into the database and links shops and products so that one product may be assigned to multiple shops. It allows viewing of product prices at multiple shops at the same time. After that, the system scrapes product data: price, title, category, shipping cost, product availability.
In the backend, a user can schedule the parsing process and setup timeout delay for every hour of work. For instance, at night timeout delay is longer (and scraping speed is lower, accordingly) and during the day the speed is max.
The system watches price changes and notifies a user when the price goes down. Each product price goes through the following process:
- adding the cost for the shipment, considering the user's location.
- checking the price against external coupons (added by a user before)
- re-calculating price for hidden discounts (some shops give a discount when a product is added to the shopping cart).
When the price goes down to the amount previously set by a user, the product is added to a section for the most attractive deals and prices, where a user can monitor them. So even the price hasn't changed but shipping cost changed, a user will be notified.
Let us know if you require something like that, or take a look at the similar price monitoring solutions we’ve created.