View Single Post
  #16  
Old 05-15-2007, 05:50 PM
TLC TLC is offline
Senior Member
 
Join Date: Aug 2003
Location: Virginia
Posts: 106
Default Re: Price Comparison Website Questions

You could write a multi-threaded screen scraper which would glean the pricing/availability info from the vendor sites in realtime.

This is pretty easy if there is a common "item number" like an ISBN for books. If there is not, then you will need to set up a process to "crawl" a vendor's site daily to store a mapping between your item # and each vendor's item #.

I did something this about 7 years ago using ASP and MTS. It typically went to a vendor's product page using the Item # and parsed the HTML using a vendor-specific RegEx expression. I did this for multiple vendors concurrently and had a timeout to "give up" on a vendor if that request took too long.

That's all I can remember w/o reviewing the code.
Reply With Quote