Price Comparison Website Questions

[Neko]

[ QUOTE ]
How are you going to generate revenues? Will you act as an affiliate for the sites you're comparing prices from?

[/ QUOTE ]
Yes...sorry I didn't make that clear in my o.p.

[elus2]

Some companies have xml feeds of their product and price list. Best thing to do is to email them once you've setup affiliate accounts.

[BradleyT]

Most of the large book sites have feeds that you can grab into Amazon Web Services. So it does a web service lookup each time a price is requested.

Sites like Shopping.com have an internal database where the clients input all the data.

Not sure how other sites do it. You could do it using scraping technology but that would require constant updates. For example you request site.com?productID=23 and parse through the entire html text for "Our Price: $" and then take all the text that appears after that string. You'd have to write a seperate routine for each site you wanted and any time they changed thier text that your scraping you'd need to update your routine.

[Evan]

[ QUOTE ]
Thanks for your help. Yeah, I would be generating revenue only through affiliate programs, and maybe adsense and not handling any of the actual transactions myself.

I (perhaps naively) thought that my users could enter the product/keywords into my site and I could somehow do a realtime search of the major sites in my niche and return the price from each of them. The user could then click on the link (including my affiliate info) wherever the cheapest price was and make the purchase there, hence generating me affiliate revenue.

If you look at the site anybook4less.com and search for "theory of poker" it returns all the prices for the theory of poker from amazon, barnes and noble, books a million etc. Are you telling me that all of that data is stored locally in a database on the anybook4less.com server?

If so, you're right that is a much bigger task than I had thought.

Thanks again for your help.

[/ QUOTE ]
I have never used this book site but most search services don't do true live search because it's incredibly inefficient.

[Neko]

[ QUOTE ]
You could do it using scraping technology but that would require constant updates. For example you request site.com?productID=23 and parse through the entire html text for "Our Price: $" and then take all the text that appears after that string. You'd have to write a seperate routine for each site you wanted and any time they changed thier text that your scraping you'd need to update your routine.

[/ QUOTE ]

I think that would be very computationally intensive given the huge number of items available. Hopefully many of the sites will have xml feeds I can use.

Thank you both very much for your help. I'll start doing research on this by contacting the sites I have in mind. If anybody else has any experience in this area I'd love to hear about it.

[TLC]

You could write a multi-threaded screen scraper which would glean the pricing/availability info from the vendor sites in realtime.

This is pretty easy if there is a common "item number" like an ISBN for books. If there is not, then you will need to set up a process to "crawl" a vendor's site daily to store a mapping between your item # and each vendor's item #.

I did something this about 7 years ago using ASP and MTS. It typically went to a vendor's product page using the Item # and parsed the HTML using a vendor-specific RegEx expression. I did this for multiple vendors concurrently and had a timeout to "give up" on a vendor if that request took too long.

That's all I can remember w/o reviewing the code.

[Evan]

[ QUOTE ]
You could write a multi-threaded screen scraper which would glean the pricing/availability info from the vendor sites in realtime.

This is pretty easy if there is a common "item number" like an ISBN for books. If there is not, then you will need to set up a process to "crawl" a vendor's site daily to store a mapping between your item # and each vendor's item #.

I did something this about 7 years ago using ASP and MTS. It typically went to a vendor's product page using the Item # and parsed the HTML using a vendor-specific RegEx expression. I did this for multiple vendors concurrently and had a timeout to "give up" on a vendor if that request took too long.

That's all I can remember w/o reviewing the code.

[/ QUOTE ]
I really can't imagine this being necessary. Unless his niche market is some kind of super efficient financial instrument (stocks, for example) there's no point to dedicating resources to real time search. Take books, for instance. How often do their prices change? Maybe once every two weeks? Probably a hell of a lot less than that. Why the hell would you design a system to do real time searches rather that just having a daily (or hourly or whatever) crawler to update prices in your local database?

[TLC]

Pricing (and availabilty) are more dynamic than you may think. Some vendors use competitor pricing and availability to determine their own pricing.

If his product universe is large, it may take a lot of time and bandwidth to crawl and scrape all that info (assuming there isn't a XML datasource available to him). So, depending on the product universe size and the anticipated queries per day, it may be more efficient to query the vendor sites in realtime.

One additional advantage of the realtime query is that the customer is less likely to be upset when he clicks through to the vendor's site and feels misled by his site because of stale data.

[Neko]

[ QUOTE ]
I really can't imagine this being necessary. Unless his niche market is some kind of super efficient financial instrument (stocks, for example) there's no point to dedicating resources to real time search. Take books, for instance. How often do their prices change? Maybe once every two weeks? Probably a hell of a lot less than that. Why the hell would you design a system to do real time searches rather that just having a daily (or hourly or whatever) crawler to update prices in your local database?

[/ QUOTE ]

This makes good sense. The majority of the products in this area will have fairly static prices.

[Evan]

[ QUOTE ]

If his product universe is large, it may take a lot of time and bandwidth to crawl and scrape all that info (assuming there isn't a XML datasource available to him). So, depending on the product universe size and the anticipated queries per day, it may be more efficient to query the vendor sites in realtime.

[/ QUOTE ]
How could doing it in real time possibly be less resource intensive? I guess if your set of data to search is huge and the average search is tiny, but I still think that if you set up a proper dedicated crawler you'd make things a lot easier on yourself.