Price Comparison Website Questions
 

I have an idea for a price comparison website (pcw) that would focus on a niche area that I don't believe currently has a good pcw. There is a lot of money being spent in this niche and prices often vary wildly from one online vendor to the next so I believe the potential to create a useful, profitable (from affiliate programs and maybe adsense) website is fairly good.

(For an example of the type of site I'd be interested in creating see http://www.addall.com/ a book price comparison engine.)

Couple questions:

1. Does anybody have any personal experience with price comparison sites? If you do, do you mind sharing your experience a bit?

2. Any links to where I can learn about setting up a website like this?

3. What are peoples opinions on the profitability of this type of website in general?

4. I would like to build the site myself mostly for the learning experience. My goal would be to spend anywhere from 1 to 15 hours a week on the site and have it up and running by the end of the year. I am a competent self taught programmer and have experience in a number of different languages but only a small amount of html/javascript/php experience (since I work as a computational scientist most of my experience is in C/C++/Fortran). With my goal in mind would this be foolish? Would I be much better off hiring someone to build the site myself?

Thanks for any info you can provide me!

Re: Price Comparison Website Questions
 

I can't be of much help, but I solely use www.pricegrabber.com and it's probably the #1 I hear people speak of.

Re: Price Comparison Website Questions
 

[ QUOTE ]
I can't be of much help, but I solely use www.pricegrabber.com and it's probably the #1 I hear people speak of.

[/ QUOTE ]

Pricegrabber is one of the biggest but it is not targeted at one segment. My site would be targeted at a niche market that places like pricegrabber/froogle etc. don't do a great job of covering.

Re: Price Comparison Website Questions
 

It would be somewhat difficult without 2-4 years (full-time) experience in a web programming language. You're also going to need a good amount of SQL experience.

Re: Price Comparison Website Questions
 

[ QUOTE ]
It would be somewhat difficult without 2-4 years (full-time) experience in a web programming language. You're also going to need a good amount of SQL experience.

[/ QUOTE ]

Well that's certainly discouraging! My understanding was that learning something like PHP/MySQL was relatively easy if you're already coming from a C/C++/Linux background. Do you mind elaborating a bit? I'm curious why you believe I would need so much experience before embarking on this development.

Thanks,
Neko

Re: Price Comparison Website Questions
 

Doing javascript, html, and css is incredibly frustrating. You got cross browser issues, different versions of different browsers, I.E. bugs, and etc. I always feel like I'm wasting my precious time when I end up spending hours to get something to look exactly right in 7 different browsers.

Re: Price Comparison Website Questions
 

I'm not saying you can't do it, just that without much web experience you're going to have a long road ahead of you.

How are you getting data into your database?
How are you handling payments?
How are you getting traffic to your website?
How are you handling client reporting?

Re: Price Comparison Website Questions
 

Well if you're doing adsense or ads you can skip #2 and #4 but #1 is still huge. You'll want to accept standard file types (xml,csv,txt) and then you have the fun task of parsing each of those files to get the data inserted properly into your database. And you're going to have to emulate the same input format as the large sites so that people don't have to make a new XML file to submit data to your site.

At my last job I took our custom e-commerce database data and created XML files to feed froogle and shopping.com. The rest of the PPC shopping sites were out of luck as it was way too much work converting our data into their format and the traffic from those sites suck compared to google adwords.

Re: Price Comparison Website Questions
 

How are you going to generate revenues? Will you act as an affiliate for the sites you're comparing prices from?

Re: Price Comparison Website Questions
 

Thanks for your help. Yeah, I would be generating revenue only through affiliate programs, and maybe adsense and not handling any of the actual transactions myself.

I (perhaps naively) thought that my users could enter the product/keywords into my site and I could somehow do a realtime search of the major sites in my niche and return the price from each of them. The user could then click on the link (including my affiliate info) wherever the cheapest price was and make the purchase there, hence generating me affiliate revenue.

If you look at the site anybook4less.com and search for "theory of poker" it returns all the prices for the theory of poker from amazon, barnes and noble, books a million etc. Are you telling me that all of that data is stored locally in a database on the anybook4less.com server?

If so, you're right that is a much bigger task than I had thought.

Thanks again for your help.

Re: Price Comparison Website Questions
 

[ QUOTE ]
How are you going to generate revenues? Will you act as an affiliate for the sites you're comparing prices from?

[/ QUOTE ]
Yes...sorry I didn't make that clear in my o.p.

Re: Price Comparison Website Questions
 

Some companies have xml feeds of their product and price list. Best thing to do is to email them once you've setup affiliate accounts.

Re: Price Comparison Website Questions
 

Most of the large book sites have feeds that you can grab into Amazon Web Services. So it does a web service lookup each time a price is requested.

Sites like Shopping.com have an internal database where the clients input all the data.

Not sure how other sites do it. You could do it using scraping technology but that would require constant updates. For example you request site.com?productID=23 and parse through the entire html text for "Our Price: $" and then take all the text that appears after that string. You'd have to write a seperate routine for each site you wanted and any time they changed thier text that your scraping you'd need to update your routine.

Re: Price Comparison Website Questions
 

[ QUOTE ]
Thanks for your help. Yeah, I would be generating revenue only through affiliate programs, and maybe adsense and not handling any of the actual transactions myself.

I (perhaps naively) thought that my users could enter the product/keywords into my site and I could somehow do a realtime search of the major sites in my niche and return the price from each of them. The user could then click on the link (including my affiliate info) wherever the cheapest price was and make the purchase there, hence generating me affiliate revenue.

If you look at the site anybook4less.com and search for "theory of poker" it returns all the prices for the theory of poker from amazon, barnes and noble, books a million etc. Are you telling me that all of that data is stored locally in a database on the anybook4less.com server?

If so, you're right that is a much bigger task than I had thought.

Thanks again for your help.

[/ QUOTE ]
I have never used this book site but most search services don't do true live search because it's incredibly inefficient.

Re: Price Comparison Website Questions
 

[ QUOTE ]
You could do it using scraping technology but that would require constant updates. For example you request site.com?productID=23 and parse through the entire html text for "Our Price: $" and then take all the text that appears after that string. You'd have to write a seperate routine for each site you wanted and any time they changed thier text that your scraping you'd need to update your routine.

[/ QUOTE ]

I think that would be very computationally intensive given the huge number of items available. Hopefully many of the sites will have xml feeds I can use.

Thank you both very much for your help. I'll start doing research on this by contacting the sites I have in mind. If anybody else has any experience in this area I'd love to hear about it.

Re: Price Comparison Website Questions
 

You could write a multi-threaded screen scraper which would glean the pricing/availability info from the vendor sites in realtime.

This is pretty easy if there is a common "item number" like an ISBN for books. If there is not, then you will need to set up a process to "crawl" a vendor's site daily to store a mapping between your item # and each vendor's item #.

I did something this about 7 years ago using ASP and MTS. It typically went to a vendor's product page using the Item # and parsed the HTML using a vendor-specific RegEx expression. I did this for multiple vendors concurrently and had a timeout to "give up" on a vendor if that request took too long.

That's all I can remember w/o reviewing the code.

Re: Price Comparison Website Questions
 

[ QUOTE ]
You could write a multi-threaded screen scraper which would glean the pricing/availability info from the vendor sites in realtime.

This is pretty easy if there is a common "item number" like an ISBN for books. If there is not, then you will need to set up a process to "crawl" a vendor's site daily to store a mapping between your item # and each vendor's item #.

I did something this about 7 years ago using ASP and MTS. It typically went to a vendor's product page using the Item # and parsed the HTML using a vendor-specific RegEx expression. I did this for multiple vendors concurrently and had a timeout to "give up" on a vendor if that request took too long.

That's all I can remember w/o reviewing the code.

[/ QUOTE ]
I really can't imagine this being necessary. Unless his niche market is some kind of super efficient financial instrument (stocks, for example) there's no point to dedicating resources to real time search. Take books, for instance. How often do their prices change? Maybe once every two weeks? Probably a hell of a lot less than that. Why the hell would you design a system to do real time searches rather that just having a daily (or hourly or whatever) crawler to update prices in your local database?

Re: Price Comparison Website Questions
 

Pricing (and availabilty) are more dynamic than you may think. Some vendors use competitor pricing and availability to determine their own pricing.

If his product universe is large, it may take a lot of time and bandwidth to crawl and scrape all that info (assuming there isn't a XML datasource available to him). So, depending on the product universe size and the anticipated queries per day, it may be more efficient to query the vendor sites in realtime.

One additional advantage of the realtime query is that the customer is less likely to be upset when he clicks through to the vendor's site and feels misled by his site because of stale data.

Re: Price Comparison Website Questions
 

[ QUOTE ]
I really can't imagine this being necessary. Unless his niche market is some kind of super efficient financial instrument (stocks, for example) there's no point to dedicating resources to real time search. Take books, for instance. How often do their prices change? Maybe once every two weeks? Probably a hell of a lot less than that. Why the hell would you design a system to do real time searches rather that just having a daily (or hourly or whatever) crawler to update prices in your local database?

[/ QUOTE ]

This makes good sense. The majority of the products in this area will have fairly static prices.

Re: Price Comparison Website Questions
 

[ QUOTE ]

If his product universe is large, it may take a lot of time and bandwidth to crawl and scrape all that info (assuming there isn't a XML datasource available to him). So, depending on the product universe size and the anticipated queries per day, it may be more efficient to query the vendor sites in realtime.

[/ QUOTE ]
How could doing it in real time possibly be less resource intensive? I guess if your set of data to search is huge and the average search is tiny, but I still think that if you set up a proper dedicated crawler you'd make things a lot easier on yourself.