Need to calculate confidence for an A/B test with excel
 

For each of the two test cases in an A/B test, I know the number of visitors to the page and the number of those visitors which took a specific action.

Is there an easy way in Excel to calculate how much confidence I have in the results?

Re: Need to calculate confidence for an A/B test with excel
 

What you're trying to do is estimate, with a certain degree of confidence, how close your numbers are to the TRUE mean. You need an estimate of the population mean. Do you have such a figure?

If so then it is easily done in excel and even by hand for that matter.

Re: Need to calculate confidence for an A/B test with excel
 

The only numbers I have are:

Test A: total number of visitors and the percentage that took the desired action.
Test B: total number of visitors and the percentage that took the desired action.

What I'd like to end up with is a result that says "given the sample size of the test, you can be XX% confident that future results will be within YY% of the current results regardless of how much larger your sample size gets."

Re: Need to calculate confidence for an A/B test with excel
 

A few questions before I venture an answer.

How many records of records do you have for both A and B?

How was the data collected and because of that method do you have any reason to believe that the sequence the records were collected in has any effect on whether or not the person "took the desired action"? Stated another way, are records collected at the beginning just as likely to have taken the desired action as those recorded last?

How large of a difference will subsequent there be between this sample size and subsequent samples you want to compare these results to?

Given this information I can put together a spreadsheet with some formulas for you.