weighted coins question

[ig06]

I have a question which I will express in terms of weighted coins:

Alice has a huge pile of coins, each with different weightings on them.
She has some idea of the weighting on each coin but her estimates may be anywhere from exact to only very rough estimates. Alice now gives these coins to Bob, telling him her estimate of the weighting on each coin. Bob is allowed to toss each coin once and only once and then has to decide how much trust he places in Alice's estimates.

Bob could do the following:

1. Where P(heads_i) is the probability of heads for the i_th coin, Bob computes the sum of these over all coins. This is the mean number of expected heads IF Alice's estimates are assumed to be exact. Call this MEAN(Heads)
2. Compute the sum over all i of P(heads_i)*(1-P(heads_i) then take the square root of this. This is the standard deviation in the number of expected heads IF Alice's estimates are assumed to be exact. Call this STDEV(Heads)
3. Now let the number of heads observed when Bob tossed the coins once each be H. Compute (MEAN(Heads)-H)/STDEV(Heads).
This is a measure of how many standard deviations we are away from the mean number of expected heads if Alice's estimates are accurate. If the magnitude of this number is large then it is unlikely that Alice's initial estimates were particularly precise.

However, it is not clear to me how best to interpret this value more formally. Also, how does the total number of coins affect the usefulness of this measure and can I quantify this formally? Finally, is there a better approach to this problem?
The actual problem I am interested in is where Party A and Party B are agreeing contracts and for each contract Party A provides a probability that it fails to fulfill its obligations to Party B. However Party B would like to evaluate the precision of Party A's estimates over time (since Party B could be poor at estimating or deliberately lying). Any ideas?
Cheers

[AaronBrown]

This is not a good test. The problem is it concentrates on the average P(i)'s, not the accuracy of P(i)'s. For example, suppose Alice said the first N coins had P(i) = 0 and the next N had P(i) = 1.

Bob finds out Alice is exactly wrong. The first N all come up heads and the second N all come up tails. Yet Mean(Heads) = H exactly. In this case the standard deviation is zero, so we can't compute your statistic, but the same problem would occur for any set of P(i)'s if Alice had the average about right but was wrong on individual coins.

A better test statistic is the amount of money you would make betting on Alice's advice. Suppose Bob mentally bets P(i) - 0.5 on heads for each coin. So if P(i) = 0.75 he bets $0.25 on heads; if P(i) = 0.3 he bets -$0.20 on heads, which means he bets $0.20 on tails.

If Alice's predictions are exact, Bob's expected profit is the sum of 2*[P(i) - 0.5]^2 with a variance of the sum of P(i)*[1 - P(i)]*[P(i) - 0.5]^2. If the actual profit is above zero by, say, two standard deviations or more, Alice's predictions seem to have some value.

Depending on the precise application, you might want to vary this method. For example, this assumes in the absence of information you would guess P(i) = 0.5. If that's not true, for example, if P(i) = 0.01 is more typical, you could test against that, or against all P(i) equal the average P(i).

Also, this makes the assumption that errors in P(i) hurt you according to their square. That is, twice as big an error hurts four times as much. That might not be appropriate as well.

In general, rather than thinking about how accurate Alice is, think about how valuable her information is. That's what you really care about. Make your test statistic as closely as possible the result of relying upon her information. Then it will tell you most precisely what you want to know.

[ig06]

Thanks for the reply. I had appreciated this problem with my proposed method. Indeed if Alice knows the average weighting she can just tell Bob that is the weighting for all the coins. This spells disaster because the variance is large and the number of heads observed will usually be close to the predicted number.In these contracts there would be a number of other attributes enabling this measure to be computed for different subsets of the data and this was my initial idea to address this problem. However this runs into problems:
- the subsets would have much less data in them
- its not clear how to choose these subsets if there are many parameters.
-I'm sure there are other problems.

I think your betting algorithm is very clever and I will have a think about this. Thanks again