weighted coins question

[ig06]

I have a question which I will express in terms of weighted coins:

Alice has a huge pile of coins, each with different weightings on them.
She has some idea of the weighting on each coin but her estimates may be anywhere from exact to only very rough estimates. Alice now gives these coins to Bob, telling him her estimate of the weighting on each coin. Bob is allowed to toss each coin once and only once and then has to decide how much trust he places in Alice's estimates.

Bob could do the following:

1. Where P(heads_i) is the probability of heads for the i_th coin, Bob computes the sum of these over all coins. This is the mean number of expected heads IF Alice's estimates are assumed to be exact. Call this MEAN(Heads)
2. Compute the sum over all i of P(heads_i)*(1-P(heads_i) then take the square root of this. This is the standard deviation in the number of expected heads IF Alice's estimates are assumed to be exact. Call this STDEV(Heads)
3. Now let the number of heads observed when Bob tossed the coins once each be H. Compute (MEAN(Heads)-H)/STDEV(Heads).
This is a measure of how many standard deviations we are away from the mean number of expected heads if Alice's estimates are accurate. If the magnitude of this number is large then it is unlikely that Alice's initial estimates were particularly precise.

However, it is not clear to me how best to interpret this value more formally. Also, how does the total number of coins affect the usefulness of this measure and can I quantify this formally? Finally, is there a better approach to this problem?
The actual problem I am interested in is where Party A and Party B are agreeing contracts and for each contract Party A provides a probability that it fails to fulfill its obligations to Party B. However Party B would like to evaluate the precision of Party A's estimates over time (since Party B could be poor at estimating or deliberately lying). Any ideas?
Cheers