Stats Terminology: Hypothesis Testing and Significance vs. Confidence

[uDevil]

If I can reject the hypothesis that I am a losing player at the 10% level of significance, is it correct to say "I have 90% confidence that I a winning player"? Or am I mixing up different concepts?

As an example, a player who has broken even over a sample of hands might say he has 50% confidence that he is a winner. Makes sense to me. Is it right?

[DrVanNostrin]

In a hypothsis test there are two hypothises (sp?):
Null hypothosis (H0): my true win rate is zero
Alternative hypothsis (HA): my true win rate is not zero (this is for a two sided test; HA for a one sided test would be my true win rate is greater than 0 or my true win rate is less than 0)


Think a hypothisis test as a criminal trial and think of H0 as innocent and HA as guilty. Defendants will not be found innocent--they'll be found either guilty or not guilty. Note that not guilty does not mean they're innocent; it only means there is not enough evidence to show they are guilty. Similarly, the result of hypothosis test will be either reject H0 or fail to reject H0. Failing to reject H0 is not saying that H0 is correct; it is saying only that there is not enough evidence to reject it.

There are two sided tests and one sided tests. A one sided test would be appropiate is you were confident that one thing could not be better than another. For example, if you compared amatures' results to professionals' results you could use a one sided test.
[ QUOTE ]
If I can reject the hypothesis that I am a losing player at the 10% level of significance, is it correct to say "I have 90% confidence that I a winning player"? Or am I mixing up different concepts?

[/ QUOTE ]
If you used a two sided test and got a p-value of .1 (or rejected H0 on the 10% level) there's only a 5% chance you're a losing player. There's also a 5% chance you're true win rate is more than the high end of your inteval. So you could actually say there's a 95% chance you're winnning player.

[uDevil]

I'll try to be more specific:

Ho= I am a losing player, winrate<0
H1= I am a winning player, winrate>=0

The null hypothesis will be rejected when winrate is significantly larger than 0 so I do a one-tailed (right-tailed) hypothesis test. I select a 10% level of significance and find the critical value of z such that 10% of the area under the normal curve is to the right and 90% of the area is to the left. I find z=1.28:



If I calculate the test value for a winrate of 2 BB/100, standard deviation of 16 BB/100, and 20,000 hands,

z= (winrate-mean)/(stdDev/sqrt(n))= (2-0)/(16/sqrt(200))= 1.78

The test value is greater than the critical value, so Ho is rejected at the 10% level of significance. Is this the same as saying that I have greater than 90% confidence that I am a winner?

I'd just like to be sure I'm using the word 'confidence' correctly.

I wrote a javascript program to find the minimum level of significance (=highest confidence value?) at which Ho can be rejected for a given winrate, sd, and number of hands:

http://www.castrovalva.com/~la/winlose.htm

[JennEric Screen Name]

[ QUOTE ]
If I can reject the hypothesis that I am a losing player at the 10% level of significance, is it correct to say "I have 90% confidence that I a winning player"?

[/ QUOTE ]

Yep.

[uDevil]

[ QUOTE ]
[ QUOTE ]
If I can reject the hypothesis that I am a losing player at the 10% level of significance, is it correct to say "I have 90% confidence that I a winning player"?

[/ QUOTE ]

Yep.

[/ QUOTE ]

Ok. Thanks guys.

[jogsxyz]

Poker tracker and other utilities use the old B&M method for calculating standard deviation. They calculate by session data. Only possible way to make these cals in the B&M days. Now they should be using hand-by-hand cals. The old methods understates s.d. by 10 to 15%. Your true s.d. is probably 18-19 bb/100.

[uDevil]

[ QUOTE ]
Poker tracker and other utilities use the old B&M method for calculating standard deviation. They calculate by session data. Only possible way to make these cals in the B&M days. Now they should be using hand-by-hand cals. The old methods understates s.d. by 10 to 15%. Your true s.d. is probably 18-19 bb/100.

[/ QUOTE ]

Shouldn't the two methods eventually converge to the same value? Is there still a 10%-15 difference after say 20,000 hands (maybe ~100 PokerTracker sessions)?

[AaronBrown]

Not quite. Neither statement is precise, although in common usage they mean the same thing. Things get trickier when the outcomes are more complicated.

In a hypothesis test, you will test that you are a break-even player. That is, you will compute the probability that a break-even player will win as much or more money as you did, given your standard deviation and possibly some other data. If that probability is less than 10%, you can reject the hypothesis that you are a break-even player at the 10% level. Since you win too consistently to be break-even, you're even less likely to be a losing player.

Instead of doing a hypothesis test, you could put a confidence interval on your win rate. It might be 0.3 BB/hour to 1.6 BB/hour. If you set a symmetric two-sided 80% confidence interval, we know that it does not include zero (if it did, you couldn't have rejected the hypothesis that you are a break-even player at the 10% level).

In English, you could say, "I'm 90% confident that I'm a winning player," but that has no precise statistical definition.

Also the calculation is only as good as the assumptions underneath it. If your win rate is not constant, or if hands are not indpendent, it may be seriously misleading.

[BruceZ]

[ QUOTE ]
In English, you could say, "I'm 90% confident that I'm a winning player," but that has no precise statistical definition.

[/ QUOTE ]

We can certainly say that his win rate is greater than zero with a confidence greater than 90%, and this is well-defined statistically to mean just what he has described. See DeGroot p. 399.