How many hands is enough?

[brianmarc]

Many posters claim that you need huge numbers of hands in order to identify valid characteristics of a player’s record. Numbers like 50,000, 100,000 are bandied about, as if there is no problem in waiting long enough to collect this data. Even more problematic is analyzing opponent play from systems like Poker Tracker where it is almost impossible to collect data of this magnitude.

To date, I have not been able to find any rigorous analysis of this. But I do know of standard statistical tests that can be used to identify the confidence level at which a sample mean is meaningful: all one needs is the Z-score and a normal-distribution table.

Consider my personal data from 12/1/05 when I drastically changed my SH (6 max) playing style: No. Hands : 8,792
Win Rate/100 hands: $27.67
Std. Dev./100 hands: $110.5
Std. Error: 11.8
Z-Score: 2.3
Confidence Level: >98%

Do I really need to play another 40,000+ hands to satisfy myself that my new style is a winning style?

[beyeond]

[ QUOTE ]
Many posters claim that you need huge numbers of hands in order to identify valid characteristics of a player’s record. Numbers like 50,000, 100,000 are bandied about, as if there is no problem in waiting long enough to collect this data. Even more problematic is analyzing opponent play from systems like Poker Tracker where it is almost impossible to collect data of this magnitude.

To date, I have not been able to find any rigorous analysis of this. But I do know of standard statistical tests that can be used to identify the confidence level at which a sample mean is meaningful: all one needs is the Z-score and a normal-distribution table.

Consider my personal data from 12/1/05 when I drastically changed my SH (6 max) playing style: No. Hands : 8,792
Win Rate/100 hands: $27.67
Std. Dev./100 hands: $110.5
Std. Error: 11.8
Z-Score: 2.3
Confidence Level: >98%

Do I really need to play another 40,000+ hands to satisfy myself that my new style is a winning style?

[/ QUOTE ]

I hope you need more, because i'm down about 240bbs over my last 9K hands /images/graemlins/frown.gif

[brianmarc]

Ouch!

But if you run the hands through PT and perform the analysis described under Win Rate and Standard Deviation, you might be able to know for sure if you're a definite loser

http://forumserver.twoplustwo.com/showfl...&PHPSESSID= (http://forumserver.twoplustwo.com/showflat.php?Cat=0&Board=micro&Number=2913856&fpar t=1&PHPSESSID=)

[timex]

[ QUOTE ]

Do I really need to play another 40,000+ hands to satisfy myself that my new style is a winning style?

[/ QUOTE ]

This is very opinionated, some people may think 60% confidence is enough, others may require 99.999+, its hard to answer your question of how to satisfy yourself, that question is most easily answered by you.

[brianmarc]

Of course, the required confidence level is subjective. However, when it is well above 95% that question ceases to be relevant. A much more important one is, Why all this focus on raw # of hands, when, in fact, there are powerful statistical tools to tell you when enough is enough?

One posible answer is that my assumptions and methodology are incorrect. There may be others.

[beyeond]

[ QUOTE ]
Ouch!

But if you run the hands through PT and perform the analysis described under Win Rate and Standard Deviation, you might be able to know for sure if you're a definite loser

http://forumserver.twoplustwo.com/showfl...&PHPSESSID= (http://forumserver.twoplustwo.com/showflat.php?Cat=0&Board=micro&Number=2913856&fpar t=1&PHPSESSID=)

[/ QUOTE ]

Only down over my last 9k, still have a positive winrate though...

[SumZero]

I don't think you can do the simple test you suggest as the population of players is drawn from un unknown distribution with mean -rake. I.e., the average player loses, and likely the vast majority of players lose, etc. Therefore it is more likely you are a losing player running good than an even better than your current ROI running bad when you have a positive ROI. If you knew what the distribution of true talent was across all players then you could do a maximum likelihood estimate of your true ROI and you'd find it is lower than what your ROI predicts and may negate your z-score analysis.

[brianmarc]

A very thoughtful and interesting answer.

However, if the test is invalid for the reasons you suggest, does this mean it is incorrect in the sense that there is no information, or is it a question of degree? i.e., if my confidence level is 98%, but the test is flawed, do I have more information than if my CL was, say 65%?

Also, what represents a valid test, since the hypothetical you propose is impractical?

[hellite]

I suggest you take a look at Mason Malmuth's "Gambling Theory and Other Topics" if you are concerned about identiftying your true win rate. I think you would be very surprised. Its fairly complicated but you probably need around 100,000 hands minimal to figure out your winrate depending on your standard deviation.

[pzhon]

[ QUOTE ]
To date, I have not been able to find any rigorous analysis of this.

[/ QUOTE ]
Real explanations tend to get drowned out by people repeating numbers they don't understand, or making up higher numbers in an attempt to sound even wiser than the last person who gave a large number.

[ QUOTE ]
But I do know of standard statistical tests that can be used to identify the confidence level at which a sample mean is meaningful: all one needs is the Z-score and a normal-distribution table.
...
Do I really need to play another 40,000+ hands to satisfy myself that my new style is a winning style?

[/ QUOTE ]
No, you don't need to play another 40k hands.

Your methodology is good for answering the question of whether your style is a winning one, though not perfect. The main problem is that you didn't set a stopping criterion ahead of time. It's much more likely to be ahead by 2 standard deviations at some point than after a prespecified number of hands. The result is that you don't have the same level of confidence that you would have had if you had decided to play 8792 hands ahead of time. Exactly what level of confidence you have can only be determined with some soul-searching to figure out what your stopping condition was.

You would need a larger number of hands to estimate your win rate accurately, but that wasn't your question, so people should stop telling you that 100k hands is the answer.

[pzhon]

[ QUOTE ]
[ QUOTE ]
No. Hands : 8,792
Win Rate/100 hands: $27.67
Std. Dev./100 hands: $110.5
Std. Error: 11.8
Z-Score: 2.3
Confidence Level: >98%



[/ QUOTE ]

I hope you need more, because i'm down about 240bbs over my last 9K hands

[/ QUOTE ]
Non sequitur. You lost 2.7 bb/100, much lower than his win rate. Your 95% confidence interval extends well beyond 0. His doesn't include 0.

[brianmarc]

Interesting perspective that establishing precisely the question that needs to be answered could impact the analysis. (The statistical analog of the Heisenberg uncertainty principle maybe?!)

So let's now consider some other questions:
"What is my actual win rate at the 95% confidence level?"

"What criteria should I use to identify the players that I would want to play against, as well as those whom I would want to avoid?"

How might you look at these results differently from when the question is "Am I a winning player?"

[brianmarc]

Thanks for the suggestion. I ahve the book, and had not thought to review it. His section Win Rate Accuracy is right on point.

[DrVanNostrin]

Your test is valid if you wish to only determine whether or not you're a winning player. Your test is not valid is you're trying to determine your actual win rate.

This tells you your win rate is $27.67/100 +/- $27.67/100 with a confidence of 98%. Use the same z-score table and determine with what confidence your win rate was $27.67/100 +/- $2.76/100.

That should explain why so many hands are necessary to determine one's true win rate. However you're absolutely correct that fewer hands are necessary to determine whether or not a player is a winner.

[brianmarc]

Yes; I now have the methodologies to answer both questions. In the end, the exact win rate will be what it will be - i.e. since so many hands are needed to know it with high certainty, my bankroll will have told me long before I get there is I am on the right track.