Latest cliffsnotes on Absolute soulreading.

[GaryTheGoat]

[ QUOTE ]
i want to create a graph showing how out of whack the cheaters are.

to do this, i want to run a multi-million hand database through hold'em manager (since it handles big DBs faster than PT) and graph the output (i'll just use excel 'cause i have it accessible).


if anyone has some NL hand histories - of any stake, any site - can you please zip them up and email them to me at michael@michaeljosem.com? they can be old, and no data will be individually identifiable anyway.


[/ QUOTE ]

Josem:

Maybe you can get in touch with Pokey here at 2p2. In May of 2006 he prepared Summary of the Vol;untary Statistics Survey

He may have 500,000+ nl hand histories full ring and 6 max
that you can "borrow".

gg

[flight2q]

[ QUOTE ]
What is a priori distribution?

[/ QUOTE ]An a priori distribution is an assumed or known probability distribution for the object of study in advance of looking at the data.

For example, tonight I hear a wolf howling. About 10% of nights this is so. I don't say, okay I'm 90% convinced that there's a full moon tonight (even without a model of the relationship between howling and full moons!).

But suppose I knew that wolf always howls when there is a full moon, and sometimes at other times. But I still don't know how likely it is that there is a full moon tonight. Now I add an a priori distribution that moon is full 1/28th of the time. Now I can deduce a posteriori that there is a 35.7% probability that the moon is full based on my knowledge of wolves, moons, hearing the howling, and assuming I have no other information such as whether I heard howling last night.

[Josem]

So what would a priori distribution be for a heads up game vs one of the alleged cheaters?

[flight2q]

Say a few weeks ago you sat down at a table with Graycat HU, and after the session you asked yourself did that guy see my holecards. The a priori distribution is whatever probability you thought he had of seeing your hole cards before you sat down with him. 1 in a million? 1 in a billion? or whatever.

Then you analyze the session to adjust that a priori probability based on the data. But in doing so, you need to realize that your data is biased, because you don't do this analysis after every session. But if you sit at a table with Graycat right now and play until he quits you (or play until a prescribed number of hands, and throw out the sample if he quits you earlier), and analyze based on those results, good or bad, then you won't have selection bias.

[Josem]

[ QUOTE ]
Say a few weeks ago you sat down at a table with Graycat HU, and after the session you asked yourself did that guy see my holecards. The a priori distribution is whatever probability you thought he had of seeing your hole cards before you sat down with him. 1 in a million? 1 in a billion? or whatever.

[/ QUOTE ]

I don't understand; this does not make sense to me. Why does have a pre-existing estimate of the chance you were cheated matter?

[ QUOTE ]
Then you analyze the session to adjust that a priori probability based on the data. But in doing so, you need to realize that your data is biased, because you don't do this analysis after every session. But if you sit at a table with Graycat right now and play until he quits you (or play until a prescribed number of hands, and throw out the sample if he quits you earlier), and analyze based on those results, good or bad, then you won't have selection bias.

[/ QUOTE ]

My criteria to minimise selection bias will be every accessible heads up hand history vs him. While there will be "selection bias" in favour of hands played with people who browse 2p2 and similar sites (and thus, able to provide the data), I don't see how such a bias is likely to distort the outcome.

We're studying a phenomenon that has already happened. Judging by the crazy chip dumping after this thread became public, any future play will be contaminated by this very study/thread/publicity.

[apefish]

Are historical data sets from other players playing a similar style to the suspected sessions a somewhat valid substitution for previous sessions against the same user in this instance?

I realize it isn't perfect and I am not asking that.
What I am saying is that instead of having to play against Graycat repeatedly, isn't historical data over a large set of samples a reasonable substitute for comparison and analyzing?

And isn't that precisely what we are suggesting- that this falls so far out of line it sways heavily one way?

It isn't perfect- but I think there are some valid comparisons to be made that way.


edit: heh, josem wrote something similar while I was writing this out.

[flight2q]

@Josem: Why do we need an a priori distribution to state we conclude there is a x% probability of cheating? Let's go back to the howling moon. Everything is the same, wolves always howl at full moons, and overall wolves howl 10% of nights. We hear howling and ask whether there is a full moon. We also ask whether there is a full moon with a halo. Full moons happen 1 of 28 nights, and full moons with halos happen 1 of 2000 nights. The data is the same and we calculate the wolf howling event at 0.10, and by Josem method conclude that it's 90% chance of full moon. The same applies to full moons with halos too, 90% chance. Obviously full moons with halos are rarer though. That's why we use the 1/28 and 1/2000 a priori knowledge to conclude that there is 35.7% chance there is a full moon and 0.5% chance there is a full moon with a halo.

@apefish: It's possible to find a use for historical data sets of players with similar styles. For example, we could look at all hands from other 80/60 players that are headsup on the flop. Then we could ask questions like, if they were in the lead and checked to on the turn, how often did they bet between half pot and full pot and how often did their opponent fold/call/raise, and what were results when opponent did not fold. If we hypothesize that suspect player doesn't try to get opponents out on flop, but does try to fold them out or extract on turn, then we can make a test to compare how this player is doing at this compared to the benchmark who are likely to have similar hand quality on the turn. The tricky part is figuring out how to analyze a test like this. It hurts that (based on what I saw from a limited sample of Doubledrag hands) any exploitation is likely mostly occurring on turn/river, so it is harder to gauge what hand quality suspect player and opponents have at that point.

[Josem]

[ QUOTE ]
@Josem: Why do we need an a priori distribution to state we conclude there is a x% probability of cheating? Let's go back to the howling moon. Everything is the same, wolves always howl at full moons, and overall wolves howl 10% of nights. We hear howling and ask whether there is a full moon. We also ask whether there is a full moon with a halo. Full moons happen 1 of 28 nights, and full moons with halos happen 1 of 2000 nights. The data is the same and we calculate the wolf howling event at 0.10, and by Josem method conclude that it's 90% chance of full moon. The same applies to full moons with halos too, 90% chance. Obviously full moons with halos are rarer though. That's why we use the 1/28 and 1/2000 a priori knowledge to conclude that there is 35.7% chance there is a full moon and 0.5% chance there is a full moon with a halo.

[/ QUOTE ]

It seems to me that you're making the same mistake as pineapple.

We're not studying whether the moon is full or not - that, although it changes from night to night, is not something we're testing. We are seeking to test whether there is a relationship between the dog howling and the moon. If the dog howls 100% of the time when the moon is full, and 0% of the time when the moon is new, then one would think there is a relationship.

Using the very simple mathematics of dog howls 1/28th of the time, if you ran this mathematically for 84 days (3 moon cycles) I think you'd have the following statistic to determine how likely the causal effect exists:

(1/28)^3 * (27/28)^81

thus, we're finding the percentage chance that the dog just randomly barked on the 3 moon days, and also didn't bark on the 81 remaining days. Incidentally, the chances of the dog randomly howling on just the moon days - and none others - works out to happen randomly 0.000239% of the time.



Here's another example, with round numbers. I think your 1/28 and 35.7% figures are confusing me, as they're not obvious how they exist. I hope that you can adapt it to your "priori distribution."

Choosing the number on a single six-side die. Villain claims to be able to pick what number the die will roll. Villain claims that it will come up as a 6.

Die is rolled, and a 6 comes up. It seems to me that 1/6th of the time, villain

[apefish]

[ QUOTE ]

@apefish: It's possible to find a use for historical data sets of players with similar styles. For example, we could look at all hands from other 80/60 players that are headsup on the flop. Then we could ask questions like, if they were in the lead and checked to on the turn, how often did they bet between half pot and full pot and how often did their opponent fold/call/raise, and what were results when opponent did not fold. If we hypothesize that suspect player doesn't try to get opponents out on flop, but does try to fold them out or extract on turn, then we can make a test to compare how this player is doing at this compared to the benchmark who are likely to have similar hand quality on the turn. The tricky part is figuring out how to analyze a test like this. It hurts that (based on what I saw from a limited sample of Doubledrag hands) any exploitation is likely mostly occurring on turn/river, so it is harder to gauge what hand quality suspect player and opponents have at that point.

[/ QUOTE ]

Here's why I asked that. If we accept that no losing player playing that style is cheating in the manner we are talking about, and further that the more one is winning while playing that style the better chance is they are cheating (among other factors) then what we start to have as we accumulate data sets on the most maniacal of maniacs is a different version of what you are talking about.

Question we seek to answer: can a player playing this style recreate the results.
Maybe even more precise - HAS a player recreated these results.

Putting it side by side with previously known results starts to put into perspective how far out of the ordinary it is.

As some have said... it's doubtful we can even call it "just an outlier".

We don't need a perfect replication of known methods to get a strong general idea.

Something about walking like a duck...
I doubt anyone is asking for/expecting exact numbers/answers.

[flight2q]

Josem, here is how you test whether there is a correlation between wolf howling and full moons. I will describe likelihood ratios. There are other ways, but this is very standard. What knappis was doing was close to a likelihood ratio, that's why I was saying he was on a good track.

You plan to observe wolves on Df days with a full moon and Dn days with no full moon. You will end up recording Hf howlings with full moons and Hn howlings without. You premise that each day the wolves howl with some probability, either Pf or Pn, depending on whether there is a full moon, and that each day is independent. You don't know the values of Pf and Pn.

If howlings are uncorrelated to full moons, then Pf=Pn. You set this as the null hypothesis. For the test, you calculate the likelihood (so to speak) of the observed data having occurred if the null hypothesis were true; and also if it were false. Since there are many possibilities for the pair <Pf,Pn> either way, you decide to pick the pair where the likelihood is maximal. So the uncorrelated and correlated likelihoods used will be:

Lu = [(Hf+Hn)/(Df+Dn)]^(Hf+Hn) * [1-(Hf+Hn)/(Df+Dn)]^(Df+Dn-Hf-Hn)
Lc = [Hf/Df]^(Hf) * [1-Hf/Df]^(Df-Hf) * [Hn/Dn]^(Hn) * [1-Hn/Dn]^(Dn-Hn)

And we call Lc/Lu the likelihood ratio. The way this experiment is set up, the likelihood ratio is guaranteed to be at least 1.

We pick a value in advance of the experiment, call it L. We reject the null hypothesis if the likelihood ratio is greater than L. (This is a two-sided test. For a one-sided test, we would reject the null hypothesis only if both Lc/Lu>L and Hf/Df>Hn/Dn.) What we choose for L depends on various things, but it depends a lot on how we will use the results of the experiment. If the purpose of the experiment is merely to decide whether we should invest time and resources in a full scale investigation, then we can set L to a moderate value. If the purpose of the experiment is to take a legal action, then we would want a rather high value. In any case, L>>1. We also need to consider any effects that might arise if our data is not so good - for example, if our data came from going over records of people reporting wolf howlings, who might mention a full moon if there were one, but not mention it otherwise.

How good is our experiment? To evaluate this it would be useful if we had an a priori probability distribution on the pair <Pf,Pn>. Then we could crunch a bunch of numbers and determine two probabilities of likely interest. The first is the probability that we reject the null hypothesis, even though it is true. The second is the probability that we fail to reject the null hypothesis even though there is significant correlation. These are often called Type I and II errors. We can do these calculations for different values of L, our threshold for rejecting the null hypothesis, and tradeoff these risks.

Generally, we don't have an a priori distribution, but we can make these calculations for a few different values for the pair <Pf,Pn> and ask ourselves what we think of the effectiveness of the experiment. If there is significant probability of failing to reject the null hypothesis, even though there is correlation between howling and full moons, then we say that the power of the experiment is low. A typical way to increase the power is to increase the sample size, Df and Dn.

If our use of the data is such that we will use a rather high value for L, we must take especial care to ask ourselves a lot of questions about whether our model is correct or our data is invalid. For example, our model might be bad if there is serial correlation in the observations (our calculations assumed they were independent from day to day). Our collected data can be tainted in various ways, and we have to question our procedures if we rejected some of the data (e.g., we assume the wolves heard about our experiment and started deliberately howling when there was no full moon). For some of our concerns we may be able to devise tests to check our assumptions - i.e., a test for whether there is serial correlation.