ICM Quantified - First Set of Results

[rvg72]

[ QUOTE ]
[ QUOTE ]
I'm going to add some basic "skill" data to each data point to give us some way to measure this. I'm going to include Saw Flop %, Pre Flop Raise % and Post Flop Aggressiveness although it will only do it for this tourney. You rarely have good players seeing more than 15-20% of flops so you could use that as a basic way to differentiate or combine different factors to try to draw some conclusions.

[/ QUOTE ]
I don't want to sound overly negative -- I think this is outstanding work that could turn out to be very valuable. But, I don't think these "skill" measures will be useful. You rarely have good players seeing more than 15%-20% of flops, but that doesn't mean every good player falls outside that range, especially not during a single SNG. A good (or bad) player may have a run of good cards in a single SNG which causes them to see more flops. Any skill measure which is limited to the individual SNG in question is going to be of questionable value.

[/ QUOTE ]

The reason I think it has at least some value is that it does give you a reasonable idea about who you are dealing with. I would expect more value from that than a players ROI based on a sample of 5 or 10 tourney's.

I will include it in the database - how much value it adds remains to be seen but it is easier to include it now than try to include it later.

But point taken,

rvg

[AliasMrJones]

[ QUOTE ]
[ QUOTE ]
I think the best way to get unbiased results is to create a simulation where players start at a set chip stack at a push/fold level of blinds. They play according to ICM and then take those simulated results and we will see if the actual fits to the ICM model.

[/ QUOTE ]
I completely agree with this. A simulation would provide purer data than simply amassing huge samples.

Great work rvg, I look forward to seeing where this goes.

[/ QUOTE ]

ICM is a way to determine prize money equity given stack sizes. It isn't a play style. Whether or not ICM-derived "correct" plays are indeed "correct" is a different problem than whether ICM is a valid model of prize money equity.

Besides that, how are you going to simulate ICM-style play? You need calling ranges for opponents -- can the simulation do that? Will you include non-ICM-style players? Many/most players at the $55's don't understand pushbot ICM-style play and won't follow ICM-derived calling.

I just don't see what a simulation is going to get you. We're looking for real-world validation of the model, not simulated results compared to the model.

[mgross]

I see what you mean. But, at the same time, I think we could simulate ICM play (yes, with calling/pushing ranges), and we would see that even the simulated results would model the real-world much more closely than ICM does.

[AliasMrJones]

[ QUOTE ]
I see what you mean. But, at the same time, I think we could simulate ICM play (yes, with calling/pushing ranges), and we would see that even the simulated results would model the real-world much more closely than ICM does.

[/ QUOTE ]

I'm sorry, but this makes no sense. A simulation based entirely on ICM would model the real world more closely than ICM does? The simulation would be based on ICM!!!

Frankly, I think this discussion about an ICM-style-play simulation should be moved to a separate discussion as it really doesn't have any bearing on the original research, which was to validate the ICM prize money equity model with real-world data.

[Izverg04]

You made a cool-looking plot but it is deceiving without any attempt at estimating errors, either statistical or systematic.

You have plotted 4 data points (5-1 to account for total equity=1). You observe a deviation from ICM only at 2 of them, the one at low chip count, and the one at high chip count.

These are the points where you have the least data. You combine data at each hand, which might make you think you have a lot more data points than you actuall have, but you are counting some of them multiple times. If there is a >3500 chip stack at the table, you will count that stack several times as separate data points, while it is actually just one point. Without going into detailed analysis, I'd say the statistical deviation that you observe from the ICM is very, very small, less than 1 sigma.

Also, even after you get a statistically significant deviation with much more data, you can't make any claims about accuracy of the ICM before your correct for systematic effects. As pointed out already, there is a correlation between skill and chip counts on the bubble. This skill correlation has to be subtracted or you don't have a result.

[mgross]

Just because you make a move that ICM says is +$4.01 does not mean it is +$4.01. What I'm saying is that we can model play based on ICM and actually derive some pretty optimal calling/pushing strategies (ie. If I know my opponent is using ICM to model how to call to say a push from a random hand, I can then adjust my strategy. If he knows I adjusted, he can adjust again, etc...). Then, we can see really what the +$4.01 really is because it is not going to be +$4.01 just because ICM says it is (even though it is a simulation using ICM). We will get results based on this play and I think it would make for interesting results.

I don't think you can say that this is bogus.

[schwza]

hey rvg,

first, congrats on getting some results. i remember that we had swapped some data (and that mine wasn't useful to you b/c it didn't start from the beginning). my attempt to empirically calculate an icm-type curve stalled when pvs and the other guy i was working with got busy with some other things.

one thing i'd be curious to see is results that take into account more than one stack. for example, icm gives almost no difference between a stt with 4 left and stacks of 4k,2k,1,3999 and 4k,2k,4k even though in real life the second situation is obviously much better for the 2k stack guy because he can't get abused by the bigger guys. from what i can tell, the graph you showed just indicated the value of a 2k stack without taking into account what the other stacks were.

regarding the question of using more than one data point from a particular stt: me and eastbay have had this discussion a number of times, starting when i published an article in the 2+2 magazine in may '05 outlining how one would empirically calculate an icm-type curve. his position is that you can't do it. mine is that you can, but that i don't know the statistics to actually do it properly. the other guy i was working with along with pvs is an ecomonics professor who does a lot of econometrics and he didn't think it would be a problem. but you have to know some grad-level stats. i think he called it "panel data."

[rvg72]

[ QUOTE ]
You made a cool-looking plot but it is deceiving without any attempt at estimating errors, either statistical or systematic.

You have plotted 4 data points (5-1 to account for total equity=1). You observe a deviation from ICM only at 2 of them, the one at low chip count, and the one at high chip count.

These are the points where you have the least data. You combine data at each hand, which might make you think you have a lot more data points than you actuall have, but you are counting some of them multiple times. If there is a >3500 chip stack at the table, you will count that stack several times as separate data points, while it is actually just one point. Without going into detailed analysis, I'd say the statistical deviation that you observe from the ICM is very, very small, less than 1 sigma.

Also, even after you get a statistically significant deviation with much more data, you can't make any claims about accuracy of the ICM before your correct for systematic effects. As pointed out already, there is a correlation between skill and chip counts on the bubble. This skill correlation has to be subtracted or you don't have a result.

[/ QUOTE ]

You have some valid points although a few things you said were not totally accurate.

I think there is something to be learned from this and this could be the start of something much bigger. That is why I would be more than willing to compile the data and share it in its raw form so that people like you can analyze it further.

I'd like to point out that you are never going to collect a dataset that proves anything because there are so many ways that it can be picked apart. But, a concept like ICM is also imperfect in many ways and can be picked apart yet it has formed the basis of most 2+2ers entire late game strategy.

I have had about a half-dozen offers to send hand histories and this should take it to about 7,000 or 8,000 SNG's. I hope to get this much higher since undoubtedly some of these offers will fall through.

Thanks,

rvg

[rvg72]

[ QUOTE ]
his position is that you can't do it. mine is that you can, but that i don't know the statistics to actually do it properly.

[/ QUOTE ]

Ditto here - I also think it is valid obviously with the disclaimer that the sample needs to be large but also don't have the staistics background to prove that this is the case.

I'm a good programmer (at least I think so!) and also good with numbers / probability / statistics but there are others that majored in Statistics or use it daily in their line of work so I am happy sticking with the programming part and providing the data to the number crunchers to prove or disprove this. Whatever the outcome is I think it has value.

rvg

[rvg72]

[ QUOTE ]
Just because you make a move that ICM says is +$4.01 does not mean it is +$4.01. What I'm saying is that we can model play based on ICM and actually derive some pretty optimal calling/pushing strategies (ie. If I know my opponent is using ICM to model how to call to say a push from a random hand, I can then adjust my strategy. If he knows I adjusted, he can adjust again, etc...). Then, we can see really what the +$4.01 really is because it is not going to be +$4.01 just because ICM says it is (even though it is a simulation using ICM). We will get results based on this play and I think it would make for interesting results.

I don't think you can say that this is bogus.

[/ QUOTE ]

Well I do agree with a previous poster that this should be in its own thread... but...

I think what would make this interesting is adjusting the pushing / calling ranges of the players to various levels below optimal and used that to determine how this affects Real $ vs ICM. I don't know how practical this would be since in real life there are more than two options but it would be interesting to me regardless.

rvg