ICM Quantified - First Set of Results

[gumpzilla]

1) As is being discussed here and elsewhere, I think taking multiple data points from the same game is a problem, because it pulls in multiple data points with set skill disparities. It's possible that these disparities will average out, but given that there is a correlation between length of game and stack sizes (also skill disparities too, most likely), certain data points are going to be weighted more heavily than others. How seriously this mucks things up is unclear.

2) More seriously, are you taking all of the stacks into account? The predictions of ICM for a given stack can't be considered in a vacuum, since the stacks of everybody else is involved in making the prediction. The difference between me having 500, 2000, 2000, 2000 and 500, 1, 1, 5998 is quite strong.

Thanks for putting the time in to do this. I'm not trying to discourage you from continuing, just pointing out that assessing this information requires some thought to make sure you're probing the right things.

[rvg72]

[ QUOTE ]
2) More seriously, are you taking all of the stacks into account? The predictions of ICM for a given stack can't be considered in a vacuum, since the stacks of everybody else is involved in making the prediction. The difference between me having 500, 2000, 2000, 2000 and 500, 1, 1, 5998 is quite strong.

[/ QUOTE ]

Good point - I did not factor this since each time I further break down the results into different criteria I cut down the amount of data I can use and that, I think, has a more negative effect than simply lumping them all together - filtering out anomolies like this would probably be a better approach although I'm not sure if this would change the numbers due to the infrequency of it. But that's just my gut and a little bit of my head telling me it's ok to do that at least for the first set of results.

But, with the new data structure I have suggested, it will include hand number with every data point so that anyone could filter out or in some other way manipulate the data based on others stack sizes.

The comments that are being made are all constructive - so far no "rvg, you suck" comments so I'm happy with the dialog.

thanks,

rvg

[Phil Van Sexton]

Sorry if this has been said already.

I really like the addition of the hand #.

I'd also like to know the sequence of hands. For example, you are UTG in a 4 handed game, what is the $EV of folding? Well, that it's not really how we do it now because a player behind you may be knocked out. The *actual* EV of folding is your EV to start the next hand, not the EV you started this hand with.

I'd also like you to add a player_id. This will allow people to figure out the ROI of the various players, but without knowing everyone's names. Of course, you would need to keep a map somewhere of name->id that is only for your use.

If we could calculate player ROI, people could do whatever skill factor stuff they want.

Also, handle case where there is no SB. This may needed if someone wants to calculate their own EV for a hand.

Anyway, here is the MySQL schema that I was using if you care (yes, I know it's not normalized):

CREATE TABLE tournament (
tournament_id INTEGER not null primary key auto_increment,
number CHAR(12) not null,
buyin DECIMAL(6, 2),
fee DECIMAL(6, 2),
start DATETIME,
site CHAR(3),
seat1 CHAR(30),
seat2 CHAR(30),
seat3 CHAR(30),
seat4 CHAR(30),
seat5 CHAR(30),
seat6 CHAR(30),
seat7 CHAR(30),
seat8 CHAR(30),
seat9 CHAR(30),
seat10 CHAR(30),
finish1 TINYINT,
finish2 TINYINT,
finish3 TINYINT,
finish4 TINYINT,
finish5 TINYINT,
finish6 TINYINT,
finish7 TINYINT,
finish8 TINYINT,
finish9 TINYINT,
finish10 TINYINT
) type = BDB
;


CREATE TABLE game (
game_id INTEGER not null primary key auto_increment,
number CHAR(12) not null,
sb INTEGER not null,
bb INTEGER not null,
button TINYINT,
num_players TINYINT,
chips1 INTEGER,
chips2 INTEGER,
chips3 INTEGER,
chips4 INTEGER,
chips5 INTEGER,
chips6 INTEGER,
chips7 INTEGER,
chips8 INTEGER,
chips9 INTEGER,
chips10 INTEGER,
tournament_id INTEGER
) type = BDB
;

[rvg72]

[ QUOTE ]
I'd also like to know the sequence of hands.

[/ QUOTE ]

So you would like to know if it is the 44th hand of the SNG - I can add that easily enough.

[ QUOTE ]
I'd also like you to add a player_id. This will allow people to figure out the ROI of the various players, but without knowing everyone's names.

[/ QUOTE ]

Ok, I think this is needed in order to take this to the next level... What I can do is apply some type of hash to the players name so that every name has a unique hash (ID) while ensuring that there is no way to determine the original player's identity without knowing the key. So you could see that so and so has x ROI and y VPIP% and you see which hands he was involved with but you would have no way of determining who he is.

I will also put some type of hash on the game and tourney numbers so that nobody gets tricky and tries to put together the original player names by looking up the hand numbers.

[ QUOTE ]
If we could calculate player ROI, people could do whatever skill factor stuff they want.

[/ QUOTE ]

This could be extracted through the raw data but to make it mor accessible I'll create a table that contains the hashed ID of every player in the sample, their ITM, ROI and Saw Flop%. Any others?

[ QUOTE ]
Also, handle case where there is no SB.

[/ QUOTE ]

I'll have blind amount posted for each player and each hand - good enough, right?

Thanks for the feedback,

rvg

[rvg72]

Well since I'm going to be going through a lot of trouble putting this together I thought I'd put together a Table structure that could potentially open this up to other analysis. Nothing much new here that hasn't already been discussed in this thread but here it goes. Please provide comments. I still need many more hand histories but the result is going to be a database that contains no personal information but so much potential.

I will probably include a second database that is not normalized because sometimes that makes things a little easier. I put this together pretty quickly so let me know if I'm missing anything.

Players
===========
PlayerID - Auto Number
PlayerHash - GUID

Tourneys
===========
TourneyID - Auto Number
TourneyHash - GUID
TourneyBuyIn - double

PlayerSummarizedResults (summarized results for a player for all tourney's with the specified buyin)
========================
PlayerID - long
Buyin - double
TourneyCount - int
ROI - double
ITM - double
First_Percent - double
Second_Percent - double
Third_Percent - double
VPIP_Percent - double
SawFlop_Percent - double
PFR_Percent - double
Agg_Rating - double
StealAttempt_Percent - double

Hands
=============
HandID - Auto Number
HandHash - GUID
TourneyID - long
HandNumberInSNG - int
BigBlindAmount - int
SmallBlindAmount - int
PlayersLeft - int

Player_Hands
=============
PlayerID - long
HandID - long
StackSize - int
BlindAmountPosted - int
NumberOfBigBlindsRemaining - int
Level - int
ICMValueBeforePosting - double
ICMValueAfterPosting - double
RealDollarResult - double
ChipRanking - int
DistanceFromBigBlind - int

rvg

[Madd]

Thank you very much for your effort. This is really interesting.

Your results indicate that ICM undervalues bigstacks and overvalues shortstacks. That might be true, however, I'm wondering about cause and effect.
Better players are supposed to have more chips on average and will finish even higher than their expectation.
Weaker players have less chips and tend to finish lower than their expectation.
So I'm not yet convinced that your data refutes the assumption of the accuracy of the ICM.

[rvg72]

[ QUOTE ]
Thank you very much for your effort. This is really interesting.

Your results indicate that ICM undervalues bigstacks and overvalues shortstacks. That might be true, however, I'm wondering about cause and effect.
Better players are supposed to have more chips on average and will finish even higher than their expectation.
Weaker players have less chips and tend to finish lower than their expectation.
So I'm not yet convinced that your data refutes the assumption of the accuracy of the ICM.

[/ QUOTE ]

Skill definately plays a role in this - many good points have been made around this argument and I think it is valid but I also think that when we remove skill from the equation we will still see this although likely not as large a difference.

Phase 2 of this study will attempt to expand the scope of the data so that this and many other things can be proven out by the resident analytical experts.

Thanks,

rvg

[Izverg04]

[ QUOTE ]

Your results indicate that ICM undervalues bigstacks and overvalues shortstacks.

[/ QUOTE ]
They don't -- not yet. But this project is only starting.

[checkers777]

Could it be possible that there is nothing wrong with ICM and that these conficting results are based on the fact that "in general" players with bigger stacks on the bubble are better players and "in general" players with smaller stacks are worse players?

[HesseJam]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
I'm going to add some basic "skill" data to each data point to give us some way to measure this. I'm going to include Saw Flop %, Pre Flop Raise % and Post Flop Aggressiveness although it will only do it for this tourney. You rarely have good players seeing more than 15-20% of flops so you could use that as a basic way to differentiate or combine different factors to try to draw some conclusions.

[/ QUOTE ]
I don't want to sound overly negative -- I think this is outstanding work that could turn out to be very valuable. But, I don't think these "skill" measures will be useful. You rarely have good players seeing more than 15%-20% of flops, but that doesn't mean every good player falls outside that range, especially not during a single SNG. A good (or bad) player may have a run of good cards in a single SNG which causes them to see more flops. Any skill measure which is limited to the individual SNG in question is going to be of questionable value.

[/ QUOTE ]

The reason I think it has at least some value is that it does give you a reasonable idea about who you are dealing with. I would expect more value from that than a players ROI based on a sample of 5 or 10 tourney's.

I will include it in the database - how much value it adds remains to be seen but it is easier to include it now than try to include it later.

But point taken,

rvg

[/ QUOTE ]

First: Great job, rvg!

I won't harm to include the figures but I also think that the variance of Saw Flop %, Pre Flop Raise % and Post Flop Aggressiveness might be a tad high for the individual tourney. You could check that with your own stats.