Will computers dominate poker as they did with chess?

[JavaNut]

That could actually happen. At least for the lower limits. It will force pros up in limits and bots having limited playing time at lower limits might not be profitable at all.

[Artsemis]

[ QUOTE ]
aaron, what proof do you have of computers' learning abilities?

[/ QUOTE ]

Neural Networks

[Artsemis]

[ QUOTE ]
Poker: You can't really teach a computer effective Level 2 thinking or beyond. Sure, if you have a player who's playing a little too agressively, the computer can start to reraise and slowplay. But computers will never know if the limper in early position is trying to trap with aces, or get in cheap with 54s. They will never know if the check-raise when the third heart on the turn hits means they have it. They won't be able to put you on a range of hands.

[/ QUOTE ]

Wrong, and they will not only be able to put you on a range of hands, but be able to crunch the numbers to figure out based on that range, what the most EV play is.

What you guys don't realize is that given enough examples and standard "rules", a program is going to be able to determine things just as you are able to. You gave the example of a computer not knowing what a checkraise on the turn means -- correct, but do you? No, but you can narrow it down, as can a computer... yet the computer is going to be able to check hundreds or thousands of hands with that same player to be even more accurate than you.

We already have Poker Tracker and that is the core of information a bot would need.

- A software engineer

[pzhon]

[ QUOTE ]
[ QUOTE ]
You have confused separate events.

[/ QUOTE ]
Sorry, but this isn't quite correct either. [img]/images/graemlins/smile.gif[/img]


[/ QUOTE ]
Please explain exactly what you think was incorrect. My version is based on the papers by Tesauro I read, in for example, the journal Machine Learning, and correspondence with Tesauro, among other things.

[ QUOTE ]
[ QUOTE ]
Tesauro created a backgammon bot which learned from self play and the rules of the game alone to play better than all other backgammon bots at the time (in the late '80s). This was a triumph of self-learning over older methods of trying to feed programs human expertise. However, this bot still played poorly by human standards.

[/ QUOTE ]
Tesauro's first bot created in 1989 (called NEUROGAMMON) actually didn't learn from self-play - it's heuristic was trained using expert annotated training data
(see here).

[/ QUOTE ]
Yes, I wasn't referring to Neurogammon, or Tesauro's other earlier experiments on parts of the game. I was referring to TD Gammon. I believe TD Gammon version 0 beat Neurogammon based on self-learning from the rules of the game, although I don't know the date its training was completed. TD Gammon 0 was not a great player, though.

[ QUOTE ]

[ QUOTE ]
Tesauro then created a strong bot which was close to the strength of the top players of the time. However, this was based on a better method of encoding the board, and other information from people.

[/ QUOTE ]
Tesauro's second bot created in 1995 (called TD-GAMMON) did use a method of reinforced learning called "Temporal Differencing" (see here)

[/ QUOTE ]
I'm quite familiar with TD(lambda) learning.

Your chronology is wrong. It does not agree with the sources you cite. Bill Robertie, a 2+2 author, published a booklet about playing TD Gammon version 1 a series of games in October 1991.

At that time, Robertie's estimate of a surprisingly small advantage over TD Gammon was used as evidence of TD Gammon's strength. However, I believe Robertie overestimated his advantage. When Robertie disagreed with TD Gammon, TD Gammon was sometimes right. Robertie was stronger, but he also won the world championship in backgammon twice. TD Gammon 1.0 played at a competitive level for the time.

Jellyfish, based primarily on the same technology as TD Gammon, was released in 1994.

Some version of TD Gammon might have been released in 1995, but it's not the one people talk about. I believe you are referring to the version included in the "Family Funpack for OS/2" or something similar. Did anyone ever use that? This was a half-hearted attempt by IBM to commercialize what had been a theoretical triumph years earlier.

[RedManPlus]

[ QUOTE ]

When it comes to Poker AI specifically, I think one of the main stumbling blocks is the "get rich quick" mentality which is stifling current research. This not only applies to the wave of bot-builders trying to cash in from using their bots online, but is also evident in academia (eg: the lack of repeatability in the UofA papers and the subsequent commercialization of their work in the "Poker Academy" software). Past research on games like chess did not have this problem and I think the sharing of information means it was not as handicapped by greed as Poker AI is currently. I'm quite sure that if all the academic man-hours that were aimed at computer chess were somehow re-applied to Poker AI then we would have some very strong poker engines available today.


[/ QUOTE ]

Actually, no one is getting "rich quick" building Poker Bots. It's something that looks good to penniless students.
Older people think in terms of stable business models. 99% of kind of people with world-class talent required to advance Poker AI... have chosen the financial markets, internet start-ups, or even the Spam World (all MUCH more lucrative, any IPO can net you millions).

For example... I run a successful trading business centered on Automated Trading Systems. I have the background and resources to get into the Poker Bot business. It's very interesting mathematically. But you cannot build a stable business around it... and it's very low on the social acceptability scale... so I have not been able to justify moving resources from a very profitable trading operation in order to build a Poker Bot operation.

Did you see any "great minds" involved with the Full Tilt Bot thing? Relative to what people are doing on Wall Street... that was/is a Mickey Mouse operation.

[JocK]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
You have confused separate events.

[/ QUOTE ]
Sorry, but this isn't quite correct either. [img]/images/graemlins/smile.gif[/img]


[/ QUOTE ]
Please explain exactly what you think was incorrect. My version is based on the papers by Tesauro I read, in for example, the journal Machine Learning, and correspondence with Tesauro, among other things.


[/ QUOTE ]
All of this is irrelevant. The simple point is that self-trained backgammon bots do prove that computer learning is possible. (That was the question, isn't it?) In fact, self-trained backgammon bots discovered novel opening moves overlooked by human experts and thereby elevated our understanding of the game. So, these bots didn't just learn, they also invented... [img]/images/graemlins/wink.gif[/img]

[jukofyork]

[ QUOTE ]
Please explain exactly what you think was incorrect.

[/ QUOTE ]
I just assumed when you said Tesauro's first "learning" bot was created in the "late 80's" you were referring to his Neurogammon bot (based on the list of references to Neurogammon listed this page among others).

Juk [img]/images/graemlins/smile.gif[/img]

[pzhon]

[ QUOTE ]
The simple point is that self-trained backgammon bots do prove that computer learning is possible. (That was the question, isn't it?)

[/ QUOTE ]
The question was not whether a neural net can be trained to learn that 2+2=4. The question is whether a computer program can be trained to play poker at a very high level.

The example you raised of backgammon does not suggest that computer programs can learn to play poker well through self-play, as that didn't happen in backgammon. It suggests that the help of poker experts may be highly beneficial in the training process.

[ QUOTE ]
In fact, self-trained backgammon bots discovered novel opening moves overlooked by human experts and thereby elevated our understanding of the game.

[/ QUOTE ]
The main contributions of backgammon bots have not been in the opening. Human experts strongly disliked making the 2 point with the opening 6-4. Early backgammon bots strongly favored it. The current understanding is that the 3 natural plays 8/2 6/2, 24/14, and 24/18 13/9 are very close; both human and bot evaluations were wrong.

[ QUOTE ]

So, these bots didn't just learn, they also invented... [img]/images/graemlins/wink.gif[/img]

[/ QUOTE ]
I'm a big fan of AI. I own an AI company. However, you are overstating the case.

Most of the people working on poker bots are trying limited techniques which have no chance of playing poker well.

[6471849653]

It's already known how how fast it happened with chess; it all happened during the time Kasparov was the world champion. He even himself called computers stupid, and it did not take more than ten years if even that and he lost a match against Deep Blue II. In poker the same thing is hardly going to take 50 years - 10 years looks more like it though I have been around since 1999 and have been looking how the commercial poker softwares play and things haven't improved there other than on additional ideas. But putting it all together - and more that are already known - I have been able to see how the bots soon or even currently on poker sites play and will play, and though it was slow in the beginning, it can happen any time now when a serious product will be out there somewhere. We have also advanced a lot with computers since the old days.

[6471849653]

[ QUOTE ]
chess is a game of perfect information, wheras poker is one of imperfect info

[/ QUOTE ]

A well known fact is that chess is similarly a game of imperfect information as they can't see it all the way to the mate. What they do is that they just calculate, as far as one gives them time to calculate, and compare the results and pick a path (a position) that gives the best +EV, no different from poker - that's exactly what I do when I play poker, it's just that the computers can do more accurate calculations, and that's the way they beat all humans at chess and will soon at poker.

Heads up limit poker is the easiest as there are strategies that can't be beaten even if one knows exactly how the other plays; though not that they will win then but they will not lose for the simple fact that poker supports aggression (it's the way one must play and putting the opponent into the situation where he will lose no matter what he does - until he finds the exact ways of not to lose - is the name of the game, e.g. one has either a draw or a better solid hand, that's the situation one will face the opponent with and it can't be beaten).