Bayes' Theorem/Bayesian Probability

[samsonite2100]

Okay, I know I'm setting myself up for a lot of abuse here, but screw it. Can someone pretty please explain Bayes' Theorem/Bayesian Probability in a concise and intelligible way for someone who's pretty bad at math? Just pretend you're talking to a retarded person, okay? I've wiki'ed it, but the entry got complicated fairly quickly. BT is namechecked around 2+2 constantly and I'd like to understand what people are actually talking about.

[PairTheBoard]

They are different things. Bayes Theorem is pretty easy to understand. Suppose you have two Mutually Exclusive Exhastive events, A,B. All outcomes fall in one of the two but no outcome falls in both. A random outcome has probability P(A) for being in A, P(B) for being in B. Now an outcome x occurs and what you know about x is that it is in Event C. What can you say about the probabilty x is in A? Is it still P(A)? Not necessarily. Bayes Theorem says, now that we know C has occured (ie. x is in C) we now know more about the probability that A has also occured (x is in A). In fact we know it is,

P(A and C)/P(C)

where P(A and C) is the general probabilty a random outcome is in both A and C. ie. The general probability that A and C both occur. P(C) is the general probability that C occurs.

Heres a good example: A test for HIV is performed. If the person has HIV the test will be positive 90% of the time (10% false negatives). If the person doesn't have HIV the test will be negative 90% of the time (10% false positive). Now a person is chosen at random from a population where 98% do not have HIV and 2% do have HIV.


The Test is given to the person. He tests positive for HIV. What's the probability he actually has HIV? Is it 90% because the test is 90% accurate both for positive and negative results? Is it 2% because he came from a 2% HIV population? Neither. Bayes Theorem say the probability he is actually in the 2% of the population who has HIV is,

P(A random person Has HIV and tests positive for it)
divided by/ P(a random person tests positive for HIV)

Now, P(A random person Has HIV and tests positive for it) is just
(2%)(90%)

For the denominator, realize that a random person can test positive for HIV in two ways. He can have HIV and the test shows positive. Or he can not have HIV and the test shows a False Positive. The probabilility that one of these two things happens is:

(2%)(90%) + (98%)(10%)

So Bayes tells us that this random person who tested positive for HIV on a test with 90% accuracy, actually has a probability of having HIV of,

(2%)(90%)/[(2%)(90%) + (98%)(10%)] = about 15.5%

That's much lower than a lot of people would think. This shows that tests for rare events must be very very accurate if you want them to provide a high probability that the rare event is actually happening.

Here's a simpler example. You have two dice. Die A has 3 red sides and 3 blue sides. Die B has 5 red sides and 1 blue side. Now you choose one of the Dice at random. You roll it and it comes up red. What's the probability that it is Die B? Bayes says it is:

(5/6) / [(5/6)+(3/6)] = 62.5%

You use Bayes Theorem all the time in Poker without realizing it. Before any action takes place, what's the odds your opponent has Aces? 220-1? Now he goes All-In. Do you still think the odds against his having Aces are 220-1?
No. You apply Bayes' Theorem and adjust them.

Bayes Probability is a whole different thing however. The Bayes Probablists have a different approach to Probability than what's called the "Frequentists". Frequentists think of Probability as the frequency which an event will occur if the random experiment is repeated a large number of times. For example, a standard deck of cards is shuffled. What's the Probability the top card is a Spade? The frequentist thinks of repeating the experiment of shuffling the cards many times and looking at how often a Spade comes up. To him, the Probabilty is that frequency, 25%. The Baysian Probablist looks at it more as a one time event. He thinks of the Probability as being a degree of confidence. ie. I have a 25% degree of confidence or partial belief that the card on the top of This Particular Deck for This Particular Shuffle is a Spade. It is really a different kind of philosophical attitude toward what Probability means.

The advantage for the Baysian is that he is comfortable dealing with One Time Events. The Frequentist has to jump through some mental hoops sometimes when dealing with One Time Events. For example, the Baysian approach is probably the way people really think when they estimate odds on a Sporting Event. I have about 3% confidence the Jets will win the Superbowl next year. The Frequentist is a little hard pressed to theorize that Event according to some repeatable model.

The tricky thing is that all Mathematical Probability calculations are done based on Frequentist models. And those are the models that Baysian Probablists use as well. They both use the same Math. However, even though they use the same Math, there are Baysian Statisticians who do Statistics differently than Frequentist Statistician. I'm not an expert here. But as I understand it, Baysian Statisticians often think it is better to make informed assumptions and estimates for the population you are studying prior to applying statistical methods to the data you collect.

For example, in the HIV test above. Suppose no data was available for the percent in the population that had HIV. Suppose the 2% was not known. Suppose no data was available for that. The frequentist might interpret data he collects about that population under the assumption that nothing is known about it. He might then conclude 90% confidence that the person who tested positive has HIV. That might work well if he is taking a large sample of such people to try to determine how prevelant HIV is in the population.

But it sucks in the One Time Event which that particular person is involved with. He would be very interested in the report given him by the Baysian Probablist who assumes HIV is relatively rare in the population. If that Baysian Probablist assumed the rate was 2% he would report the 15.5% confidence level to the Patient. If the Baysian assumed a rate of 20% he would report a different figure, but it would still be less than the 90% reported by the frequentist.

It can make a big difference what kind of Statistician you call to the witness stand in a trial.

PairTheBoard

[jason1990]

Here is a partial answer to your questions. Most often, when someone mentions Bayes Theorem on 2+2, they simply mean

P(A|B) = P(A and B)/P(B).

You may already know this, but the left-hand side means "the probability of A given B." This is actually not Bayes Theorem. This is the definition of conditional probability. Bayes Theorem appears in many different forms in various textbooks. One common version is this:

P(A|B) = P(A)P(B|A)/(P(A)P(B|A) + P(A^c)P(B|A^c)).

In this formula, A^c is the complement of A. That is, A^c is the event that A does not happen. This formula is useful in situations where it is easy to compute probabilities given A and A^c, but it is not easy to compute probabilities given B.

For example, you may "know" (or assume) that your opponent will raise/call/check with certain probabilities, given that he holds hand XX. But in practice, you are interested in reversing this. You really want the probability he holds hand XX, given that he raised (or called or checked). There are a lot of other interesting examples. One common example involves doing blood tests to check for rare diseases. An example about an eye-witness to a car accident involving green and blue cabs was recently(?) discussed in the probability forum. You should be able to search around and find some.

[PairTheBoard]

[ QUOTE ]

Here is a partial answer to your questions. Most often, when someone mentions Bayes Theorem on 2+2, they simply mean

(*) P(A|B) = P(A and B)/P(B).

You may already know this, but the left-hand side means "the probability of A given B." This is actually not Bayes Theorem. This is the definition of conditional probability. Bayes Theorem appears in many different forms in various textbooks. One common version is this:

(**) P(A|B) = P(A)P(B|A)/(P(A)P(B|A) + P(A^c)P(B|A^c)).

In this formula, A^c is the complement of A. That is, A^c is the event that A does not happen. This formula is useful in situations where it is easy to compute probabilities given A and A^c, but it is not easy to compute probabilities given B.


[/ QUOTE ]

Something to notice for how (*) and (**) are related. The Denominator in (**) is just a way of computing P(B).

So (**) reduces to:

P(A|B) = P(A)P(B|A)/P(B)

Also,

P(B|A) = P(A and B)/P(A)

So (**) reduces further to

P(B|A) = P(A)P(A and B))/P(A)(P(B)

The P(A)'s cancel and you get

P(B|A) = P(A and B)/P(B)

This in fact amounts to a proof of Bayes' Theorem with the step left out showing that the Denominator in (**) is indeed P(B).

PairTheBoard

[samsonite2100]

Hey, thanks guys. PTB--the HIV example is good, helped out a lot. Cheers.