The Cab Problem

[PrayingMantis]

[ QUOTE ]
[ QUOTE ]
phzon,

How about if I put the problem this way:

Suppose every night there's a hit and run accident, caused by the same cab every night, for 1000 nights. Every night a different person sees the accident. All the witnesses have this 80%/20% probability of being right/wrong with identifying the color. This is some eye-brain problem with the people of this town.

Every day there's a "trial" with regard to the previous day's accident.

Now I randomly put you in court to see the 465th trial. The witness says the car from yesterday's accident was blue (we all know that there are 15% blue cabs in town, as in the original problem).

What would you say is the probability of the car being blue?

[/ QUOTE ]

I always like to look for a hidden assumption in a flawed argument. I think your setup above points to the hidden assumption being made by Cohen. Just like in your setup where the same car is causing all the accidents, I think Cohen is essentially saying, look, it was a Blue Cab that caused the accident. I know it was a Blue Cab that caused the accident. Therefore, the fact that only 1% or 2% or 15% of the cabs on the road were Blue that night is irrelevant. But don't take my word for it. Take the word of the witness who saw it as Blue with 80% accuracy and be 80% confident that he was right.

PairTheBoard

[/ QUOTE ]

PairTheBoard,

There isn't any hidden assumption in my variation of the problem. It's the same problem from a different perspective. I don't know about Cohen, but as to myself, I certainly do not say something like "look, it was a blue car". All I did was using some "trick" that should not matter at all when you think about the actual situation - you came into the court on the 465th trial, you know that it was the same car all previous days but you don't know which, and you don't know what did the previous witnesses said. It is exactly the same problem, as (a), (b) and (c), with regard to the previous day, are exactly as they are in the original problem. There is no hidden assumption here, absolutely none. The relevant information is exactly the same. The only thing is that, in my opinion, it sheds a different light on the potentially absurd side of using Bayes here.

[pzhon]

[ QUOTE ]
there is 0.8 probability that the 465th witness is correct -

[/ QUOTE ]
Ok.

[ QUOTE ]
so if he says "blue" there's 80% the cab was blue.

[/ QUOTE ]
Non sequitur.

There is an 80% chance that what he says is correct.

The conditional probability that he is correct when he says "blue" (12/29) need not be the conditional probability that he is correct when he says "green" (68/71). These conditional probabilities average to 80% (when weighted appropriately, 29:71).

[SplawnDarts]

[ QUOTE ]
[ QUOTE ]
Bayes Theorem is NOT the entirety of what's going on here.

The real question is whether the accuracy figure for the witness is "conditionally robust" or not. If it is, then the problem is internally self contradictory. If it's not, then you can potentially apply Bayes.

[/ QUOTE ]

What do you mean by "conditionally robust"? I've never heard of that term before.

PairTheBoard

[/ QUOTE ]

I'm sure there's other terms for it, but the concept comes up in a lot of fields, especially measurement devices.

Suppose I say P(x) = .8 - that's all well and good. Now, a stronger statement is to say that P(X|Y) = .8 for some set of Ys. That means that the original statement about P(X) was conditionally robust, that is to say it's independent of many conditions that one might think would influence it. This is not just an on/off property. If P(X|Y) is nearly .8 for a bunch of Ys but not all Ys, then it's somewhat robust.

Now, let's apply the concept to this problem. The witness is supposedly 80% accurate. So we know P(witness right about color) = .8. But is that conditionally robust? What about:
P(witness right about color | color is green)?
P(witness right about color | color is blue)?
P(witness right about color | witness says color is green)?
P(witness right about color | witness says color is blue)?

If all 4 of those are .8, you can do some algebra and prove that P(color is green) = .5, ditto P(color is blue). Which is a problem, because the question defines them to different values. Hence a contradiction & all the arguing in circles.

However, if we say our understanding of the witness' behavior is not conditionally robust, then it's very sketchy to apply Bayes theorem, because the value of P(witness right about color | witness says color is blue) is no longer .8 and we have no alternate value to use.

In other words, either the problem is internally contradictory or insufficiently specified depending on how you read it. The witness testing should have been carried out in a conditional manner so that the needed values were available and the witness behavior fully understood.

However, in the absence of such testing, you usually assume that measurements are conditionally independent until proven otherwise, which feeds more into the "Cohen" argument than the "Bayes" argument. That's why I said I supported "Cohen".

[PrayingMantis]

[ QUOTE ]
[ QUOTE ]
there is 0.8 probability that the 465th witness is correct -

[/ QUOTE ]
Ok.

[ QUOTE ]
so if he says "blue" there's 80% the cab was blue.

[/ QUOTE ]
Non sequitur.

There is an 80% chance that what he says is correct.

The conditional probability that he is correct when he says "blue" (12/29) need not be the conditional probability that he is correct when he says "green" (68/71). These conditional probabilities average to 80% (when weighted appropriately, 29:71).

[/ QUOTE ]

phzon,

Do you agree that among the 1000 witnesses (in the last variation of the problem), about 80% said "X" and 20% said "Y" (or vice versa)? and also, that the color that was mentioned about 80% of the time (X or Y) is the correct color of the cab that consistently caused the accidents? The same car that caused the accident on the 465th night?

Now, when the 465th witness says "X" or "Y", I know nothing but the notion that the color he is claiming the cab had is 80% to be the true color of the cab.

[PairTheBoard]

[ QUOTE ]
Now, let's apply the concept to this problem. The witness is supposedly 80% accurate. So we know P(witness right about color) = .8. But is that conditionally robust? What about:
P(witness right about color | color is green)?
P(witness right about color | color is blue)?
P(witness right about color | witness says color is green)?
P(witness right about color | witness says color is blue)?


[/ QUOTE ]

I take the 80% accuracy figure to mean that the first two conditional probabilties above are 80% while the second two are 80% if he is tested by being shown half blue and half green cabs. So the 80% is "conditionaly robust" under testing conditions.

But then the second two conditional probabilities change when confronted with the real world proportion of 85-15 for Green and Blue cabs on the road.

PairTheBoard

[SplawnDarts]

[ QUOTE ]
[ QUOTE ]
Now, let's apply the concept to this problem. The witness is supposedly 80% accurate. So we know P(witness right about color) = .8. But is that conditionally robust? What about:
P(witness right about color | color is green)?
P(witness right about color | color is blue)?
P(witness right about color | witness says color is green)?
P(witness right about color | witness says color is blue)?


[/ QUOTE ]

I take the 80% accuracy figure to mean that the first two conditional probabilties above are 80% while the second two are 80% if he is tested by being shown half blue and half green cabs. So the 80% is "conditionaly robust" under testing conditions.

But then the second two conditional probabilities change when confronted with the real world proportion of 85-15 for Green and Blue cabs on the road.

PairTheBoard

[/ QUOTE ]

Ok, that's not what the problem says but you're going to have to assume something to proceed so that's as good as any other option. If you assume that, you can apply Bayes which is nice. However, were this the real world I wouldn't advocate choosing assumptions based on math convenience.

[PairTheBoard]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
Now, let's apply the concept to this problem. The witness is supposedly 80% accurate. So we know P(witness right about color) = .8. But is that conditionally robust? What about:
P(witness right about color | color is green)?
P(witness right about color | color is blue)?
P(witness right about color | witness says color is green)?
P(witness right about color | witness says color is blue)?


[/ QUOTE ]

I take the 80% accuracy figure to mean that the first two conditional probabilties above are 80% while the second two are 80% if he is tested by being shown half blue and half green cabs. So the 80% is "conditionaly robust" under testing conditions.

But then the second two conditional probabilities change when confronted with the real world proportion of 85-15 for Green and Blue cabs on the road.

PairTheBoard

[/ QUOTE ]

Ok, that's not what the problem says but you're going to have to assume something to proceed so that's as good as any other option. If you assume that, you can apply Bayes which is nice. However, were this the real world I wouldn't advocate choosing assumptions based on math convenience.

[/ QUOTE ]

What we are told in the problem is this:

[ QUOTE ]
(c) when tested, the witness correctly identified the two colors 80% of the time.

[/ QUOTE ]

The question then is, how was this witness "tested"? In the real world I believe it would be proper to examine the testing method. Being unable to do so here we have to assume something about the testing method or else (c) might just as well read "(c) this statement carries no useful information". The common sense interpretation is I believe as I stated it. The witness was shown half blue and half green cabs. He identified the blue cabs correctly 80% of the time and identified the green cabs correctly 80% of the time. How else would you think the test might have been carried out so as to produce the statement (c)?

I suppose the test could have involved taking the witness out on the street to identify cabs as they drove by with an 85-15 Green/Blue cab ratio for his exposure. If that's what happened and 80% of the time a Blue Cab passed he said Blue and 80% of the time a Green Cab passed he said Green then I believe the Defense would have an extremely nice ripe question to ask on cross examination. With the statistics that were kept for that test, what percent of the time that the witness said "Blue" was the cab actually Blue?


PairTheBoard

[PairTheBoard]

[ QUOTE ]
Do you agree that among the 1000 witnesses (in the last variation of the problem), about 80% said "X" and 20% said "Y" (or vice versa)? and also, that the color that was mentioned about 80% of the time (X or Y) is the correct color of the cab that consistently caused the accidents? The same car that caused the accident on the 465th night?

Now, when the 465th witness says "X" or "Y", I know nothing but the notion that the color he is claiming the cab had is 80% to be the true color of the cab.


[/ QUOTE ]

Ah, you say there are no hidden assumptions like I described in my previous post. But I still contend that what you are doing is psychologically sweeping such a prejudgement under the rug.

Suppose I have a bag with 100 5 sided dice. 85 of the dice are Green on 4 sides and Blue on the 5th. 15 of the dice are Blue on 4 sides and Green on the 5th.

Suppose I randomly pull one die from the bag and roll it. Suppose further that it comes up Blue. What's the probablilty that it's a Die with 4 Blue sides? That's essentially the original problem and pzhon's Bayes calculation gives the answer.

Now suppose that same die is rolled 1000 times. If it's the Mostly Green Die it will come up green 80% of those 1000 times rolled. If it's the mostly Blue Die it will come up blue 80% of the 1000 times rolled. But we don't know which it's doing those 1000 times. All we know is that on 465th roll it comes up Blue. Since the results of the other 999 rolls are unknown to us they might as well not exist for purposes of our calculation. The 465th roll might just as well be the one and only roll and when asked what the probabilty is that it's a Mostly Blue Die producing that roll the answer is the same as in the First Case. That is, unless you have somehow tricked us psychologically into being predisposed into thinking that it's probably the Mostly Blue Die getting rolled all the time.

PairTheBoard

[pzhon]

[ QUOTE ]
The common sense interpretation is I believe as I stated it. The witness was shown half blue and half green cabs. He identified the blue cabs correctly 80% of the time and identified the green cabs correctly 80% of the time. How else would you think the test might have been carried out so as to produce the statement (c)?

[/ QUOTE ]
In some statements of the problem, the procedure is stated more explicitly to be this common sense interpretation. In fact, I believe Kahneman and Tversky stated in the problem, "the witness correctly identified each one of the two colors 80% of the time and failed 20% of the time, " and in their analysis, "i.e., p(b|B) = p(g|G) = 80%." (I'm relying on quotes in secondary sources. It is easy for quotes to be shortened, but harder for writers to add phrases.)

To clarify another assumption that is sometimes also made explicit, the drivers from each cab companies are supposed to be indistinguishable, so that the a priori probability of an accident is proportional to the number of cabs.

The legal issue is interesting in that civil cases are decided by the preponderance of credible evidence, i.e., you should be able to sue the Green Cab company if you establish that the company had a greater than 50% probability of being the cause of your injuries, despite the presence of a witness who claims the cab was blue.

[ QUOTE ]

I suppose the test could have involved taking the witness out on the street to identify cabs as they drove by with an 85-15 Green/Blue cab ratio for his exposure.


[/ QUOTE ]
That still wouldn't justify Cohen's position. It introduces a variable, but it's still unreasonable to assume that the frequency of the cabs is irrelevant.

[pzhon]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
there is 0.8 probability that the 465th witness is correct -

[/ QUOTE ]
Ok.

[ QUOTE ]
so if he says "blue" there's 80% the cab was blue.

[/ QUOTE ]
Non sequitur.

There is an 80% chance that what he says is correct.

The conditional probability that he is correct when he says "blue" (12/29) need not be the conditional probability that he is correct when he says "green" (68/71). These conditional probabilities average to 80% (when weighted appropriately, 29:71).

[/ QUOTE ]

[/ QUOTE ]
You quoted all of that. Did you read it? It answers what you wrote in response.

[ QUOTE ]
Do you agree that among the 1000 witnesses (in the last variation of the problem), about 80% said "X" and 20% said "Y" (or vice versa)?

[/ QUOTE ]
With high probability, either roughly 80% said green and 20% said blue, or 80% said blue and 20% said green. I doubt they would say X or Y.

[ QUOTE ]
Now, when the 465th witness says "X" or "Y", I know nothing but the notion that the color he is claiming the cab had is 80% to be the true color of the cab.

[/ QUOTE ]
When you willfully throw information out, your estimates of probabilities will be bad. You are combining the actual case where the witness says "Blue" with another situation in which the witness says "Green." This is an error.

I'm surprised that this is not intuitive to you. Applications of conditional probability (strength inferences) are common in poker. Do you also disagree with the standard problem from introductory probability about the fallible test for a rare disease?