New Thread On Sklansky Extrapolation Question

[David Sklansky]

There were too many side issues discussed on the other thread. Here is a clearer restatement of the question.

Say people are carefully evaluated and are classified by how likely they are to get things right on yes or no questions. They are rated from A to G. A's are historically the most likely to get things right. But even they are far from perfect.

A specific question comes up- "Is Y true?"

The general consensus is that it is not. 70% of A's think it is not.

The thing is that 100% of G's think Y isn't true. Same with 95% of F's, 90% of E's, 85% of D's, 80% of C's and 75% of B's.

There is a clear pattern. And it is heading toward a conclusion that if there were people substantially better than A's at getting answers (call them Omegas), most of them would in fact believe that Y IS true.

The question is whether it is reasonable, given no other information, to think the pattern will continue and that it would be a good bet to put your money on Y's truth. Or should we assume the pattern probably WON'T continue. Meaning that most omegas, if they existed, would agree with the majority of the A's (and everyone else).

Here is another way of looking at it. Without knowledge of the survey, seventy percent of A's think not only that Y is true but also that most omegas also think that y is true.
Should they change their mind when apprised of the survey and the pattern it seems to show? Remember that the survey will tell them that 70% of the smartest people agree with them.

In order to be persuaded to change their minds an A who is one of the 70% to believe Y, must somehow think changing is right, though 70% of the smartest agree with him, soley because even MORE than 70% of the less smart agree with him. Could that make sense? Yet if he sticks to his guns, he is defying a pattern when there is no good reason to think it wouldn't continue. (To make the impact clearer, one might imagine that the known pattern ends with 55% rather than 70% of the smartest people disagreeing with Y. Now only a small extrapolation has smarter people yet, agreeing with Y.)

[PairTheBoard]

Did you read Tom Crowley's analysis on the original thread? He points out that the fattening of the Yes percent may be due to the G group overestimating the correct Price say of 67% for a No answer. As you go up the scale from G to A, if the distribution doesn't tighten much, you will see the tail of the distribution move down into the Yes area while the mean moves accurately downward and closer to the correct 67% Price for No.

If better evidence evaluators were available you would eventually see the spread of the distribution tighten up, and the tail in the Yes area disappear as opinion converged on a 67% Price for No. The percent of people actually answering No would then concurrently rise back up to 100%.

So your extrapolation idea is simply unsound even under the kind of assumptions you want to make. I suppose you might speculate on the relative sizes of classes of propositions for which G's overestimate and underestimate the correct price, but that seems pretty futile to me.

PairTheBoard

[ZeeJustin]

2 thoughts come to mind.

1) The answer to this question doesn't necessarily have to be the same if Y is changed to "There is no God". I hope that is obvious to anyone that responds.

2) I think Y will almost always be true. There will be rare exceptions. In those rare exceptions, 2 things are probable
1: Nearly 100% of Omega's are wrong, and the G's are right, but for the wrong reason.
2: There is some sort of curve. Perhaps Y seems obviously false, so the G's are right without thinking. The A's have some advanced concept that makes some of them disagree with the G's. The Omega's are smart enough to understand the flaw in the concept. Basically there is some fancy play syndrome going on.

[ZeeJustin]

I also want to add that I can't think of a single case in my entire adult life where I had been told that "most the experts think Y", and my response wasn't either of the following:

"Well I guess Y must be true."

"There's a good chance you are wrong about most the experts thinking Y is true, but if you are right, then Y is probably true."

I guess I need to exclude poker from this, because I think I have enough knowledge in several areas of poker theory to reasonably disagree with even the most qualified experts.

Edit: That last paragraph I wrote made me think of something. Assuming that the A's are the top echelon, it might be rational for an A to think Y is false even after seeing the survey results, because it is possible he is better qualified than anyone else. But it is extremely unlikely that a B shouldn't change his mind after seeing the results of the survey, because it is unlikely that he is more qualified than the A's that disagree with him. G's should absolutely change their mind once hearing of the survey.

I think the answer to David's question is obviously, "Y is probably true even though 70% of A's disagree", but I think it is a lot more interesting if you instead ask, "You are an A. You were 95-98% confident that Y is false. Should you change your opinion after hearing the results of the survey?" I think the answer is probably yes more often than it is no.

[not a model]

english isnt my first language, but i still speak it with a lot of beauty and grace. with this in mind, what do you mean by "Price," pairtheboard?

it seems to me like you are just being captious and overcomplicating matters to try to find a specific instance where the answer is "no" to david's question of whether or not it's reasonable to assume that Y is probably true.

i would say of course it's a reasonable assumption that Y is true. and i could think of many easy examples where it is the case. some conceivable examples of Y:

the monty hall problem (especially if you asked it back in the 1950s)

asking whether the word "toward" is defined as "afoot"

asking whether water can stay liquid at -10 degrees C at standard pressure

i could think of thousands more questions that would conceivably show a near linear relationship over the groups A-G where most people in A wouldn't answer them correctly. basically it would be most any difficult question where most people in A didn't get it right. (obviously very few would have a perfect linear relationship, but that's the case in the hypothetical OP, and there certainly isn't anything wrong with it happening to be a linear relationship.)

im sure i could come up with questions that would conceivably satisfy the conditions in the OP where Y was actually false, but these would be rare and anamolous questions, like some sort of trick questions where knowledge or IQ act against you in coming up with the right answer. it is a rare instance when being smarter than someone leads you to be incorrect against them.

i don't understand why in this thread or the other thread people aren't just admitting that it is obviously very reasonable to think that Y is probably true. sure there are instances you could think of where it would be false, but these would be rare in comparison to the instances of it being true.

generally, the vast majority of questions where the majority of the A group got it wrong (say it was a very difficult question or it required understanding at great depth in a particular field like probability or vocabulary) would have a similar breakdown of extremely few dumb people getting them right, few average people, more smart people, even more very smart people (even if the number of very smart people were less than a majority). however, very very few questions where the majority of the A group got it right would have that breakdown. so if you answer "no" to david's question, you would only be right in one of these very rare instances.

the answer to david's question is such a clear "yes" that im a bit bewildered by all the controversy and debate it's causing. if you want to say it's unrealistic or situation-specific, fine. but so far as the actual question is concerned, it's a resounding "yes" that Y is probably true for any given question.

[bunny]

[ QUOTE ]
the answer to david's question is such a clear "yes" that im a bit bewildered by all the controversy and debate it's causing. if you want to say it's unrealistic or situation-specific, fine. but so far as the actual question is concerned, it's a resounding "yes" that Y is probably true for any given question.

[/ QUOTE ]
It seems strange to do a survey, discover that most super-geniuses think Y is false and then conclude that it's true because even more geniuses think it is false and an overwhelming number of dumb people agree.

I still dont think it's justified to claim it is probably true based on a meta-trend. I think it's more likely an odd property of Y itself which leads clever people astray.

[FortunaMaximus]

[ QUOTE ]
I think it's more likely an odd property of Y itself which leads clever people astray.

[/ QUOTE ]

Perhaps it's not an odd property per se, but that Y could be an abstract question. If it was a defined statement of fact with an obvious y/n answer, then you could discern the breakdown of the implied reverse logic of less intelligent people being more statistically correct.

Otherwise, I think there exists a phenomenon in where the smarter a class of people are, the more divisive they will be. Intellectual Darwinism, perhaps?

[madnak]

I think this is a judgment call. I'm almost positive there is no logical answer.

[calcbandit]

I think the real interesting question is given the above survey results, is the probability of A being true greater than 30%?

If the survey results were that all groups A-G homogeneously responded 30% yes and 70% no, the probability of Y being true would pretty much have to be set at 30%, barring any other information on the topic.

So, do we have a case here where Y is more likely to be true specifically because less people believe it than the above, even distribution example?

[ZeeJustin]

[ QUOTE ]
So, do we have a case here where Y is more likely to be true specifically because less people believe it than the above, even distribution example?

[/ QUOTE ]

The point isn't how many people believe it, but what kind of people believe it. The people that believe it are those most likely to be correct. If you were to assume the pattern would follow (which of course you cant assume 100%), you could also assume that if there were people out there that were always correct, 100% of them would believe Y to be true.

[ZeeJustin]

[ QUOTE ]
the answer to david's question is such a clear "yes" that im a bit bewildered by all the controversy and debate it's causing. if you want to say it's unrealistic or situation-specific, fine. but so far as the actual question is concerned, it's a resounding "yes" that Y is probably true for any given question.


[/ QUOTE ]

I feel the same way, hence my OP about the atheism intelligence correlation being the #1 reason to believe.

[hitch1978]

I agree with model.

[Borodog]

David,

There is no way to say without knowing what Y is and the backgrounds of the people answering the survey.

Edit: On second thought, I agree with not a model.

Re-edit: On Third thought, maybe I don't.

Re-re-edit: Yes, yes I do. But as soon as you know ANYTHING about Y and the survey group, then the trend can be thrown out the window if necessary.

[Taraz]

[ QUOTE ]
David,

There is no way to say without knowing what Y is and the backgrounds of the people answering the survey.

[/ QUOTE ]

This is what I was trying to get at in the other thread. It depends on what kind of belief Y is.

I don't see why people think that a linear relationship between intelligence and believing in Y is indicative of Y's truth when the overall percentage of believers is still so low. It seems much more plausible that there would be a systematic bias of some sort unless Y is a very technical concept which requires a lot of background knowledge.

[PairTheBoard]

[ QUOTE ]
it seems to me like you are just being captious and overcomplicating matters to try to find a specific instance where the answer is "no" to david's question

[/ QUOTE ]

Actually that's Tom Crowley's job to accuse me of being captious. Ironically, I was merely reiterating the analysis Tom Crowley gave on the other thread for this topic. The irony goes even deeper when you consider that David Sklansky has crowned Tom Crowley King of SMP posters, declaring that Tom Crowley has never made an incorrect post.

It's really not being captious here anyway. Sklanksy himself is not sure if his notion for extrapolating the trend is valid. I also doubt if Sklansky would object to introducing the idea of evidence evaluators estimating a Price on which of the Yes-No answers is correct as a means of deciding which answer to give. In fact his new formulation of the problem in terms of people skilled at getting Yes-No questions correct lends itself very naturally to those people estimating a Price on the answers. Especially since we know nothing about the type of question being asked.

So introducing the concept of Price and a distribution of opinion for that price as the model for the situation is something that really needs to be done in order to argue coherently about the possibilities of what's happening. Once we do that we can see that what you might think must be happening isn't necessarily the case and cannot be the only factor involved in deciding if it's logical to deduce the extrapolation result Sklansky hypothesises. It's not the slam dunk it might be if there were only one logical possible factor possible under Sklansky's assumptions. You can argue further based on guesses for relative sizes of categories of propositions that satisfy one factor or the other. But that argument is much more speculative.

Furthermore, it's not clear that Tom Crowley's model fits all types of propositions. The notion that the distribution for the opinions of Price behaves in some kind of aproximately normal way may not be the case for all types of propositions. There may very well be propositions where double humps can develop in the distribution, which is what Sklanksy envisioned in his original formulation of the problem. Crowley claims this phenomenon is not realistic and implies an irrational model. I'm not so sure about that myself. There may be questions that produce polarity of opinion among even the best human minds but which can still be answered definitively in the limit toward infinite intellegence. If so that would bring in even more categories of propositions for you to guess the relative sizes of.

PairTheBoard

[Piers]

Nature is full of semi linear trends. Trends that are linear for a significant part of their range, but non linear outside that range. This is because curves often look linear over a small portion of their range, ever seen the approximation x~sin(x) for small x.

So when I see a naturally occurring scatter graph that a linear regression will closely fit, my initial assumption is that we are looking at a curve that has a semi linear part rather than a strictly linear relationship. The tendency to assume linear relationships reflects our tendency to simplify complex patterns so they can fit into our limited ability to understand rather than an underlying feature of process being analysed.

[David Sklansky]

[ QUOTE ]
Nature is full of semi linear trends. Trends that are linear for a significant part of their range, but non linear outside that range. This is because curves often look linear over a small portion of their range, ever seen the approximation x~sin(x) for small x.

So when I see a naturally occurring scatter graph that a linear regression will closely fit, my initial assumption is that we are looking at a curve that has a semi linear part rather than a strictly linear relationship. The tendency to assume linear relationships reflects our tendency to simplify complex patterns so they can fit into our limited ability to understand rather than an underlying feature of process being analysed.

[/ QUOTE ]

That's all well and good but it misses the point of this particular question. Even if it was the case that almost everything fits a linear relationship, it is very possible this does not. There is a paradoxical aspect to this problem.

Everybody knows the paradox of the statement "everything I say is a lie". But what of the statemnt "90% of the things I say are a lie"? If that is to be believed and it applies to the statement itself, then what are the odds the speaker has red underwear if he says he does? This isn't of only theoretical concern either. Tom Cowley says that 95% of Pair The Board's comments are moronic. Pair The Board goes on to laud Tom's intelligence. So you can see that these kind of things can give even me headaches.

[Piers]

[ QUOTE ]
Tom Cowley says that 95% of Pair The Board's comments are moronic. Pair The Board goes on to laud Tom's intelligence. So you can see that these kind of things can give even me headaches.


[/ QUOTE ]

/images/graemlins/grin.gif

[PairTheBoard]

[ QUOTE ]
i don't understand why in this thread or the other thread people aren't just admitting that it is obviously very reasonable to think that Y is probably true.

[/ QUOTE ]

[ QUOTE ]
it seems to me like you are just being captious and overcomplicating matters to try to find a specific instance where the answer is "no" to david's question

[/ QUOTE ]

Looking closer at Tom Crowley's model I see that it gives us much more than just a "specific instance" where Sklansky's hypothesis fails. Crowley assumes that there is an aproximately normal type distribution for the opinions of the Price on this proposition being False within each group. He futher assumes that the mean of the distributions moves closer to the correct Price as you move from group G up to group A. He futher assumes that the distributions tighten up with shrinking standard deviations as you move from group G to A and that the distributions would converge to a point mass on the correct Price if you could go to the limit toward infinite intellegence.

Crowley believes this model covers all rational propositions, which I think is debatable. However it surely covers an extremely large category of rational propositions, maybe even in some common sense way nearly all of them. Common sense tells me that relatively few rational propositions produce double bump polarities of opinions among human experts.

So focusing on Crowley's common sense model and large category of propositions for which it fits, we ask how is Sklansky's scenario possible? There are two ways. Under Crowley's model, Sklansky's scenario logically implies that the G group is overestimating the Price on "False". The true Price on False may be <50% or >50%. In either case, as you move up the scale from group G to group A you would expect to see the lower tail of distributions for opinions to shift down into the <50% area. The question is, how fast does the distribution simultaneously tighten up? If it does so slowly we get Sklansky's scenario even when the true price for False is >50%. In the other case, we always get Sklanky's scenario when the true price is <50%.

Now think about it. How big is the category of propositions fitting Crowley's model for which G people answer No and where they are right? Out of that category, how likely is it that as we move from group G to A the mean of distributions for opinions on the Price of False moves more rapidly toward the correct price >50% than the Tightness of the distribution converges? What type of propositions might affect this relative speed of movement of mean and shrinkage of tightness for the distributions?

Maybe it's anybody's guess. But my guess is that it's a relatively large subcategory. Why? Because when there is unanimity among the G group as in Sklanky's scenario they can easily be Extreme in their overestimation of the Price for False, even when the evidence for the Price is difficult to evalutate. For such cases I would expect experts to quickly dismiss the extremist bias of lower groups, moving the mean of their distribution relatively quickly closer to the true Price. If the evidence is difficult to evalute the tightness of their distributions may not shrink so quickly. This defines a very large category of propositions satisfying Sklansky's scenario but for which his extrapolation hypothesis fails.

Remember, the subcategory above comes out of what should be the common sense much larger of the two categories. Certainly the vast majority of propositions for which G group people answer No with 100% unanimity are actually False. In fact, this is part of the reason why they can have exteme opinions overestimating the true price for False based on the evidence.

So Crowley's model does much more than just offer one counter example. It indicates what I think is a good case for the subcategory of propositions which satisfy his model, satisfy Sklansky's scenario, and are counter to Sklansky's hypothesis - and what you think is "obvious" - is indeed the common sense largest category of propositions under consideration. Knowing nothing else about the proposition, the better conclusion under Sklansky's assumptions, if you insist on making one, is that the propostion is "probably" False, just like everybody in the G group thinks.

PairTheBoard

[TomCowley]

I didn't think that normal-ish was the only possible distribution. Double-humps are plausible if there is one iffy piece of evidence that has two plausible evaluations, one of which ends up with a price of "about 70%", and the other "about 40%". Everybody is pretty confident that one of these two answers is correct, but is only slightly confident in their own answer- my specific objection was to large double-humped groups with high confidence.

Once you postulate wacky distributions, you're basically assuming the extrapolation isn't valid, which isn't necessarily wrong, but it's not an interesting result. I wanted to start with the friendliest assumptions to see if the extrapolation worked in those circumstances, so I could see what conditions were necessary to make it true, but it didn't work even then.

That being said, if the trend is 5% up to 30% with increasing IQ (yes/no skill), then I think it's safe to rate Y a higher probability to be true than if the trend is 30% down to 5%. That's also justifiable by simply giving smarter people more weight and not extrapolating. I'm not sure about the 5%-30% trend case compared to other cases that end up at 30% (flat 30%, decreasing to 30%, etc). As much "intuitive" sense as it makes to extend the line, if I want to set a price without knowing anything specific about Y, I have yet to come up with any justification for putting the probability of Y being true far away from 30% in any of those cases.

I'm also not convinced that it's even meaningful to discuss a price here, since this has a lot of similarity to the bent coin/personal probability stuff (that I really don't want to rehash), but if I had to set a price to prop-bet somebody with the same information, I haven't yet found a justification for it being far away from 30% (possibly a little different based on some form of weighted averaging across the groups, but not based on extrapolation).

[gaming_mouse]

[ QUOTE ]
Say people are carefully evaluated and are classified by how likely they are to get things right on yes or no questions. They are rated from A to G. A's are historically the most likely to get things right.

[/ QUOTE ]

Who determines what a valid "yes or no" question is, and what the correct answer is?

It seems ironic that the model being used here implicitly is one of an omniscient being, a Great Teacher who can keep score of the quizzes administered to his imperfect students -- a great mass human beings of varying skill levels.

Without such a scorekeeper, it is human beings who must determine the answers to these yes or no questions. But now the argument becomes circular.

I think all the confusion surrounding the orginal question stems from the flaw in this basic assumption: that we can coherently talk, in some general sense, about one's skill at answering a yes or no question.

[ZeeJustin]

[ QUOTE ]
I think all the confusion surrounding the orginal question stems from the flaw in this basic assumption: that we can coherently talk, in some general sense, about one's skill at answering a yes or no question.

[/ QUOTE ]

It's trivially easy to "talk, in some general sense, about one's skill at answering a yes or no question".

Why would you think otherwise?

[gaming_mouse]

[ QUOTE ]
[ QUOTE ]
I think all the confusion surrounding the orginal question stems from the flaw in this basic assumption: that we can coherently talk, in some general sense, about one's skill at answering a yes or no question.

[/ QUOTE ]

It's trivially easy to "talk, in some general sense, about one's skill at answering a yes or no question".

Why would you think otherwise?

[/ QUOTE ]

Maybe I wasn't being clear. You can obviously talk about people being able to answer yes or no questions about arithmetic, or about things such as the freezing point of water, or about the capitals of different countries, etc. But the point is that all of these questions are admissible precisely because, as a community, we overwhelmingly agree on the correct answer.

But you can't now start talking about something like "the complete set of yes or no questions." That is what I meant by "general." And I think it is an assumption of the original question that David asked that you can meaningfully talk about the complete set of yes or no questions. I think I explained why I believe you cannot do this in my first post, but if it's still not clear I can try to elaborate.

[madnak]

If each yes or no question has a correct answer, independently, then we can talk about skill in answering such questions. This is an epistemological question. I think David is assuming for the purpose of argument that at least some questions have answers.

[gaming_mouse]

[ QUOTE ]
If each yes or no question has a correct answer, independently, then we can talk about skill in answering such questions. This is an epistemological question.

[/ QUOTE ]

You are right that it is an epistemological question, and that's my point. Your assumption that every yes or no question has a correct answer is a philisophical, even a religious, one. You might like to say we humans don't always know the answer, but still it must exist. But that is just something you take on faith (hence the irony I mentioned in my first post). But it just doesn't make sense to talk about the correct answer to many yes or no questions. As with other questions of faith (like the existence of God), people won't agree on the methods of proof.


[ QUOTE ]
I think David is assuming for the purpose of argument that at least some questions have answers.

[/ QUOTE ]

He is assuming much more than that. He's assuming that we can talk about a thing called "one's skill at answering yes or no questions," and further assuming that this phrase has a meaning beyond the ability to answer yes or no questions whose answers are part of the agreed upon body of human knowledge.

[madnak]

Yes, but only for the purpose of this hypothetical.

[gaming_mouse]

[ QUOTE ]
Yes, but only for the purpose of this hypothetical.

[/ QUOTE ]

Okay, he may not really believe that. But my point is that the hypothetical is founded on this nonsensical assumption. That is, I know what he is getting at with the original question, obviously. But I'm saying that, if you really think about it, the question is, to borrow a term from mathematics, ill-defined.