• TalkBass has been independent since 1998. Add your voice.
    Create a free account to reply to discussions, view embedded media, and browse with fewer display ads.
    Join freeLog in
    Want zero display ads or expanded classifieds tools? Compare plans.

Statistics question

I wish to apologise to Kjung and any one who is genuinely trying to make the world a better place. I made an attack on psychology generally as a science which sometimes presents itself as superior to sociology which often is under pressure to prove itself as a legitimate science due to popular perception of it as a bit too flakey. This applies especially to pure sociology with its potentially too subjective 'participant' observation techniques. Obviously generalisations from any perspective or discipline are problematic as are extreme positions such as I most likely am guilty of. Such views are important though to maintain a balance somewhere in the middle. I am merely trying to encourage debate so that people who havn't had the luxury of a university education may start to challenge the so called facts often presented by those who intend to decieve them into voting a particular way or buying a product that may not be all it's cracked up to be. As you already agreed that plenty of that goes on Kjung, I am sure we are singing from the same sheet. Philosophy probably does belong in letters but it is also an essential part of sociology and politics. I think a degree of suspicion and cynicism is healthy particularly when dealing with statistics if only because it is too easy to take them at face value and not ask why or who did the research.Science has improved our world in many ways but it sometimes comes at a price. Only a fool believes everything they read.

Nah, it is a fun discussion:). No need to apologize! Strongly held opinions based on experience and training from the opposite sides of the aisle on a topic usually results in interesting discussions, as long as everyone stays polite.

Carry on! I obviously love the topic, and do quite a bit of 'history of science' reading in my spare time, which covers quite a bit of both the good, and the bad that you mention.

FYI, I am currently re-reading Daniel Boorstin's 'The Discoverers', which is a single volume romp through the whole concept of 'human discovery', from 'time', to geographic expansion, to language, to printing, to medicine, to measurement, etc. Wonderful read that takes a pretty nice 'neutral' approach, and is one of the rare 'history of science' books that spends quite a bit of time on the Eastern cultures (primarily China).
 
Wow, a lot of good discussion here. I'm in a bit of a hurry, but I'd like to comment on a few things. First, I definitely agree that there are some issues with how results are presented in journals. This is a problem not only because of the pressure to publish in R1 institutes, but also because of how concise journals want scholars to write their articles. Pages are at a premium in journals, and a lot of things get cut out that could help decision making with vetting methodology. I'm a huge advocate of that statistical analyses in journals need to be given a "Criticism" section. We may never know that people are reporting significant ANOVA findings, but they never told us whether or not the groups were equivalent. It's even worse for regression, which requires a lot of vetting before you can be confident about its validity. I bet there are zillions of regression reports being passed off as valid, but have multicollinearity throughout them. Of course, I doubt anyone in power is gonna take seriously my, "We need regression criticism sections in journal articles" plea. :D

Yes, grant money may have political agendas attached to it. However, not all research completed at a university is grant-funded. Much of it is basic research, and is just related to the scholars' muses. With that said, I'll always automatically trust university research more than research that comes out of a politically-oriented think tank. The latter is almost always going to be slanted to a specific political leaning and it's often conducted by hacks, some who may have never even went to grad school. At the end of the day, however, regardless of where research is conducted, whether it's grant funded, and whether it has a political slant, you have to just vet the methodology, the theory, and the analysis, and judge it on its own terms.

rubadubdub, if you haven't, read the best policy book ever written, Policy Paradox by Deborah Stone. She talks about a lot of the things you are talking about.

I don't know if quanitative sociology is really falling by the wayside. I can only speak to my department, which is pretty much split down the middle between postpostivist and quantitative people and constructivist and qualitative people. As I said earlier, whereas psychology has established itself as a postpositivist field, sociology will probably forever be in an ontological existentialist crisis. But I think that's really a function of the scope of items of investigatory interest that permeate sociology. Coming from a psych background, I've always been an empirical, quanitative guy, but my dissertation is qualitative, because, well that's the paradigm I have to work from to answer my research question. Positivism and quantitative analysis is simply not equipped to deal with my question. With that said, I generally don't consider qualitative research "science," because, I consider science to be primarily interested in prediction and replication. Ethography and phenomenology really aren't interested in these things. That is in no way to denigrate those methods of inquiry. They are very valid epistemological tools for helping us understand the world. I just don't think of them as scientific.

Ironically, I'm actually an antipostivist, even though I'm a quantitative guy. I see the social world as existing as a pure social construction, but through that construction, different patterns can emerge. I'm also neither a humanist nor an anti-humanist on epistemological grounds. What I mean is that the question of whether or not a such thing as "human nature" exists is technically a falsifiable question. However, to answer that question, you'd practically have to raise people in Skinner boxes to control for all of the confounding variables, which raises both methodological and ethical issues.

Agreed that SPSS and Systat aren't really geared towards large data sets. Being ANOVA-based, they are geared more towards psychologists, who are probably dealing with n sizes that are on the relatively smaller scale. One thing I never liked about SPSS and Systat is the output page, which is rather unituitive. I like Stata, because I can just punch in commands down at the bottom and then have my output pop up right there in a nice, neat package. I also prefer Stata's graphical tools. It's a lot easier to look for heteroskedasticity in output and the such with Stata. Just my two cents on the matter. I've never used SAS, although I have used the programming part of SPSS before. Not a fan and totally unintuitive to anyone who has done programming in any other capacity.
 
Thanks for the further reading tips Kjung and liquidmidnight. I am well out of my depth here and I will confess that I am probably a little oversensitive. It probably has something to do with an economic and social history professor telling us that we would score higher marks for essays submitted to that department than we would score for those submitted to our own sociology department due to the fact that at the time Leicester Universities sociology dept. was under pressure to assert it's credibilty as a serious acadamic establishment or something like that. All I can tell you with confidence is that some of us did find that to be the case. Maybe it was easier to score firsts in economic and social history or maybe a few of us read the wrong subject. Whatever, my real downfall is my inability to cope with maths. As to why .... I wont bore you with that. Respect to you and keep up the good work.
 
Yes, grant money may have political agendas attached to it. However, not all research completed at a university is grant-funded. Much of it is basic research, and is just related to the scholars' muses.

The bad news is that "higher education reform" movements that are gaining alarming amounts of traction seek to eliminate precisely this type of research. Google "Texas Public Policy Foundation" for an example. The belief is that commercially supported research is the only good research, and any other research has no place in universities--those types of "lazy perfessors" should be teaching instead. For the past century the US higher educational system has generally been viewed as the best in the world, but it's becoming a political football and I worry about future of scholarship in these parts.
 
The bad news is that "higher education reform" movements that are gaining alarming amounts of traction seek to eliminate precisely this type of research. Google "Texas Public Policy Foundation" for an example. The belief is that commercially supported research is the only good research, and any other research has no place in universities--those types of "lazy perfessors" should be teaching instead. For the past century the US higher educational system has generally been viewed as the best in the world, but it's becoming a political football and I worry about future of scholarship in these parts.

That's definitely bad news, Rick. Many people don't understand that the important applied research that cures diseasese and finds a way to build a better mousetrap almost always starts off in the humble world of basic research, where the eventual application of that knowledge may be fuzzy at that point.

Yes, there are people researching the correlation between toe nail lenght and hours spent watching Seinfeld or whatever. :p But a lot of basic research that doesn't seem "important," actually leads to some astouding insights.

Of course, I can't say I'm surprised by this. There is a big anti-intellectual atmosphere in America.
 
If a quiz with two choices to identify A from B results in 73% correct and 27% incorrect what does that tell us about the identifyable properties?

It is complicated by the fact that a group of monkeys would get 50% by guesswork alone.

Rigorous statistical analysis please. Assume a sufficiently large sample size for Joe Bloggs to be satisfied it is reliable, whatever that is.

When you figure it out please join in here http://www.talkbass.com/forum/f15/necessary-have-full-tube-amp-959702/index20.html#post13951773
 
If a quiz with two choices to identify A from B results in 73% correct and 27% incorrect what does that tell us about the identifyable properties?

You need one more piece of information: Were people allowed to see the earlier results when they took the quiz? If so, they might have been biased by the choices made by the first few people, so they were not all independent samples. This is typically handled by reducing the effective number of participants.

With that said, if the result is statistically significant and robust upon repetition, it tells us that there are identifiable properties, but it doesn't tell us what those properties are. For instance, if the choices are two brands of coffee, we don't know which chemical compounds in the two brands are responsible for the test results unless we absolutely know that we are measuring the effect of a single variable.

A remarkable number of studies that get breathlessly reported in the press are never repeated, or are overturned at a later date. Because the press (including professional journals) favors positive results over negative ones, popular beliefs fueled by bad science can persist over long time periods.
 
Nah, it is a fun discussion:). No need to apologize! Strongly held opinions based on experience and training from the opposite sides of the aisle on a topic usually results in interesting discussions, as long as everyone stays polite.

Carry on! I obviously love the topic, and do quite a bit of 'history of science' reading in my spare time, which covers quite a bit of both the good, and the bad that you mention.

FYI, I am currently re-reading Daniel Boorstin's 'The Discoverers', which is a single volume romp through the whole concept of 'human discovery', from 'time', to geographic expansion, to language, to printing, to medicine, to measurement, etc. Wonderful read that takes a pretty nice 'neutral' approach, and is one of the rare 'history of science' books that spends quite a bit of time on the Eastern cultures (primarily China).

Ben Goldacre's 'Bad Science' should go to the top of your 'must read' list.
 
You need one more piece of information: Were people allowed to see the earlier results when they took the quiz? If so, they might have been biased by the choices made by the first few people, so they were not all independent samples. This is typically handled by reducing the effective number of participants.
I think the results are viewable before voting, can't be sure. Let's assume they weren't, or if they were, the respondents were responsible types who made up their own minds, since group think is beneath bassists. <cough>
 
I think the results are viewable before voting, can't be sure. Let's assume they weren't, or if they were, the respondents were responsible types who made up their own minds, since group think is beneath bassists. <cough>

JimmyM ran a couple of similar polls, may 2 or 3 years ago. The results at that time were negative.

I test gear for a living. Not amps, but scientific instruments. We are always running tests to see how changes to our products affect the results that our customers get. I can tell you that getting a conclusive answer is hard, because there is always the danger of introducing uncontrolled variables. Then you don't know if it's your controlled variables, or your uncontrolled ones, that influenced your results.

I really appreciate that JimmyM has run these polls. It's one of the more interesting and enjoyable ideas that's come across TB since I've joined. But turning those polls into actual controlled tests, where conclusive results would actually teach us something, will require taking things to another level.

By the way, at the place where I work, we were using 20 bit A/D converters more than 20 years ago, and we had solid data that demonstrated the benefit of the extra 4 bits.
 
If a quiz with two choices to identify A from B results in 73% correct and 27% incorrect what does that tell us about the identifyable properties?

It is complicated by the fact that a group of monkeys would get 50% by guesswork alone.

Rigorous statistical analysis please. Assume a sufficiently large sample size for Joe Bloggs to be satisfied it is reliable, whatever that is.

When you figure it out please join in here http://www.talkbass.com/forum/f15/necessary-have-full-tube-amp-959702/index20.html#post13951773

This is an example of the use of the binomial distribution. If we take the "monkey" hypothesis that they'll be right 50% of the time, what are the odds of seeing 73% right? The answer depends upon the sample size. Here is a link to a binomial calculator:

http://www.talkbass.com/forum/f15/necessary-have-full-tube-amp-959702/index20.html#post13951773

So, if you plug in .50 as the chance probability, and 73 observations of 100 as the observed probability, you'll see that it is very unlikely (theoretically almost impossible) that you would observe a value of 73% or more by chance alone. However, smaller sample sizes lead to much less confidence in that conclusion.
 
My computing power is presently very limited.

Without looking up the thread let's assume there were 200 respondents.

My query boils down to: "What proportion of responders need to be truely skilled in identifying A vs B for a 73% correct result to be acheived?"

Please leave aside the possibilty that responders may have been influenced by the trend before they joined the poll.
 
My computing power is presently very limited.

Without looking up the thread let's assume there were 200 respondents.

My query boils down to: "What proportion of responders need to be truely skilled in identifying A vs B for a 73% correct result to be acheived?"

Please leave aside the possibilty that responders may have been influenced by the trend before they joined the poll.

The calculators online don't take any more computing power than talkbass does! But anyhow, your query has to be qualified by confidence level, etc. I think the issue is, given that people can be right 50% of the time by chance alone, does observing them being right 73% of the time mean that there is a true skill there? And the answer is, yes. If you flipped a balanced coin 200 times, you would almost never get as many as 146 "heads"....the percentage of times that it would be fewer seems to be 99.999999971629%, looking at the binomial results.
 
I teach statistics at a graduate school (actually going to teach in two hours), but it seems the thread has veered away from the OP's question, although sounds sorted out already. fdeck seemed to describe the Central Limit Theorem as I would. As Ziltoid mentioned, a normal distribution does not follow from ordinal data, such as surveys on agreement/liking/rating scales as political surveys often are. I chose to focus on political science and social psychology myself rather than sociology...but don't feel like jumping into the sociology discussion; the other theoretical perspectives just suited me (and my interests) better.

FYI, I am currently re-reading Daniel Boorstin's 'The Discoverers', which is a single volume romp through the whole concept of 'human discovery', from 'time', to geographic expansion, to language, to printing, to medicine, to measurement, etc. Wonderful read that takes a pretty nice 'neutral' approach, and is one of the rare 'history of science' books that spends quite a bit of time on the Eastern cultures (primarily China).

Great book! Still have my copy...not going to reread it all though!
 
The calculators online don't take any more computing power than talkbass does! But anyhow, your query has to be qualified by confidence level, etc. I think the issue is, given that people can be right 50% of the time by chance alone, does observing them being right 73% of the time mean that there is a true skill there? And the answer is, yes. If you flipped a balanced coin 200 times, you would almost never get as many as 146 "heads"....the percentage of times that it would be fewer seems to be 99.999999971629%, looking at the binomial results.
My internet PC is having problems, my cellphone computing is limited, gimme a break.

What I would like to know is what percentage of respondents are expected to have skill, given a 200 sample and a 73% "accuracy", given a 50% sucessful guessing rate. I don't think that's a difficult proposition for a stats teacher. Make it 100 respondents if you like.
 
What I would like to know is what percentage of respondents are expected to have skill, given a 200 sample and a 73% "accuracy", given a 50% sucessful guessing rate. I don't think that's a difficult proposition for a stats teacher. Make it 100 respondents if you like.

That's not really a "statistical" question, but if chance is 50% and observed is 73%, then the estimate of "skill" in that sample that I would use would be (.73-.50)/(1-.50) = 46% of the sample really had the "skill", rather than being correct due to chance....that would be chance-corrected item difficulty.