Experts, studies, etc.

[Bob Lee (QSC)]

Expertise is great (it's certainly better than ignorance and superstition), but don't bet the farm …
http://news.yahoo.com/s/time/2010062...08599199864400

[i_got_a_mohawk]

That guy sounds like a bit of a douche. Going to take his advice on not trusting what his book says tho.

[jmattbassplaya]

I`d like to know where he gets his facts from.

[biobass]

As a working scientist, my job is to be a professional skeptic. Rational skepticism is an integral part of critical thinking, however it has to be tempered with the acknowledgement of one's own limits of factual knowledge and nuanced understanding of a particular field.

Cheers,
Dennis

[LiquidMidnight]

I know this is going to sound incredibly elitist, but the the lay public really has no idea how scientific research actually works and it often infuriates me to no end when someone who doesn't really understand research tries to put forth a valid opinion on the matter.

One thing many people fail to realize is that a scientific paradigm is built upon multiple studies from a myriad of scholars, each whose research interest is generally very narrow; however, each scholar's studies add up and shape the field and area of research. One study alone in any scientific field does not confirm or refute a theory or phenomenon. Which takes me to my next point...

People want reductionist answers to complex phenomena and think the science is bad if it doesn't provide said reductionist answers. Science rarely provides reductionist answers for a complicated world.

Often when it comes to social sciences, people always say, "Well, that study's false because my cousin Billy Bob Joe Bob Eddie Bob Jr. does not fall into that category" when they don't understand how concepts such as inferential statistics work and external validity really work.

Also, people have this view of science as being strictly empirical and you either witness a phenomenon or you don't. Science has been and will always be intimately interwined with philosophy, and many people don't realize how much inductive and deductive reasoning is used in science.

And perhaps most importantly, many people get their scientific information from popular sources. First, popular sources often have the word "prove" written in them, which is a word that any respectable scientist is not going to have in his or her venacular. A popular source is not going to have an in-depth discussion about how the study fits in with the other research, how the internal validity could have been improved, or what was the possibility of a Type I error. These are the types of things that need to be considered when considering research.

[i_got_a_mohawk]

Too true.

Things like this do tend to irk me, I'm guessing you guys find it to be the same way?

(I'm working on my PhD (in Physics) right now)

[biobass]

Liquidmidnight, well said! It argues for more and better science teaching and understanding in our schools and homes. Our complex and technological society depends on a true understanding of how science operates.

Dennis

[EBodious]

boy, this agrees with my perspective. is that why i think he is right? my experience with tax professionals is exactly what he mentioned. and for me, a red flag goes up when people start quoting statistics. i hadn't considered the idea that studies get refuted and we often don't hear about it.

i remember an article i read in 1990. two guys did their dissertation on proving that copernicus had fudged his data. he could not prove the planets went around the sun with the data available at the time, but, it really seemed like the right answer to him. i think his intuition was correct. but he fudged his data.

i struggle with the notion in a practical way, but in my gut, i believe the buddhists are correct: reality is subjective.

Originally Posted by i_got_a_mohawk

That guy sounds like a bit of a douche. Going to take his advice on not trusting what his book says tho.

ok, i gotta admit i think its pretty funny that you think "douche," i read it and thought, "right on." to me, this aligns with my thinking (hence, my agreement). this furthers my belief that ultimately, people will support the "truth" that works for them. we all have a world perspective based on something that makes us comfortable for various reasons. i know people who are too attached to new agey spiritualism (in my opinion) and those who are too attached to scientific materialism (again, in my opinion). i don't believe there are any single right answers, which is sort of his point. i do think that we start out seeing the world as black and white, and that as we get older, wisdom becomes realizing that everything is shades of gray.

also, please know that i am not meaning to single you out. i like your posts and value your point of view. i am just remembering our vaccination debate. this guy reflects the perspective i was coming from, so i thought it was funny that you didn't like it. i hope you take no offense. i find everything i learn to be fun, and try not to get to attached.

[i_got_a_mohawk]

He's suggesting that 66% of studies are refuted within years? I'd like to know where he got that little nugget for a start. Studies can be refuted, a prime and somewhat famous one being the 'legit' study that linked vaccinations with autism (to bring back that old topic ). I mean, science does work by disproving something, but to suggest that the majority of studies are, is just false IME. I think he is trying to fudge the basic premise of science.

When a paper is finally published (depending on the journal, you'll get hefty scrutiny from peers and other experts), other people working in that field read it, and can adapt research around that paper or try and advance from the point made in that paper. Advancing you can find that some rationalisations may not be entirely correct and the field moves on.

I'd say it is common for parts of a study to possibly be disproved down the line, but that is all part of how science advances. But the author has worded it (IMO) to make it sound as if our entire understanding changes, when in fact it is fine tweaks (generally) as opposed to stating that something is unequivocally wrong. If you follow what I'm getting at?

For what it matters, I've never been religious. It's something which tends to inhibit questions. It's something that tells you what to do, with no logical reasoning. The difference with research, experts in their fields, if you doubt a point they are making, or research they have undertaken, question them. Ask them why they haven't tried X, Y, Z, ask them where they have adapted something, ask why they didn't elaborate on A, B, C, ask them why their findings don't seem to fit with what Group A reported.

And no, you're alright bud, no offence taken

It just bugs me when people seem to forget that others devote their entire lives to further the knowledge of society. And those people tend to not like it when a simple answer can't be presented. They seem to loose sight of the fact that a true expert in the field has 20,30,40+ years of experience and knowledge, and you can't simply condense than down into lay terms perfectly. Yet, celebrities can come along (going back to the vaccine one), say A causes B and people will listen to them, because they said so and, as you say, that supports their idea. While there is nothing wrong with an idea, attempting to make it out to be sound and true while having no support for it, is a dangerous thing to do.

IMO if people are to have faith, they should have it in each other. Sure, that can be hard a lot of the time. But for the best part the scientific community is looking for people to question things, adapt theories and work towards furthering knowledge. People in general (not just in research, not just experts) are wanting to work to try and help make things easier, better, (however you want to call it) for others.


[this is probably ranty as hell and full of holes, but I'm overdue for some kip , g'night! ]

[LiquidMidnight]

Originally Posted by EBodious

a red flag goes up when people start quoting statistics.

Again, because most people don't really understand statistical analysis (both the lay people throwing out stats to support their view point and the lay people trying to interpret them), particularly the difference between descriptive and inferential statistics. A quoted percentage is not necessarily incorrect; however, many people fall into the trap of believing that a percentage is an inferential stat and explans some sort of causality. When it doesn't adhere to their misinformed view of statistics, they cry foul and say that statistics are a bunch of BS.

Also, people often don't understand the difference between statistics, which are mathematical computations performed on samples, and parameters, which are mathematical computations performed on populations. This comes back to people's ignorance on the concept of external validity, which is how well a finding regarding a sample is representative of the population. Just because a statistical test is "significant" (i.e., the finding is not likely due to chance), it does not necessarily mean the sample is representative of another population.

Also, analyzing a statistic is so much more than a single number or knowing that the stat is significant. A person with an understanding of statistics could take an ANOVA, for example, look at the F score, look at the alpha level, look at the degrees of freedom, look at the effect size, look at the post-hoc test, consider the study's internal validity, and then make a cogent determination of whether the stat is good or worthless.

i struggle with the notion in a practical way, but in my gut, i believe the buddhists are correct: reality is subjective.

ok, i gotta admit i think its pretty funny that you think "douche," i read it and thought, "right on." to me, this aligns with my thinking (hence, my agreement). this furthers my belief that ultimately, people will support the "truth" that works for them. we all have a world perspective based on something that makes us comfortable for various reasons. i know people who are too attached to new agey spiritualism (in my opinion) and those who are too attached to scientific materialism (again, in my opinion). i don't believe there are any single right answers, which is sort of his point. i do think that we start out seeing the world as black and white, and that as we get older, wisdom becomes realizing that everything is shades of gray.

Again, I think your view of science is limited, and I don't mean that in a condesending way. Theories are not holistic constructs that shape our reality. They are just abstract explantations of verifiable phenomena. True, science does strive for parsimony (i.e., Occam's Razor), but again, reductionism and science rarely go together at the macro level. Take psychology, for example: One psychological phenomenon could be explained by a theory in social psychology, a theory in clinical psychology, a theory in evolutionary psychology, a theory in psycho-dynamic psychology, and a theory in cognitive psychology, after which, it is applied in the real world by a community psychologist or an I/O psychologist. All of these theories are different and come from different perspectives, and none of them are necessarily "incorrect." Even in the physical science realm, modern quantam physics has challenged people's beliefs about what constitutes objective reality.

Also on that note, it should be stated that some sciences have strived to be more objective and positivist, such as psychology and much of the physical sciences. Some sciences have gone a much more subjective and antipositivist route, such as anthropology. Yet some sciences, such as sociology, are both positivist and antipositivist.

[Deluge Of Sound]

Originally Posted by LiquidMidnight

People want reductionist answers to complex phenomena and think the science is bad if it doesn't provide said reductionist answers. Science rarely provides reductionist answers for a complicated world.

Sounds very similar to:

Originally Posted by OP's Article

What have you learned about bad advice?
Bad advice tends to be simplistic. It tends to be definite, universal and certain. But, of course, that's the advice we love to hear. The best advice tends to be less certain - those researchers who say, 'I think maybe this is true in certain situations for some people.' We should avoid the kind of advice that tends to resonate the most - it's exciting, it's a breakthrough, it's going to solve your problems - and instead look at the advice that embraces complexity and uncertainty.

But it's not really natural to take less-certain studies and advice seriously, is it?

You're exactly right, and that's part of the problem. It goes against our intuition, but we have to learn to force ourselves to accept, understand and even embrace that we live in a complex, very messy, very uncertain world.

[popinfresh]

Because economic depression isn't by chance, much with many other studies that get refuted such as pharmaceutical.

People love to study repetition and recurring patters, instead of thinking of something new.

[coreyfyfe]

Originally Posted by i_got_a_mohawk

He's suggesting that 66% of studies are refuted within years? I'd like to know where he got that little nugget for a start. Studies can be refuted, a prime and somewhat famous one being the 'legit' study that linked vaccinations with autism (to bring back that old topic ). I mean, science does work by disproving something, but to suggest that the majority of studies are, is just false IME. I think he is trying to fudge the basic premise of science.

He qualifies the statement by saying medical studies. IME, large scale medical studies, especially those which involve collecting data from diverse clinical locations and attempting to fit highly varying degrees of disease progress and/or treatment success/failure rates into a logical conclusion with a simple and eye catching headline are probably those that make the most impact on this statistic. And while I may agree with the idea that these types of studies are rife with error, it's not necessarily because of bad science, but more because of trying to fit a conclusion based on limited understanding of complex treatment responses or highly variable patient outcomes.

Overall, I agree with the others who have said that this guy seems to be talking out his back side in order to sell books. The basic premise of his book is that we're supposed to trust him when he tells us not to trust anyone because his experts know what they're doing when they criticize other experts.

[Number27]

dood, that guy totally stole my idea! i was going to write a book on the same things but my working title was "everone is wrong... except for me"

[fdeck]

In my view, just reading that blurb, the author is being sensationalistic. Also, it is misleading to lump all expert knowledge together, because different fields have achieved different levels of progress. Some cases in point:

1. There are theories in the sciences, such as relativity, electromagnetism, quantum mechanics, and evolution, that are remarkably coherent and successful.

2. In some fields such as theology and philosophy, profoundly contradictory theories sit side by side with no way of reconciling them.

3. In some fields such as business and medicine, experts have to act with urgency on an often ambiguous or contradictory set of information about the situation.

4. I daresay that some fields attract charlatans and produce continuous drivel.

5. Financial bubbles, where it turns out that all of the experts are colossally wrong.

6. Popularizations often butcher the results that they claim to be based on.

[i_got_a_mohawk]

Originally Posted by coreyfyfe

He qualifies the statement by saying medical studies. IME, large scale medical studies, especially those which involve collecting data from diverse clinical locations and attempting to fit highly varying degrees of disease progress and/or treatment success/failure rates into a logical conclusion with a simple and eye catching headline are probably those that make the most impact on this statistic. And while I may agree with the idea that these types of studies are rife with error, it's not necessarily because of bad science, but more because of trying to fit a conclusion based on limited understanding of complex treatment responses or highly variable patient outcomes.

Even still, it comes over (the way I read it) that he is trying to make out that the vast majority are completelly wrong as opposed to making out that it is making steps in the right direction.

I'm actually pretty surprised that my rant was even semi coherent

[coreyfyfe]

Originally Posted by i_got_a_mohawk

Even still, it comes over (the way I read it) that he is trying to make out that the vast majority are completelly wrong as opposed to making out that it is making steps in the right direction.

That was kind of the point I was trying to make. His interpretation of a study being "incorrect" is in fact the error here. While a medical study (or any other study for that matter) may come to one conclusion, it is the nature of research to wade through the conclusions made in prior pieces to uncover the next layer of questions. If a study makes conclusions which turn out to be false, but in the end lead to better answers and expanded knowledge, then that study is not invalid. And I think he's missing that point.

And for all the research that is published, there are very few controversial retractions that need to be made, and many of those were suspect to begin with (nanobacteria, nuclear calcium partitioning, etc.).

[i_got_a_mohawk]

Sorry, wasn't meaning to come over as questioning, just that I was agreeing with your point. I need more coffee!

[XtreO]

- You say that many experts are wrong, yet you quote many experts in your book. Are these experts wrong too?

They very well may be, but these are people who study expertise. They know how other experts go wrong because this is what they study, so maybe they're better at avoiding some of these problems. Maybe they're a little more careful with their data and they work a little harder to not mislead people. That's just a suggestion. I mean, who knows? But that's the best I can do to defend myself here.



LOL

[Jonyak]

As someone who works with science day to day I approve this post.

Originally Posted by LiquidMidnight

I know this is going to sound incredibly elitist, but the the lay public really has no idea how scientific research actually works and it often infuriates me to no end when someone who doesn't really understand research tries to put forth a valid opinion on the matter.

One thing many people fail to realize is that a scientific paradigm is built upon multiple studies from a myriad of scholars, each whose research interest is generally very narrow; however, each scholar's studies add up and shape the field and area of research. One study alone in any scientific field does not confirm or refute a theory or phenomenon. Which takes me to my next point...

People want reductionist answers to complex phenomena and think the science is bad if it doesn't provide said reductionist answers. Science rarely provides reductionist answers for a complicated world.

Often when it comes to social sciences, people always say, "Well, that study's false because my cousin Billy Bob Joe Bob Eddie Bob Jr. does not fall into that category" when they don't understand how concepts such as inferential statistics work and external validity really work.

Also, people have this view of science as being strictly empirical and you either witness a phenomenon or you don't. Science has been and will always be intimately interwined with philosophy, and many people don't realize how much inductive and deductive reasoning is used in science.

And perhaps most importantly, many people get their scientific information from popular sources. First, popular sources often have the word "prove" written in them, which is a word that any respectable scientist is not going to have in his or her venacular. A popular source is not going to have an in-depth discussion about how the study fits in with the other research, how the internal validity could have been improved, or what was the possibility of a Type I error. These are the types of things that need to be considered when considering research.