Is the review process broken?

[DougD]

Not pointing fingers at anyone in particular. I'm just wondring what people REALY think about the reviews they read. It seems to me that reviews have become another form of advertising instead of an honest evaluation of a product. What do you all think?

[Phalex]

User reviews are usually written by people that have just bought a new piece of equipment and they are usually quite enamored with it at the time of their review. I don't believe they are very objective, but I do still find them useful.

Reviews from magazines and such are probably less than objective as well. If you go on record as saying that the latest offering from the Fender company is a POS they will likely make your life as rough as they can as payment for your candor.

[pointbass]

I agree that most initial reviews are, to at least some degree, inaccurate and almost always a reflection of buyer euphoria. I usually like to read reviews by people who have been using the gear for at least 3 months or longer, that's when the real story comes out. I personally try to wait at least that long before putting out a review, and then I'll go back after a year and re-read what I wrote, just to be sure it's accurate.

I love reading reviews of gear that someone has had for 2 days .... everything is always great and it's the best item ever built and they'll have it forever and ever ..... until a month down the road when it's on the TB FS forum

[DougD]

Originally Posted by pointbass

I love reading reviews of gear that someone has had for 2 days .... everything is always great and it's the best item ever built and they'll have it forever and ever ..... until a month down the road when it's on the TB FS forum

Good Point. But what realy gets me is when someone writes a review on something they dont even own. If you go over to Am___n you see it all the time.

[DougD]

Originally Posted by Phalex

User reviews are usually written by people that have just bought a new piece of equipment and they are usually quite enamored with it at the time of their review. I don't believe they are very objective, but I do still find them useful.

Reviews from magazines and such are probably less than objective as well. If you go on record as saying that the latest offering from the Fender company is a POS they will likely make your life as rough as they can as payment for your candor.

Yup

[IanStephenson]

User reviews are a "self selecting survey" - kind of like a radio phone in show. If someone things something sucks, they call in. If they think it's great they call in. the 99% of people who really don't care don't call. Those that think a piece of gear is OK - some good points some bad (those that you WANT to hear from), don't bother.

Self Selecting Surveys are always bogus.

ian

[DougD]

Originally Posted by IanStephenson

User reviews are a "self selecting survey" - kind of like a radio phone in show. If someone things something sucks, they call in. If they think it's great they call in. the 99% of people who really don't care don't call. Those that think a piece of gear is OK - some good points some bad (those that you WANT to hear from), don't bother.

Self Selecting Surveys are always bogus.

ian

Another good piont Good to know that i'm not the only one that thinks like that

[embellisher]

Another thing you have to take with a grain of salt is the reviewers experience with similar product.

I gave my Jazz Geddy Lee a 7.5, based on experience with other 4 string J type basses in the $600 - $700 range. A lot of reviewers on this forum gave it a 9 or 10. My GL is great, but if theirs is really deserving of a 9 or 10, they got a much better bass than I did.


At the same time, I gave my Zon a 9.8. I held it to a much higher standard than the Geddy Lee, since it lists for about 6 times as much. But it came through, and was as near to flawless as any other bass I have ever played.

Now, if the Geddy Lee was the only bass I had ever owned, and I only had experience with a few basses under a grand, it probably would have gotten a 9.

If the Zon were the only high end bass that I had ever owned, or had experience with, it would have definitely gotten a 10.

But I have been playing for over 25 years, and have owned quite a few basses, and have a lot of experience with hundreds more. So, I am not quite as 'OMG this is the BEST' as somebody with less experience.

[DougD]

True

[tplyons]

Even as unbiased as you can truly be:
1. Manufacturers send the gear to the reviewers.
2. Manufacturers generally send the finest example.
3. Unless contracted, manufacturers may pull critical advertising after a bad review.

Very few agencies are actually unbiased in this manner, one that I can think of is the SNELL Foundation that tests motorcycle helmets for safety. They purchase the helmets right off the shelf without informing anyone of the fate of the helmet. No freebies. No hand picked products.

[The Golden Boy]

How many bad reviews have you ever seen in a magazine?

Magazine reviews are pretty much worthless, IMO, simply because bad reviews aren't conducive to getting manufacturers to advertise in the magazine.

When reading reviews on H-C or the like- it's one thing to take at face value, exactly what's being written, and then read into what's really there. In terms of trends... when you see several reviews mentioning nuts being cut badly, finish flaws, or amps running hot or overheating... those are things to seriously take into consideration.

Just because someone is offering their opinion, you still have to do the work in evaluating that opinion.

[David Lansen]

I actually think magazine reviews are probably more useful than most user reviews on the internet, because a magazine is not going to review every bass guitar there is on the market.

For example, there was a review in BP or Bass Guitar a few months ago on the MIA Fender Jazz 24 and Jazz 24 V. However, you're probably never gonna see a MIM Fender reviewed in a magazine.

On the other hand, if you go to MF and read their user reviews, you'll see some of them along the lines of "I just got this 3 days ago as my first bass, it is totally awesome! 5/5" or "I tried one in a store the other day and there was fret buzz like no tomorrow. 1/5". Just laughable.

Biased or not, at least the magazine reviewers know what they're talking about.

EDIT: Instead of reading the reviews for something, it's probably better just to ask about it here. Most people here know what they're talking about, and aren't being paid by any manufacturers.

David

[KPJ]

Originally Posted by David Lansen

For example, there was a review in BP or Bass Guitar a few months ago on the MIA Fender Jazz 24 and Jazz 24 V. However, you're probably never gonna see a MIM Fender reviewed in a magazine.

David

You are mistaken. The Jazz 24's are made in Korea. BP reviews quite a few mid-ranged foreign made instruments.

[David Lansen]

Originally Posted by KPJ

You are mistaken. The Jazz 24's are made in Korea.

Ah, you're totally right. My mistake.

[The Golden Boy]

Originally Posted by David Lansen

I actually think magazine reviews are probably more useful than most user reviews on the internet, because a magazine is not going to review every bass guitar there is on the market.

How does that make the review useful? They don't review a lot of stuff, but what they do review they provide well written, biased results???

For example, there was a review in BP or Bass Guitar a few months ago on the MIA Fender Jazz 24 and Jazz 24 V. However, you're probably never gonna see a MIM Fender reviewed in a magazine.

How is that useful if you're looking for reviews on an MIM Fender?

On the other hand, if you go to MF and read their user reviews, you'll see some of them along the lines of "I just got this 3 days ago as my first bass, it is totally awesome! 5/5" or "I tried one in a store the other day and there was fret buzz like no tomorrow. 1/5". Just laughable.

Biased or not, at least the magazine reviewers know what they're talking about.

EDIT: Instead of reading the reviews for something, it's probably better just to ask about it here. Most people here know what they're talking about, and aren't being paid by any manufacturers.

David

Whether you believe it to be laughable or not- a "user's" review is exactly that- someone who's put down the money on the product. It's up to you to decide how much merit to give that person's opinion- whether you should accept or ignore their entire review, or just pieces or parts.

If I read a "fit and finish are flawless" review in a magazine, but read 30 kid's reviews of the same instrument, and 15 of them mention that it sounds like garbage past the 7th fret- either those 15 kids don't know how to intonate their instrument, or there's a possible inherent problem in the neck, setup, bridge...etc. It's something to look at that the professional reviewer may have conveniently missed.

If a review is biased, how is that useful? If you trust the source, but they only emphasize positive things and skip over the bad things, how is that any more useful than someone on a gear honeymoon? I'd argue that it's worse, because you'd expect a professional review to be honest.

How are opinions/reviews here different than anywhere else on the intArweb?


Case in point- look at the reviews for the SWR SM-900:

http://reviews.harmony-central.com/r...WR/SM-900/25/1

There's 15 reviews.

I count 8 reviews mentioning the amp getting hot, going into thermal shutdown or blowing fuses. (but I'm not very good at counting)

I also note that (generally) the people that appear to have older amps are happy with their amps.

What would you infer?

[srxplayer]

I have done reviews but I will wait for several months before I comment on the product. I try to point out the negatives as well. Nothing is perfect at any price point. I try to point that out. I can give something an 8 or 9 but it's bassed on something related to the price point or it's place in the market. My Fender Jazz 24 does not rate a 8 compared to a Sadowsky, but compared to Ibanez SR 500 I think it does.

What I can't stand are reviews that offer nothing insightful. Just "this bass sucks" or "I played this bass at GC and it sucks". Or "I tried this amp at a metal gig I did and it sucks". "Anyone who would buy this amp is an idiot". Ok; Why does it suck? TB is full of those types of comments.

Those types of posts should be deleted.

[David Lansen]

Originally Posted by The Golden Boy

Just because someone is offering their opinion, you still have to do the work in evaluating that opinion.

First of all, I completely agree with the above. And it holds true for magazines, talkbass, HC, MF, amazon, everything. Every day we hear or read hundreds of "opinions" in conversation or on the internet, and my own little generalization is that usually, "less biased" + "more informative" = "more useful".

Something in an 1997 article I've read recently, about hypertext and the web, got me thinking. The author says,

Links... suggest relations, but also control access to information.... to the extent that readers do recognize that such choices have been made, they will be regarded as authoritative, since in most cases the ability to create such pages - the knowledge of the code - will attach a certain status to the invisible author, as well as the presumption that one who knew enough and cared enough about the topic to create the hyperdocument must have greater than average expertise on the subject.... On the other side, as more and more persons do develop the skills to create such documents... the aura of credibility of any particular hypertext on the Web will be diminished, since there will almost certainly be more garbage than work of quality in this Brave New Self-Publishing World.

The idea is referenced to a 1995 article by the same author. 12 years later, looking at the internet as it is today, I'd say he was absolutely correct.

In the case of magazine reviewers, the first part of the argument is that they are probably more knowledgeable on the subject than average. The second part of the argument is the limited authorship of a magazine. These things I can attribute to their better quality of work, in the sense that their reviews are generally more informative.

The other part of the equation is about bias. But I think one of the reasons you almost never see bad reviews in a magazine is they just don't bother reviewing bad products. The biggest problem I can see though, as someone stated earlier, is that the product being reviewed is hand-picked by the manufacturer, and is not representative to the whole line of the same product. Other things must come into consideration as well, setup, rig, etc.

So, when we get all this information from the reviews we read, we still have to do the work of evaluating it.

On the other hand, reviews on the internet are often less informative. Also, in your example here,

If I read a "fit and finish are flawless" review in a magazine, but read 30 kid's reviews of the same instrument, and 15 of them mention that it sounds like garbage past the 7th fret- either those 15 kids don't know how to intonate their instrument, or there's a possible inherent problem in the neck, setup, bridge...etc. It's something to look at that the professional reviewer may have conveniently missed.

It is also entirely possible that those 15 kids really don't know how to intonate their instruments. Because of the unlimited authorship of the internet, you can't really give them too much credibility. What you can take from these user reviews is you can make a note to yourself to find one of these instruments and seriously checking it out if you're considering to buy one.

So how are opinions/reviews different here, in magazines, on the internet, etc? I would say for every useful review in a magazine, there are probably six useless ones, and 100 useless ones on the internet. In terms of TB vs. MF, for example, perhaps for every useful review there are 20 useless one here, 50 useless ones on MF. These numbers are completely made up, they're kind of just examples, to show you how I feel about it. (EDIT: yes, it's only my "opinion". it's up to you to evaluate it.)

The bottom line is if you read reviews in a magazine, you would probably have to grind through less garbage to make a reasonable evaluation of it, than if you were to read user reviews.

David

p.s. by the way, the article I quoted from is:
Nicholas C. Burbules, "Rhetorics of the Web: Hyperreading and Critical Literacy," Page to Screen: Taking Literacy Into the Electronic Era, Ilana Snyder, ed. (New South Wales: Allen and Unwin, 1997).

[The Golden Boy]

Originally Posted by David Lansen

In the case of magazine reviewers, the first part of the argument is that they are probably more knowledgeable on the subject than average. The second part of the argument is the limited authorship of a magazine. These things I can attribute to their better quality of work, in the sense that their reviews are generally more informative.

The other part of the equation is about bias. But I think one of the reasons you almost never see bad reviews in a magazine is they just don't bother reviewing bad products. The biggest problem I can see though, as someone stated earlier, is that the product being reviewed is hand-picked by the manufacturer, and is not representative to the whole line of the same product. Other things must come into consideration as well, setup, rig, etc.

On the other hand, reviews on the internet are often less informative. Also, in your example here,
It is also entirely possible that those 15 kids really don't know how to intonate their instruments. Because of the unlimited authorship of the internet, you can't really give them too much credibility.
What you can take from these user reviews is you can make a note to yourself to find one of these instruments and seriously checking it out if you're considering to buy one.

So how are opinions/reviews different here, in magazines, on the internet, etc? I would say for every useful review in a magazine, there are probably six useless ones, and 100 useless ones on the internet. In terms of TB vs. MF, for example, perhaps for every useful review there are 20 useless one here, 50 useless ones on MF. These numbers are completely made up, they're kind of just examples, to show you how I feel about it. (EDIT: yes, it's only my "opinion". it's up to you to evaluate it.)

The bottom line is if you read reviews in a magazine, you would probably have to grind through less garbage to make a reasonable evaluation of it, than if you were to read user reviews.

My issue with magazine reviews is that in over 25 years of reading guitar/bass/gear magazines, I don't recall EVER seeing a bad review. I've seen bad record reviews. But I've never seen a piece of equipment torn apart in the same manner. The reason is advertising. As it's often been said, "don't bite the hand that feeds you," magazines are around 50% advertisements, and many manufacturers purchase full page ads, sometimes several in the same issue. For a magazine to take an advertiser's money, and turn out a poor review is, in effect, "biting the hand..." If you'll notice here at TB, there's a disclosure rule regarding endorsements and manufacturers- Magazines don't have that.

Rule #9: Endorsed Artist / Remuneration Disclosure
If you are an artist who endorses a music-related company, or if you receive remuneration (financially or in-kind) as the result of a relationship you have with a company, please disclose this via the "Disclosures" field in your profile. The text should be plain and clear, with no disclaimers, hype etc. For example: "Endorsing Artist: Joe Doe Basses" or "Affiliated with JD Amps" or "Discounted Gear: JD Amps", etc.

Also, please preface any relevant gear reviews on product you're affiliated with, with a brief disclosure in the post itself.

The purpose of these disclosures is to provide those who read your posts with the full knowledge of your situation, when considering your opinions on gear. We recognize the fact that in some circumstances a minor discount or other "non-endorsing" remuneration may be too insignificant to consider for disclosure. We trust that as honest bass players you'll use your best judgment and moral guide to determine if your fellow members would benefit from the disclosure of this information. TalkBass reserves the right to require disclosure if the remuneration is deemed significant.

And the TalkBass Commercial User Policy:
http://www.talkbass.com/forum/faq.ph...policy#faq_cup

Also to keep in mind, a reviewer is hanging onto a piece of equipment for... what... a week? Then, again there's the whole selected product to go for review... I don't agree that magazines don't review poor products.

After scouring a few, a magazine review will tend to encourage someone to buy something if they're interested by providing a more comprehensive focus on positive aspects, not provide an actual "review" of the product.

While there is a "signal to noise" ratio with real "user reviews," you're more likely to get a larger sample of a "real world scenario" in which you can pick and choose what's credible, relevant and what's not.