True double-blind audio testing thread- bring it on...

[steve_rolfeca]

I'm tired of all the posturing that goes on around here about tubes vs modelling, micros vs lead sleads, rosewood vs maple, etc. People keep making sweeping statements like "blind testing has proven that no-one can hear xxx", but I think they're all full of B.S.

I'd be happy to be proven wrong, but to give some context, I'm an avid reader, and I've been around audio for decades, and I've only ever seen ONE scientifically correct double-blind test.

Anyway, we are a sophisticated bunch. Everybody knows about the impact of perception on the results of non-double-blind testing. If you don't, there's plenty of literature on the subject.

So. We all have our positions. We know who believes that you must be deaf if you can't pick out a rosewood fingerboard, who is involved in computer modelling, and who believes that tubes will always "feel" better than solid state, etc.

We're still free to blither away about our biases to our hearts content on other threads. But on THIS thread, if you can't back it up with a published study using TRUE double-blind methodology, don't bother posting.

If you've got something, simply post a link, we'll all read it, and report back here for discussion. Should be interesting...

[steve_rolfeca]

I'll kick it off.

This is the only really sophisticated double-blind audio test that I've ever seen published. It was reported on decades ago, in one of the popular hi-fi magazines.

Some highly-respected, expensive separates with extremely good specs were set up against a cheap, wonky 35 watt Lafayette receiver, level matched, and equalized for matching frequency response to within a fraction of a dB. An acoustically transparent screen was set up to hide the sources, and a second A/B switch behind the screen allowed a third party to randomly change which side of the remote control was the good system. The experimenters used high quality, high-resolution speakers, switches and cables. The audience varied from housewives to industry figures and published "golden ear" journalists.

Note that because of the second switch, the person hosting the sessions was in the dark as well, and couldn't influence the results via conversational cues or body language.

The results proved conclusively that no-one had a clue which set of electronics were which. When some audience members claimed that they could have picked out the inferior system given a little more time, they were allowed to listen again. And again. And again. The more they listened, the closer the results came to a random coin toss.

A subsection of the final report pointed out that while the test proved that none of the audience members could hear the difference between .0001% and about 5% distortion, most of them (not just the golden ears) could consistently hear minor variances in frequency response down to a third of an octave and a quarter of a dB, and reported those differences using subjective language like "air" and "presence".

Both of these results defied conventional conventional engineering wisdom, as well as established high-end audio folklore.

To become truly scientifically valid, the experiment should have been subject to peer review, and tested for repeatability by other researchers. Given the explosive results, I expected it to be the subject of conversation in all the audio mags for months to come. Instead, you could have heard a pin drop.

I never saw it mentioned again, presumably because it was too embarrassing for everybody concerned.

[Febs]

Originally Posted by steve_rolfeca

People keep making sweeping statements like "blind testing has proven that no-one can hear xxx", but I think they're all full of B.S.

Anyone who makes that claim doesn't understand how blind testing works. Generally speaking, blind testing can establish that there is a high degree of probability that someone can hear a difference between two things, but it cannot establish that no-one can hear a difference between two things.

In any event, this could be an interesting thread. I think it would be very difficult to construct a proper blind test of something like rosewood versus maple, but I could imagine how you could construct a test of a modeler against a tube amp, and I would love to know how the results of that sort of test would turn out.

[fdeck]

In my view the best use of DBT is in a research environment to determine reasonable limits of audibility for subtle changes in volume or tone quality. This has been done extensively.

Those results can be used to make good educated guesses as to whether two amps (or whatever) with measurable differences in response are likely to sound different. That's a lot easier than trying to conduct a DBT on every hypothesis about gear. For instance, there's no reason to conduct DBT's of cables -- the effects are easy to measure, and those who want to believe in the face of evidence would dismiss the DBT results anyway.

The lack of debate about DBT in the audiophile world may be a social convention. I know there is at least one audiophile forum where the mention of DBT is forbidden.

[seamonkey]

There are AES and IEEE papers, that do some double blind tests.
AES, IEEE are peer reviewed by engineers and scientist that have formal credentials.
Unfortunately the papers and studies are only available to society members. You can buy them but you can't share them.

Some of these papers somehow make it to some websites.

Tubes Versus Transistors in Electric Guitar Amplifiers
A little EQ and both amps sound the same.

A lot of blind tests are done by companies, payed for themselves, and remain their own intellectual property. IK gets the original companies official certification that their models match and amp, cab, and mic. Some of the guitar amps they have are from the original developer private collection.

Which we end up with a golden standard problem. For old amps they are all different, or if they use tubes, they have different sound. The model creator has to pick one to start. There are hundreds essentially non-label models that are based on the best physical amp the modeler maker could get their hands on. Take a look at Peavey Revalver and they have clear clones from other amp makers in their product, but with slightly different names.

You can find informal listening tests done right here on this board. The outcome was that statistically people couldn't tell what was SS or tube, tube or analog model.

Electronic Musician did a double blind a few years back on modelers. Their panelist couldn't tell a difference, but it was obvious that with the samples they posted they didn't spend enough time making both samples sound the same.

Other audiophile tests from several years ago around amplifiers.
Matrix HiFi --> Blind testing high end full equipments
Expensive and cheap amplifiers were the same.

This same site also did a test where they recorded LPs to CDs and their listeners couldn't tell a difference.

Protools puts up listening challenges once in a while
Avid | HEAT Listening Challenge
They had one a few years ago on analog versus digital and again - statistically, nobody can tell. Which actually means they had to degrade the digital with modeled analog distortion and artifacts so people heard the same.

I'll note also, that although they are samples, not models, the worlds oldest company, Ziljian Cymbals, now sells their own sample libraries or their own products. Plenty of reasons why, but mainly it's so they keep control of the digital samples of their products. They call them a "Digital vault" which is quite accurate.

[seamonkey]

Another listening test for guitar
Double-blind test real Deluxe Reverb vs. POD HD (clean, no FX) - Page 4 - The Gear Page

[Mad Maestro]

Interesting article by Russell. O Hamm. "Tubes vs Transistors: Is there an audible difference?" Journal of the Audio Engineering Society (1973).

[billfitzmaurice]

Originally Posted by steve_rolfeca

The results proved conclusively that no-one had a clue which set of electronics were which. .

+1. Watch this:
YouTube - ‪Audio Myths Workshop‬‏

The main benefit of DBT is finding out whether listeners really can hear the difference between different items when they don't have the benefit of their eyes telling them what they're listening to. The main detractors of DBT testing are those who make, sell, or believe the advertising of, items that they know won't succeed in one.

[steve_rolfeca]

Originally Posted by seamonkey

...You can find informal listening tests done right here on this board. The outcome was that statistically people couldn't tell what was SS or tube, tube or analog model.

...Protools puts up listening challenges once in a while
Avid | HEAT Listening Challenge
They had one a few years ago on analog versus digital and again - statistically, nobody can tell. Which actually means they had to degrade the digital with modeled analog distortion and artifacts so people heard the same.

On this thread, "informal listening tests" don't cut it. Double blind or go home, baby.

Ditto for the Protools test "from a few years ago". I tried clicking through on the current Heat Listening Challenge, but couldn't find the original test. No linky, I smell a stinky.

[steve_rolfeca]

Originally Posted by seamonkey

Tubes Versus Transistors in Electric Guitar Amplifiers
A little EQ and both amps sound the same.

That's more like it. Interesting reading, and the results wouldn't match most people's expectations. I couldn't help noticing that those results matched the old hi-fi magazine article I referred to earlier- people were less sensitive to distortion characteristics than they were to carefully-matched frequency response curves.

Some pretty big holes in their methodology, though. Steady levels didn't test the dynamic response of the amps, no detail about what solid state amplifier was used, etc...

I also can't help wondering if those were the same CBS geniuses that "improved" the silverface Twin Reverbs to the point that no-one would buy them. It didn't take double-blind testing to figure out that those things didn't sound right, until they were brought back to blackface specs.

[steve_rolfeca]

Originally Posted by seamonkey

Another listening test for guitar
Double-blind test real Deluxe Reverb vs. POD HD (clean, no FX) - Page 4 - The Gear Page

This is a perfect example of what is wrong with so many "tests" on forums.

Almost none of the important listening cues were normalized in this one. Just to name one example, the levels weren't matched between examples. Not to mention the fact that it was MILES from a double-blind situation. The person playing the guitar, was the one who set the whole deal up, and he knew which system was which.

So you start with a huge experimental bias, sloppy testing methodology means that the levels are all over the place along with other prejudicial factors, and it ends up that no-one can consistently tell the difference- hardly a big surprise.

Bzzzzttt....

[seamonkey]

I'm afraid that unless you're willing to subscribe to a professional society you're not going to find published results from independent studies. If you do join, then the studies are copyrighted and you can't share anyhow.

[steve_rolfeca]

Originally Posted by seamonkey

I'm afraid that unless you're willing to subscribe to a professional society you're not going to find published results from independent studies. If you do join, then the studies are copyrighted and you can't share anyhow.

I hope and pray that you can see the humor in what you just posted...

[seamonkey]

Originally Posted by steve_rolfeca

That's more like it. Interesting reading, and the results wouldn't match most people's expectations. I couldn't help noticing that those results matched the old hi-fi magazine article I referred to earlier- people were less sensitive to distortion characteristics than they were to carefully-matched frequency response curves.

Some pretty big holes in their methodology, though. Steady levels didn't test the dynamic response of the amps, no detail about what solid state amplifier was used, etc...

I also can't help wondering if those were the same CBS geniuses that "improved" the silverface Twin Reverbs to the point that no-one would buy them. It didn't take double-blind testing to figure out that those things didn't sound right, until they were brought back to blackface specs.

It was peer reviews by engineers in AES. Nothing wrong with the methodology from those that reviewed the study. It's actually not a public domain study so I have no idea why you can find it on the internet.

You can find info on the amps used for the study on the AES site. I don't recall what they were.

Typical reaction to published tests is some conspiracy theory or confirmation bias. So why even bother?

[steve_rolfeca]

Originally Posted by seamonkey

Typical reaction to published tests is some conspiracy theory or confirmation bias. So why even bother?

Remembering my opening statement at the top of this thread, I would prefer if people didn't fall back to motherhood statements on this thread. Lets have a real dialogue, and stick to specifics.

In that spirit, my chief objections to the Fender study, are that there is so little detail about the clean versus distorted tests, and that they controlled the levels so closely.

Why does this concern me? Because in my own experience with guitar amplification, the biggest difference I have found between ss, tube and modelled topologies, has been in the area of dynamic response. It's relatively easy to cop the sound of a given crunch distortion sound, for example, but it's something else altogether to capture the dynamic response of an amp with a strongly reactive character.

[Sartori]

I'm of the opinion that people shouldn't give a crap what others like and just use what makes them happy, whether it's got tubes, transistors, or is powered by hamsters in spinning wheels.

[fdeck]

Originally Posted by steve_rolfeca

Why does this concern me? Because in my own experience with guitar amplification, the biggest difference I have found between ss, tube and modelled topologies, has been in the area of dynamic response. It's relatively easy to cop the sound of a given crunch distortion sound, for example, but it's something else altogether to capture the dynamic response of an amp with a strongly reactive character.

Here's what I suggest. Capture it on a scope. This would take some effort to set up, but I think it would still be considerably easier than trying to set up a true controlled listening test and lining up people to participate. Most of the tools are free. Record your bass signal digitally, so that every test always uses the same input signal. Record the outputs of amplifiers with various settings. The only bit of expertise needed would be to make sure it is not possible for the full speaker level output of your amplifier to reach the input of your recorder. There are direct boxes that step down speaker level signals.

Now you have waveforms that you can compare objectively through simple visual observation on your computer screen. It doesn't tell you how things sound, but it tells you what is going on, which would let you come up with ideas for what makes one kind of amplifier sound different than another.

[Big Brother]

I think Bob Weir nailed it when he said 20 odd years ago that analog still beat digital, but eventually digital would kick its arse. Something to that effect. No link.

Just like movies and the 24 frame effect, past a certain point those bits are indistinguishable from an analog signal to the ear and it's just a matter of approximating analog formulas to recreate any analog sound or effect. There is math behind analog sound and it can be applied to the digital realm, and shaped to taste. Sound waves are sound waves and they come from analog speakers either way.

Building in a 'reactive character' or tube latency to digital systems is simply a matter of a diligent programmer with that goal in mind to improve on an existing crunch distortion formula to make it so.

One this subject, my favorite source of funny is audiophiles, we used to have a serious one at work. Great guy and I liked him but his obsessions were mind boggling. He was bragging about a $300 power cable for his office sound system and you should have seen the look on his face when I casually mentioned that he plugged it into off the shelf, 20 year old oxygenated 14 gauge electrical wire on the other side of the wall plate. Took him about 4 seconds with a real disconcerted look to rationalize that inconvenient fact out of existence

Good equipment matters, I use B$W speakers and a NAD receiver for instance because they are made with high quality parts and sound really good to me, but the extremes and the silly claims people make in that industry are just comedy sometimes. Engineers spending hundreds of dollars on a 3 foot power cable just makes me marvel at the effectiveness of modern marketing.

[fdeck]

Originally Posted by Sartori

I'm of the opinion that people shouldn't give a crap what others like and just use what makes them happy, whether it's got tubes, transistors, or is powered by hamsters in spinning wheels.

That works, up to a point. Some of us are interested in understanding how bass gear works, for hobby or professional reasons. TalkBass has graciously put up with letting us discuss the technical side of bass gear on this forum.

[Sartori]

That's all well and good, but I see so much arguing on here about what is "better" or what people should or shouldn't use. I'm guilty of it myself at times. It just seems that if the amp sounds good to you, it is good.