|
Poor Man's Double Blind Audio Test
To follow up on the audio question...
~~~ THE SETUP ~~~
I took a CD track (16/44.1) that has excellent mastering, containing both soft and loud passages, both simple and complex melodies, including orchestral accompaniment.
I ripped using CDex 1.51 into an uncompress WAV file. In turn, I used CDex to create two MP3s using the Lame 1.30 encoder. One was a CBR 320kbit file, and one was a VBR 320kbit file. Both were set for maximum quality.
Additionally, I used Media Player 9 [*] to rip three copies: WMA9 192kbit CBR, WMA9 355kbit VBR, and WMA9 Lossless.
I did not test any other formats, as these two are the only ones supported by my variety of music devices.
I used CDex to decode the MP3s back into WAV files, and then used EZ CD Creator Platinum to create an audio CD with all 6 files on them (3 were in WAV format, and 3 were in WMA, which was converted by EZCD before burning).
Take CD-R to my DVD player, fed into an outboard DAC, through an analog preamp, into two monoblock amps, into stereo speakers. A second person plays the tracks in random order, noting the track #s. Listener (me) did not know any track #s until the listening was finished.
~~~ THE RESULTS ~~~
Both MP3 files exhibited top end drop-out and ringing and edginess, although oddly enough this was less pronounced on the VBR file, which was smaller in size. They were especially garbled as the music got complex, blending together details that normally stood out.
I was unable to find any of these flaws in the WMA files or the uncompressed (original) track.
In point of fact, I didn't expect to hear many differences at all going in. 320kbit MP3 using the widely regarded Lame codec should've minimized them. I was shocked at the obvious glare and muddiness imparted by the MP3 codec.
~~~
[*] The reason for the variable of using WMP9 is that CDex does not support WMA9 (it only supports WMA8 CBR, max 160kbit). I would've preferred to use CDex for the encode and decode, but we can't have it all. :-p
Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, October 23, 2003
Lame 1.30? The current version is 3.93. That is way out of date... I'd be curious to see the results if you tested the results with a recent version.
Roose
Thursday, October 23, 2003
I'd like to know what music you're using. I once listened a great Tony Bennet CD in a high end A/V store which was perfect for evaluating speakers.
nat ersoz
Thursday, October 23, 2003
Since the standard MP3 format only allows for up to 320kb/s, encoding at VBR will never go higher than that at any point.
So the fact you found the VBR file to sound better than the CBR file at that bitrate means something is seriously screwed up with some component of your test. Maybe you had a buggy version of the encoder? I don't know, but it's certainly worth checking out.
Sum Dum Gai
Thursday, October 23, 2003
"I once listened a great Tony Bennet CD in a high end A/V store which was perfect for evaluating speakers."
Tony Bennet, although a good listening choice, is hardly challenging for a decent pair of speakers or an encoding algorithm. I've ripped my entire music collection to MP3. The most difficult task was ripping music that is very polyphonic and multitimbral. For example, I had a really hard time trying to obtain a decent rip of My Bloody Valentine's "Only Shallow" and Slowdive's "Souvlaki Space Station" and I needed a very high bit rate to do it right. Songs like Sarah Shannon's "When You Live Life Alone," which is much closer to Tony Bennet stylistically, encoded quite well with just the default settings.
Matt Latourette
Thursday, October 23, 2003
There was an audio question?
Technically a double blind test is one where the person administering the test also doesn't know which is the real deal and which is the placebo.
You can fake it with software if you're alone and can't get someone to administer a proper test.
ABC/Hidden Reference Audio Comparison Tool
http://ff123.net/abchr/abchr.html
PC ABX
http://www.pcabx.com
www.MarkTAW.com
Thursday, October 23, 2003
I'd say the best testing choice is the music you normally listen to ...
Seriously, if Tony Bennt is what you normally listen to, then you should be checking that he sounds good with your chosen encoding settings. You should not give a damn whether <insert stock sound test material> sounds good or not, since something you never listen to sounds perfect at 0kb/s.
And while I appreciate the value of double blind listening tests to assist codec developers, I question their utility to determine an acceptable rate for your own encoding. You're setting up an overly harsh test - you're specifically looking for points of failure. Under normal listening conditions, you're not. A far better test is to listen to a selection of music that you normally would for a few weeks under your normal listening conditions and see whether there is any bothersome quality differential.
You might find that although a quality level fails the ABX test for you, it's perfectly acceptable for listening to music on a day to day basis.
Sum Dum Gai
Thursday, October 23, 2003
I find Windows Media fails miserably when it comes to high hat cymbals or similar instruments. I think this is because WMA uses a fixed window over which to perform the FFT and doesn't correctly switch to short windows when dealing with fast attack/decay instruments. Listen for it yourself.
Richard Kuo
Thursday, October 23, 2003
I agree with Sum Dum Gai, an A/B test between MP3 and CD will typically reveal that there's simply more informaiton in the CD, especially on the high end.
But for background listening, or sheer portability, compression scheme A vs. compression scheme B is, for most people, simply not worth the effort it takes to evaluate it, and since music is a subjective experience anyway, different people will prefer different schemes.
You might be interested in The Great Dither Shootout:
http://www.24-96.net/dither/
even though it doesn't directly pertain to mp3 encoding, but rather going from 24/96 to 16/44.1, or actually any 24 bit to any 16 bit word length.
www.MarkTAW.com
Thursday, October 23, 2003
Sorry, here's the full line description of the encoder from CDex:
"Lame MP3 Encoder (version 1.30, engine 3.92 MMX)"
The specific track is one I've used for a long time as an acid test when trying out speakers, track #12 from Extreme's "III Sides to Every Story".
Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, October 23, 2003
"So the fact you found the VBR file to sound better than the CBR file at that bitrate means something is seriously screwed up with some component of your test. Maybe you had a buggy version of the encoder? I don't know, but it's certainly worth checking out."
I can't explain it either. I used the encoder everybody says they like the best. My purpose for testing it was to see whether I could hear any difference (since the file size with VBR was about 25% smaller). I set the minimum bitrate to 128k and the maximum to 320k.
Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, October 23, 2003
Brad,
Kudos to you. Great job.
To the folks who know better than brad -- do your own double-blind studies and then publish the results. That's how you refute studies, not by whining.
Dennis Atkins
Thursday, October 23, 2003
And then there's just whining for its own sake...
nat ersoz
Thursday, October 23, 2003
Mark,
I've heard a lot people swear by up-sampling CD players as a way to get almost all of the quality improvements that we now see with formats like DVD-Audio and SACD. The claim, one I'm not really capable of verifying or refuting, is that the simple up-sample process gives you a much lower noise floor and decoding errors that are much less severe, thereby restoring some of the "hidden" resolution in CDs thats lost in the traditional decoding process.
I can't vouch for whether that's true or not, because I've never heard it. I'm married now, so my days of buying expensive things like upsamplers are over. ;)
Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, October 23, 2003
Brad, just curious what audio equipment you use. I use rotel electronics fed into paradigm speakers w/ cardas interconnects/wires. Realize it's a bit off topic, but I'm always curious what other people use.
Elephant
Thursday, October 23, 2003
Carver Amazing Loudspeakers (hybrid ribbons), pair of Bryston 7Bs, an ancient preamp from Adcom, an equally ancient Aragon DAC, and a Toshiba SD-4900 DVD player.
Excepting the Toshiba, which is new, all the equipment I probably at least 4 years old for me. The Adcom and Aragon were both second hand, and HAVE to be 10 years old if they're a day.
The speakers are a hard load. 4 ohms constant, with dips down to 1 ohm. 65W minimum, 450W max, 87dB sensitivity (i.e., wasteful). But they're liquid and involving and the sound stage is absolutely immense. I am an absolute "line source" convert after owning this speakers; I couldn't imagine owning a traditional speaker.
Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, October 23, 2003
"You should not give a damn whether <insert stock sound test material> sounds good or not, since something you never listen to sounds perfect at 0kb/s."
Agreed. You should do your testing with a musical selection that you know well. My point about Tony Bennet is that his music leans toward reductivism. If the vast majority of you music collection consists of Tony Bennet, Sinatra, etc. then of course you should be testing with it. But, if you listen to a wide variety of music, I think it's better to pick something that has a very full sound. A selection that is very "busy" but not "muddy" will really challenge speakers and encoders. When you use a piece like that to do your tests, the flaws in the reproduction are glaringly obvious.
Matt Latourette
Thursday, October 23, 2003
I think you should really retry the test using OGG/Vorbis at Max quality, and APE Lossless...
Michaël
Tuesday, October 28, 2003
 Recent Topics
Fog Creek Home
| 
