Public, Multiformat Listening Test
p2p news / p2pnet: "After 11 days of collecting results Roberto Amorim today announced the results of his 2nd Multi-Format listening test: Vorbis fork AoTuV scored the highest and ranks as the winner together with open source contender Musepack closely followed by Apple’s AAC implementation and LAME MP3, which improved markably since last year thanks to further tunings of its VBR model done by Gabriel Bouvigne," p2pnet posted in May last year.
Now it’s test time again – the time to discover "which popular, VBR encoder outputs the best quality on bitrates around 128 kbps," says Sebastian Mares, pointing out there have been plenty of changes since Roberto Amorim’s effort in 2004 >>>>>>>>>>>>>>>>>>>>>>>>
Public, Multiformat Listening Test @ 128 kbps
By Sebastian Mares – maresweb.de
The purpose of this test is to find out which popular, VBR encoder outputs the best quality on bitrates around 128 kbps.
I – Which codecs and settings are tested?
Nero AAC 3.1.0.2
VBR/Stereo – Streaming, 100-120 kbps [LC AAC]
iTunes AAC 6.0.1.3
128 kbps, VBR
LAME 3.97 Beta 2
V5 –vbr-new
Ogg Vorbis AoTuV 4.51 Beta
-q 4.25
WMA Professional 9.1
Quality-Based VBR, Q50
Shine 0.1.4 (Low Anchor)
-b 128
II - Is it normal that the bitrate is very high on some samples (even 180 kbps)?
Yes, and that is the beauty of VBR encoding – it will simply ignore bitrate limitations whenever possible, using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it is the best way to compare modern codecs that shine most in VBR mode, like MP3 and Vorbis. Trying to force a VBR setting to match a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it’s more usual that a user just sticks to a quality setting, not caring much about a specific bitrate.
The quality settings for the VBR codecs were chosen because they average out to about 128 kbps over a number of encoded albums. It would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.
III – Who should take the test?
Anyone interested in lossy audio quality, or people who have no interest but would like to help making this test better are invited. You don’t need excellent hearing, but some good gear is welcome. Headphones are a must-have.
IV – How do I take the test?
Download ABC/HR and the readme from the following location: ABC-HR_bin.zip
If you already have ABC/HR, download the readme from here:
readme.txt
readme.htm
Further instructions and links to the sample packages are inside the readme.
V – Can I take the test even if I am not running Microsoft Windows?
Thanks to schnofler, any person running an operating system with Java Runtime Engine 1.4 can participate (JRE 1.5 is recommended, though). Instructions are available inside the ABC/HR package.
VI – When will the test finish?
The test is scheduled to end on December 25th. This date may be extended if proven to be necessary.
If you have any questions, feel free to contact me via e-mail: sebastianmares[@]web.de.
Best regards,
Sebastian Mares
December 6th, 2005 at 7:25 pm
This is nice to see. I’ve been waiting for this test to take off for a while now, ever since I heard it was under consideration a few months ago. It’s of particular interest to me because of prevalence of AACplus these days (recently added to Winamp and Nero 7) which I think is a very good codec for extremely low bitrate needs (internet radio for example).
If I had to guess I would say Vorbis will likely come out on top in this test as it has in the past. Will be interesting to see. Anyways, glad to see this covered here on p2pnet as there is truly a lot of ignorance out there regarding audio compression. Thanks primarily to marketing mumbo jumbo, most people tend to think that codecs like WMA, MP3, and AAC are CD quality at low bitrates like 64k-128k when nothing could be farther from the truth. There is also no shortage of folks out there giving bad info to people which more often than not tends to be based on very poor testing techniques, or even more often their opinion (self proclaimed ‘audiophiles’ are the worst). The undeniable fact is that only ABX testing is a truly accurate and objective means for figuring out where different codecs stand against each other in terms of true quality versus the original lossless sample (most often compact disc). Think about it; would you buy a video card based solely on someones opinion of a manufacturer, or would you prefer to read the results of objective scientific testing and benchmarking instead?