Abstract: Personal blind sound quality comparison of xHE-AAC and Ogg Vorbis at around 134 kbps, and the Fabrice Bellard (https://bellard.org/)'s new AI-based codec TSAC: Very Low Bitrate Audio Compression (https://bellard.org/tsac/) at 7.5kbps.
Encoders: Except exhale v1.1.9, all of them are latest as of 7 May 2024.
xHE-AAC: exhale-V1.1.9-00423757_x64 (https://www.rarewares.org/aac-encoders.php), exhale-v1.2.1_x64 (https://www.rarewares.org/aac-encoders.php)
Ogg Vorbis: aoTuV_b6.03_2020. (https://ao-yumi.github.io/aotuv_web/)
TSAC: Windows version (experimental): tsac-2024-04-08-win64.zip. (https://bellard.org/tsac/)
xHE-AAC commandlines: exhale-V1.1.9-00423757_x64\exhale
5 in.wav out.mp4
exhale-v1.2.1_x64\exhale
5 in.wav out.mp4
ffmpeg109823 -c:a libfdk_aac -i out.mp4 -c:a pcm_f32le out.wav
Ogg Vorbis commandlines: aoTuV_b6.03_2020\venc64
-q3.7 in.wav out.ogg
oggdecV1.10.1\oggdec -b 5 out.ogg --wavout out.wav
TSAC commandlines: tsac-2024-04-08-win64\tsac -v -m tsac-2024-04-08-win64\dac_stereo_q8.bin -M tsac-2024-04-08-win64\tsac_stereo_q8.bin
-q 12 c in.wav out.tsac
tsac-2024-04-08-win64\tsac -v -m tsac-2024-04-08-win64\dac_stereo_q8.bin -M tsac-2024-04-08-win64\tsac_stereo_q8.bin d out.tsac out.wav
Sample tracks: 15 sound samples from Kamedo2's samples (https://hydrogenaud.io/index.php/topic,98003.msg815018.html)(full download (https://listening-test.coresv.net/samples/samples.zip)).
Total
15 diverse music sound samples, including highly critical samples.
Hardware: Sony PSP-3000 + AKG K712.
Results (only traditional codecs, at around 134 kbps): (https://listening-test.coresv.net/img2/26xhe-main-en2x.png)
Results (including AI codec at 7.5kbps, it is not a bitrate-equalized comparison): (https://listening-test.coresv.net/img2/26xhe-main-en2x2.png)
(https://listening-test.coresv.net/img2/26xhe-main-en2x2-tracks2.png)
Conclusions & Observations: - MPEG-4 xHE-AAC (eXtended High-Efficiency AAC), encoded by exhale (Ecodis eXtended High-efficiency And
Low-complexity Encoder), had very high fidelity at around 134kbps, with average score over 4.5. - Ogg Vorbis, encoded by aoyumi's aoTuV beta6.03(latest version as of 2024 May), also had very high fidelity at around 134kbps, with average score more than 4.4.
- It's not clear which encoder, xHE-AAC or Ogg Vorbis, was better, from this test alone. The difference was small.
- Both xHE-AAC and Ogg Vorbis at around 134kbps were better than the TSAC: Very Low Bitrate Audio Compression at 7.5kbps, at its maximum bitrate setting as of 2024-04-08 version. TSAC used 94.4% less disk space, and this test not meant to be a filesize-wise fair comparison.
Anova analysis: FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 59 86.42
Testers (blocks) 14 1.00
Codecs eval'd 3 83.62 27.87 651.96 0.00E+000
Error 42 1.80 0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.152
Means:
exh119 exh121 ogg128 tsac12
4.56 4.55 4.45 1.79
---------------------------- p-value Matrix ---------------------------
exh121 ogg128 tsac12
exh119 0.861 0.141 0.000*
exh121 0.193 0.000*
ogg128 0.000*
-----------------------------------------------------------------------
exh119 is better than tsac12
exh121 is better than tsac12
ogg128 is better than tsac12
Raw data: exhale v1.1.9 exhale v1.2.1 aoTuV beta6.03 tsac-2024-04-08-win64
%feature 3 xHE-AAC xHE-AAC Ogg Vorbis TSAC: Very Low Bitrate Audio Compression
%feature 10 test tracks avg/albums avg test tracks avg/albums avg test tracks avg/albums avg test tracks avg/albums avg
%feature 11 138.7kbps/130.4kbps 139.0kbps/129.6kbps 143.2kbps/126.1kbps 7.4kbps/7.5kbps
%feature 12 exhale 5 in.wav out.mp4 exhale 5 in.wav out.mp4 venc64 -q3.7 in.wav out.ogg tsac -v -q 12 c in.wav out.tsac
5.000 4.500 4.400 2.300
4.400 4.500 5.000 1.700
5.000 5.000 4.300 1.900
4.800 4.600 4.400 1.900
4.400 4.600 4.800 1.800
4.500 4.600 4.700 1.500
4.600 4.700 4.300 2.100
4.700 4.500 4.600 1.700
4.500 4.400 4.700 1.700
4.100 4.300 4.400 1.600
4.400 4.500 4.200 1.500
4.700 4.500 4.300 1.800
4.500 4.700 4.300 1.700
4.500 4.400 4.200 1.900
4.300 4.400 4.100 1.800
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella
Other tests: - Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3. (https://hydrogenaud.io/index.php/topic,124080.0.html)
- Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding (https://hydrogenaud.io/index.php/topic,122575.0.html)
- Personal blind sound quality comparison of the Bluetooth codecs (AAC vs LC3) (https://hydrogenaud.io/index.php/topic,121850.0.html)
- Personal blind sound quality comparison of Opus hard-CBR with framesize options (https://hydrogenaud.io/index.php?topic=121552.0)
- Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz (https://hydrogenaud.io/index.php?topic=121155.0)
- Personal Blind Listening Test of Bluetooth codecs (https://hydrogenaud.io/index.php?topic=120182.0)
- Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder (https://hydrogenaud.io/index.php?topic=120936.0)
- Personal Listening Test of AAC-LC and xHE-AAC at 96kbps and 128kbps (https://hydrogenaud.io/index.php?topic=119861.0)
- Personal Listening Test of AAC and LAME encoders (old test, translated) (https://hydrogenaud.io/index.php?topic=102876.0)
- Personal Listening Test of AAC, WMA, and MP3 encoders (old test, translated) (https://hydrogenaud.io/index.php?topic=102875.0)
- Personal Listening Test of MP3 and Opus (https://hydrogenaud.io/index.php?topic=117489.0)
- Personal Listening Test of AAC encoders available from FFmpeg (https://hydrogenaud.io/index.php?topic=111085.0)
- Personal Listening Test of Experimental Modified Opus Encoders at 36, 48 kbps (https://hydrogenaud.io/index.php?topic=113985.0)
- Personal Listening Test of 2 Opus encoders (https://hydrogenaud.io/index.php?topic=111798.0)
- Personal Listening Test of LAME, iTunes and Helix MP3 encoders (https://hydrogenaud.io/index.php?topic=113324.0)
- Personal Listening Test of MP3/Opus/AAC at 96kbps (https://hydrogenaud.io/index.php?topic=109716.0)
ahaha, wow, TSAC-encoded files look painful to listen to!
ahaha, wow, TSAC-encoded files look painful to listen to!
Please note that TSAC used only 1/18th of the disk space of these competitors.
Thanks a lot, Kamedo2, for your meticulous listening! I know how much work blind listening tests at these bitrates are. Some distilled observations and connections with previous discussions on HA:
- In your earlier (09/2020) tests (https://hydrogenaud.io/index.php/topic,119861.0.html), exhale 1.0.6 reached a mean score of "only" 4.47 on the same 15 samples. It's nice to have some evidence now that exhale's higher-rate audio quality improved some more during the last 3-4 years.
- Regarding my score estimator (https://hydrogenaud.io/index.php/topic,118888.msg989150.html#msg989150), your average results from yesterday are now above my estimate, similar to IgorC' 2020 results at 192 kbps. Maybe I can finally update my score estimator at the higher rates :)
- AFAIR, this is the first blind test in which a machine learned end-to-end codec is compared against high-performance classical audio codecs. This is very valuable information since it puts all these "AI codecs outperform MP3" hyped claims into perspective - machine learned codecs deliver decent audio quality, but they don't scale to transparency... at least not in 2024.
Chris
but they don't scale to transparency... at least not in 2024.
They don't target transparency, which is why I think it would be more useful if:
* Samples used in tests are not handpicked for being challenging, but picked for being representative of average everyday use (talk show, pop music, ..etc).
*
Classical codecs are tested at a much lower bitrate than in this test. Maybe a something in the range of 32-48kbps.
Thanks a lot Kamedo for yet another very interesting listening test.
Nice to see Vorbis being tested and yielding quite impressive results in the ~128kps range. Is there any particular reason why you chose Vorbis in this test?
Nice to see Vorbis being tested and yielding quite impressive results in the ~128kps range. Is there any particular reason why you chose Vorbis in this test?
Because it hasn't been tested for some years.
... I think it would be more useful if:
* Samples used in tests are not handpicked for being challenging, but picked for being representative of average everyday use (talk show, pop music, ..etc).
* Classical codecs are tested at a much lower bitrate than in this test. Maybe a something in the range of 32-48kbps.
Fair points which generally apply to the design of blind listening tests, but (1) the handpicked critical samples are challenging for
classical audio codecs* - notice how no sample really stands out with a particularly bad score for TSAC, (2) we have several blind tests of classical codecs at 32, 40, and 48 kbps by Kamedo2 and guruboolez - even there, the scores average way above 2.0 IIRC.
* The most challenging content for AI codecs that I know of, Glockenspiel and triangle, isn't included in Kamedo2's test(s), by the way.
Chris
Thanks a lot Kamedo for yet another very interesting listening test.
Nice to see Vorbis being tested and yielding quite impressive results in the ~128kps range. Is there any particular reason why you chose Vorbis in this test?
Nice to see Vorbis being tested and yielding quite impressive results in the ~128kps range. Is there any particular reason why you chose Vorbis in this test?
Because it hasn't been tested for some years.
This is a great idea. Thanks a lot for your hard work.
Nice work Kamedo, as usual.
I really appreciate that we have a rational and well-organized evaluation of TSAC. Sound quality seems really far from transparency with music but the bitrate is truly amazing. I felt stupid when I posted here (https://hydrogenaud.io/index.php/topic,120997.0.html) a listening test at only 12 kbps and I didn't expect that someone will get deeper in the abyss. Good job!
@Kamedo2! Try on my samples! https://hydrogenaud.io/index.php/topic,125921.html