Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of MP3 and Opus (Read 36205 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of MP3 and Opus

Abstract:
Blind sound quality comparison of MP3 and Opus.
Opus was tested at half the bitrate of the MP3.

Encoders:
LAME 3.100 http://www.rarewares.org/mp3-lame-bundle.php
opusenc opus-tools 0.1.10-80-g1a646c4 (using libopus 1.3-rc-12-gbc4ecf19, 2018 July version. This is older than the current version. I started this listening test in 2018 August.) https://hydrogenaud.io/index.php/topic,116059.75.html

Settings:
lame -h -b 96 in.wav out.mp3
lame -h -b 128 in.wav out.mp3
lame -h -b 160 in.wav out.mp3
lame -h -b 192 in.wav out.mp3
opusenc --bitrate 48 in.wav out.opus
opusenc --bitrate 64 in.wav out.opus
opusenc --bitrate 80 in.wav out.opus
opusenc --bitrate 96 in.wav out.opus

Samples:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 sound samples.

Hardwares:
Sony PSP-3000 + RP-HT560.

Results:




Conclusions & Observations:
The Opus at 64kbps setting sounds better than the MP3 at 128kbps.
The Opus at 48kbps setting sounds better than the MP3 at 96kbps.
The Opus is likely to be the first format which delivers the equivalent sound quality as MP3 at half the bitrates, not as a marketing plot but as a fact.
The high efficiency of Opus is more pronounced at the lower rate.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 1.11E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              215          86.75
Testers (blocks)    26          19.92
Codecs eval'd        7          45.15    6.45   54.12  1.11E-016
Error              182          21.69    0.12
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.185

Means:

mp3192   opus96   opus80   mp3160   opus64   mp3128   opus48   mp3_96   
  4.27     4.25     4.07     4.03     3.86     3.51     3.32     2.90   

---------------------------- p-value Matrix ---------------------------

         opus96   opus80   mp3160   opus64   mp3128   opus48   mp3_96   
mp3192   0.844    0.035*   0.012*   0.000*   0.000*   0.000*   0.000*   
opus96            0.055    0.021*   0.000*   0.000*   0.000*   0.000*   
opus80                     0.694    0.026*   0.000*   0.000*   0.000*   
mp3160                              0.066    0.000*   0.000*   0.000*   
opus64                                       0.000*   0.000*   0.000*   
mp3128                                                0.042*   0.000*   
opus48                                                         0.000*   
-----------------------------------------------------------------------

mp3192 is better than opus80, mp3160, opus64, mp3128, opus48, mp3_96
opus96 is better than mp3160, opus64, mp3128, opus48, mp3_96
opus80 is better than opus64, mp3128, opus48, mp3_96
mp3160 is better than mp3128, opus48, mp3_96
opus64 is better than mp3128, opus48, mp3_96
mp3128 is better than opus48, mp3_96
opus48 is better than mp3_96


Raw data:
Code: [Select]
opus48	opus64	opus80	opus96	mp3_96	mp3128	mp3160	mp3192	
2.800	3.700	4.100	4.300	1.800	3.400	4.200	4.400	
4.000	4.400	4.600	4.600	3.500	4.200	4.400	4.700	
3.100	3.800	3.800	4.400	2.700	3.400	4.100	4.300	
3.000	3.600	4.100	4.200	3.300	3.800	4.300	4.500	
3.500	4.100	4.200	4.500	2.700	2.900	3.400	3.800	
2.400	3.100	3.600	3.800	3.300	3.900	4.100	4.200	
3.700	5.000	4.300	5.000	2.400	2.700	3.300	4.000	
3.500	4.000	4.100	4.300	3.300	3.800	4.400	4.600	
2.400	3.000	3.500	3.700	2.500	3.300	3.800	3.900	
2.900	3.600	3.900	4.000	2.600	3.200	4.100	4.200	
3.000	3.700	3.900	3.900	2.800	3.200	4.100	4.300	
2.600	3.600	4.000	3.900	2.400	3.100	3.900	4.200	
3.200	3.600	3.900	4.300	2.400	3.900	4.400	4.500	
2.500	3.700	4.200	4.400	2.400	3.400	4.000	4.200	
3.100	3.400	3.800	4.000	3.000	3.500	3.700	4.200	
4.000	4.100	4.400	4.500	2.800	3.300	3.500	3.700	
4.600	4.800	4.700	5.000	2.700	3.300	4.000	4.300	
3.500	3.800	3.900	3.700	2.500	2.700	2.800	3.200	
3.300	3.600	3.800	4.200	3.500	4.000	4.400	4.600	
3.000	3.300	3.500	3.700	3.100	3.800	3.900	4.200	
4.400	4.500	4.700	4.600	3.300	3.600	4.200	4.600	
2.900	3.800	4.100	4.200	2.400	3.300	4.300	4.400	
3.500	3.800	3.900	3.900	2.400	2.900	4.000	4.200	
3.900	4.300	4.300	4.500	3.500	4.200	4.400	4.400	
3.100	3.600	3.900	4.100	3.300	3.800	4.100	4.300	
4.300	4.300	4.600	4.700	4.100	4.400	4.800	5.000	
3.500	4.000	4.100	4.400	3.600	3.900	4.300	4.400	
%samples	41_30sec	Perc.
%samples	finalfantasy	Strings
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	Classic
%samples	mybloodrusts	Guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	HearttoHeart	Pops
%samples	Tom'sDiner	Acappella
%samples	01 castanets	inst.
%samples	02 fatboy_30sec	Techno
%samples	03 eig	Techno
%samples	04 Bachpsichord	inst.
%samples	05 Enola	Techno
%samples	06 trumpet	inst.
%samples	07 applaud	Live
%samples	08 velvet	perc.
%samples	09 Linchpin	Rock
%samples	10 spill_the_blood	guitar
%samples	11 female_speech	Speech
%samples	12 French_Ad	Speech


Bitrates:
Code: [Select]
%bitrate
opus48	opus64	opus80	opus96	mp3_96	mp3128	mp3160	mp3192	

56880	75747	95248	114257	96307	128285	160357	192428
66948	84991	103687	123011	96255	128230	160288	192346
54746	72574	90622	108186	96399	128372	160465	192558
52458	69741	86915	104050	96368	128319	160399	192479
65630	86473	107602	127975	96401	128368	160461	192553
50549	67344	83756	99633	96549	128440	160551	192661
53568	69782	84808	99693	96485	128458	160572	192687
56704	75748	94913	113849	96344	128271	160339	192407
56894	74481	92233	109998	96689	128827	161035	193242
55933	74323	92551	110275	96390	128295	160369	192443
49407	66008	83000	100052	96348	128244	160305	192366
56951	75077	93069	110522	96367	128405	160507	192608
53149	69985	86551	103243	96390	128353	160442	192531
52610	69890	87487	104392	96359	128304	160381	192456
56351	70628	82992	97120	96414	128470	160589	192706
61687	81349	103220	125449	97415	129023	161281	193535
69925	93369	117074	140997	96320	128336	160420	192504
64676	85344	107088	128612	96533	128613	160767	192919
69446	88817	109699	130814	96353	128396	160495	192594
54523	71962	89108	105862	96345	128301	160376	192451
68054	85839	106781	125858	96814	128666	160833	192999
58885	77742	95976	113623	97163	128850	161063	193275
54146	71966	91347	110123	96593	128639	160800	192959
46454	61405	75877	90013	96498	128588	160736	192883
59458	77547	95042	113980	96293	128264	160331	192396
48064	58813	67410	79530	96364	128432	160541	192649
57471	75071	93052	110375	96691	128663	160829	192995

%album bitrate
49285 	65423 	81510 	97711 	96031 	128029 	160037 	192044

Re: Personal Listening Test of MP3 and Opus

Reply #1 –
Interesting, but I wonder why you chose CBR, not VBR using lame.
lame3995o -Q1.7 --lowpass 17

Re: Personal Listening Test of MP3 and Opus

Reply #2 –
Quote from: halb27 on
Interesting, but I wonder why you chose CBR, not VBR using lame.

CBR was widely used in the past so it's a better anchor.

Re: Personal Listening Test of MP3 and Opus

Reply #3 –
Thank you for this very interesting test.
I use Opus 96 kbps VBR converted by foobar2000.

Re: Personal Listening Test of MP3 and Opus

Reply #4 –
Quote from: Kamedo2 on
Hardwares:
Sony PSP-3000 + RP-HT560.

Wait there's abx homebrew for the psp? This makes me think about dusting off mine to use as a portable player instead of my phone.

Re: Personal Listening Test of MP3 and Opus

Reply #5 –
Can you add also opus at 32kbps? For other part the result is no surprise for me as 64kbps have been my selected bitrate to use on the go for music (phone, car...), I also use 32kbps for audiobooks and 48kbps for podcasts (latelly I even sometimes use 32kbps for podcasts).

Re: Personal Listening Test of MP3 and Opus

Reply #6 –
Quote from: MetaPixel on
Quote from: Kamedo2 on
Hardwares:
Sony PSP-3000 + RP-HT560.

Wait there's abx homebrew for the psp? This makes me think about dusting off mine to use as a portable player instead of my phone.

Yes, I have it. http://zak.s206.xrea.com/bitratetest/ABCHR2.4.7z http://zak.s206.xrea.com/bitratetest/pspabchr24.jpg It works with LCFW 6.60 PRO-C2. https://code.google.com/archive/p/procfw/wikis/Installation.wiki

Re: Personal Listening Test of MP3 and Opus

Reply #7 –
Quote from: Elohe on
Can you add also opus at 32kbps? For other part the result is no surprise for me as 64kbps have been my selected bitrate to use on the go for music (phone, car...), I also use 32kbps for audiobooks and 48kbps for podcasts (latelly I even sometimes use 32kbps for podcasts).

I have to say that Opus at 32kbps is of low quality at this moment, and recommend upgrading to 48kbps, the rate extensively tested, for any static contents distributions. Memory and bandwidths are cheap now. https://hydrogenaud.io/index.php/topic,113985.0.html https://hydrogenaud.io/index.php/topic,111798.0.html

Re: Personal Listening Test of MP3 and Opus

Reply #8 –
Kamedo2 every bitrate have its use, normally the configs that I use for my phone is this:
Pure talk -> 32kbps
Talk + incidental music -> 48kbps
Music -> 64kbps

Actually 32kbps is more used that most people here can think, specially in places with data caps and/or low speeds.

And I miss a more aggressive VBR mode for opus, more like a dual rate mode where one can specify different rates for music and voice.

Re: Personal Listening Test of MP3 and Opus

Reply #9 –
Quote from: Phanton_13 on
Actually 32kbps is more used that most people here can think, specially in places with data caps and/or low speeds.

A devoted podcast subscriber who considers listening to the program is an important part of his/her life, may listen to 1 hour of podcast 30 days a month. In that case...

32kbps -> 0.4 GB/month
48kbps -> 0.6 GB/month
diff        -> 0.2 GB/month

Are you really sure that 0.2GB of cost reduction is more important than improving his/her listening experience?
(I may be missing some problems somewhere. Here in Japan increasing the mobile data cap 2GB more typically costs 200~2,000 JPY/month, and after using up all the data cap the speed is typically limited to 128 kbps~200 kbps.)

Re: Personal Listening Test of MP3 and Opus

Reply #10 –
From my very limited experience ;Optus 32  - very bad artifacts like no need to abx at all. Even at 64 I think i can still hear it but much better quality than 32. @ 96k I don't know.

Re: Personal Listening Test of MP3 and Opus

Reply #11 –
https://tools.ietf.org/html/rfc6716
8-12 kbit/s for NB speech,    NB (narrowband)      |           4 kHz |                   8 kHz
16-20 kbit/s for WB speech,    WB (wideband)        |           8 kHz |                  16 kHz
28-40 kbit/s for FB speech,    FB (fullband)        |      20 kHz (*) |                  48 kHz
48-64 kbit/s for FB mono music
64-128 kbit/s for FB stereo music.

Re: Personal Listening Test of MP3 and Opus

Reply #12 –
Music with Opus 1.3 at 32 kbps has too much noise and sibilance. Old samples of USAC from the original verification tests usually sound better to me than Opus 1.3 still. Below 64 kbps and especially below 48 kbps Opus loses more and more to certain higher latency codecs. For speech it is a different story though, 32 kbps is great and if limited to mono then even 16 kbps is good. I look forward to what can be done with LPCnet if the remaining issues can be addressed.

Quote from: m14u on
https://tools.ietf.org/html/rfc6716
8-12 kbit/s for NB speech,    NB (narrowband)      |           4 kHz |                   8 kHz
16-20 kbit/s for WB speech,    WB (wideband)        |           8 kHz |                  16 kHz
28-40 kbit/s for FB speech,    FB (fullband)        |      20 kHz (*) |                  48 kHz
48-64 kbit/s for FB mono music
64-128 kbit/s for FB stereo music.

With Opus 1.3 it is more like:

16-32 kbit/s for FB speech
32-64 kbit/s for FB stereo speech
48-128 kbit/s for FB stereo music

Mono music has improved a lot too, incidental mono music is okay down to around 24 kbps now IMO.

Re: Personal Listening Test of MP3 and Opus

Reply #13 –
Quote from: Jozape on
[...] more like: [...]
"more like" versus "PROPOSED STANDARD"s "these are the bitrate "sweet spots"" ? You are the speechwriter T.May?

Re: Personal Listening Test of MP3 and Opus

Reply #14 –
Quote from: m14u on
Quote from: Jozape on
[...] more like: [...]
"more like" versus "PROPOSED STANDARD"s "these are the bitrate "sweet spots"" ? You are the speechwriter T.May?

Why the hostility? That RFC is from seven years ago; since then there have been improvements to Opus especially with low bitrate speech. And those "sweet spots" are just someone's opinion.

Re: Personal Listening Test of MP3 and Opus

Reply #15 –
Quote from: Jozape on
[...]are just someone's opinion.
yes, by creators.
Quote from: Jozape on
[...]  have been improvements to Opus [...]
of course, but how much this "improvements" reduce the "comfortable" bitrate? 10% ? 30% ? or 2-5% ?
Quote from: Jozape on
Why the hostility? [...]
where? love (ἕτερος) and peace (healthy).

Re: Personal Listening Test of MP3 and Opus

Reply #16 –
Very interesting test result, thanks for this!