Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Plans for an ABC/HR test (Read 22258 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Plans for an ABC/HR test

Hello. I want to start an ABC-HR test, but am having trouble making some decisions. Actually, I was started doing the test with just one sample at mind some time ago, but I ended up cancelling it as there were 60 different encodings, so I'm here.

These are the decisions I could make:
  • People with untrained hearing will be targeted (bitrate selection will be done accordingly).
  • The encodings will be CBR (at least when possible) because I strongly think that VBR is unfair for testing. Also the VBR algorithms of many encoders give nonsense (same or higher average bitrate for trebleless or very silent moments).
  • The cutoff point used will be somewhere between ~14kHz and 16kHz and it will be the same for all encodings unless 29400Hz sampling rate is used.

The first decision I couldn't make is which types of samples will be used (options are music, speech, SFX, and other - music will certainly be included).

(The four decisions starting with this directly affect the number of encodings being tested per sample, and this is important because too many encodings to test per sample makes effectively doing the test practically impossible. This should be kept in mind.)

Another decision that I couldn't make is which codecs will be used. I'm certain that MP3 should be included, but I'm not certain about which ones should be included from the following list (this will also be affected by the types of samples chosen):
  • IMA/Flash ADPCM
  • Microsoft ADPCM
  • QOA (+unofficial BPSs)
  • Nellymoser Asao
  • Speex
  • Vorbis
  • AAC-LC
  • AAC-HEv1
  • AAC-HEv2
  • Opus
  • USAC
  • USAC+SBR
  • USAC+SBR+PS

And, another important decision that I couldn't make: Which encoders will be used? (There may be more than one for one codec.)
  • IMA/Flash ADPCM: I think that the encoder used in Flash is either the same as the standard one (higher probability) or more advanced than it, which makes it one of the two options with the other being ADPCM-XQ. But I don't think ADPCM-XQ supports bit depths other than 4bps.
  • Microsoft ADPCM: I don't even know what are the available encoders.
  • QOA: I don't know whether we can get noise shaping with 2bps and 1bps options or not. Also, it'll be good if we can get 4bps.
  • Nellymoser Asao: I don't know a way to encode other than using Flash.
  • Speex: I don't know anything other than the reference encoder.
  • MP3: I think I get the best results with MP3Enc for bitrates below 160kbps compared to Helix and LAME. Fastenc is probably better but I need to know more about its bugs, and I didn't test L3Enc and MP3 Surround at all in a way that works for this purpose. Which encoders can encode CBR at non-standard bitrates and which can't is also an important factor in the decision (MP3Enc and Fastenc can, I didn't test with others). The reason that I wrote "for bitrates below 160kbps" is that MP3Enc doesn't use joint stereo anymore from this bitrate on, and please write if you know how to override this.
  • Vorbis: I don't know anything that can work better than aoTuV.
  • AAC: I think we have two options: FDK-AAC and QAAC. QAAC is said to perform better, but it's also said to have a bug inserting momentary spaces when used with bitrates below 192kbps in a topic that I couldn't find, also it doesn't support using a higher cutoff point than the default which is a problem for low bitrates, and it can't encode AAC-HEv2 as far as I know.
  • Opus: I don't know anything that can work better than libopus.
  • USAC: It seems that Exhale is the best for its bitrates that it can give and FhG is the best otherwise (I don't know anything other) but as far as I know Exhale doesn't support CBR and this is a problem.

The sampling rate also needs to be decided (there can be more than one per codec). Some encoders are tuned better for 44100Hz, but 32000Hz makes the block lengths better for MP3 and AAC-LC, so that's hard to decide for me. And, things get even harder if there is a possibility that the "Auto" option of EZ CD Audio Converter doesn't work well with USAC.

And, of course, the bitrates should also be selected. I want to include a wide range of bitrates but of course care needs to be taken to keep the encodings per sample at a suitable level as mentioned before.

Prior to the last, the cutoff point. It should be low enough to make the most of the range heard, but if it's too low, then the loss of the highs becomes obvious and this doesn't reflect what people can use for high audible quality. This is especially hard because the highest frequency that can be heard inside a normal sound is different for different people. Also, how and when the cutoff process will be applied for encoders that don't allow setting the cutoff point needs to be decided too. I discovered that applying the cutoff before the encoding process reduces the quality so much for USAC (probably for HE-AAC too) but I don't know what is the best otherwise.

And, lastly, the test samples themselves. I need to get more test sample ideas to select from. For music, I target modern music, and I mostly have no problems with including killer samples. But I want to avoid using musics that I really like for testing either partially or completely as the musics that I test become boring to me either partially or completely. I also think different parts of the same music can be used as different test samples.

I want help for making these decisions (and getting more test sample ideas). If you have something to suggest that I didn't write about, go ahead. Thanks in advance.

Re: Plans for an ABC/HR test

Reply #2 –
Hi Klymins,

If you plan to do a personal listening test :
— try to include at least 10 different samples
— try to limit the number of challengers. You're obviously interested by a wide choice of formats, many of them being obsolete or forgotten. I'd suggest you to restrict them. You can safely discard Speex (it's more or less a draft for Opus), all ADPCM, Nellymoser Asao (you win the 2025 Award for finding the most obscure encoder). QOA is for very high bitrate only (~280 kbps for 16/44). Vorbis is good, but Opus is the improved version of Vorbis.

Now you have: MP3, Opus, AAC (LC/HE/HEv2 depending on targeted bitrate), USAC (for very low bitrate use Fraunhofer's version).
Only if you plan to test high bitrate, you may add: lossyWAVed files, Wavpack, DualStream (Optimfrog lossy), MPC, QOA.

My top recommendation: begin by identifying the desired bitrate, and then seek out suitable options.

___
• Microsoft ADPCM: I can apparently use this encoder with Adobe Audition v3 (very old but still working version)
• The latest version of adpcm-xq also includes 2-bit, 3-bit and 5-bit ADPCM (source: https://github.com/dbry/adpcm-xq)
• if you plan to test MP3 and to discard both LAME and HELIX, it won't interest many people here (and it will probably discredit the whole test). Don't shoot yourself in the foot!
• there are two other AAC encoder you may consider: FHG and Nero (I won't recommend the latter but it supports HEv2)
• Discarding VBR won't interest many people here. If you force CBR with some encoders and keep VBR with other because VBR can't be disabled, be sure that most readers will consider it as unfair. If not worse.

Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #3 –
Firstly, thank you very much for such a long answer.

I'm thinking about making this a partially or fully public test.

Quote from: guruboolez on
— try to include at least 10 different samples
I will, do you have any specific samples to suggest?

Quote from: guruboolez on
Speex (it's more or less a draft for Opus) ... Nellymoser Asao (you win the 2025 Award for finding the most obscure encoder)
These work much better with speech than music. I don't agree with what you wrote about speex because it performs much different from Opus (Speex uses CELP and Opus uses CELT, this confused me previously). And Nellymoser Asao, which is very well for speech, is used in Flash under the name "Speech", that's why I know it.

Quote from: guruboolez on
Vorbis is good, but Opus is the improved version of Vorbis.
I don't agree with that too. They perform very differently and I don't think they have any significant similarity at all.

Quote from: guruboolez on
My top recommendation: begin by identifying the desired bitrate, and then seek out suitable options.
I don't have a desired bitrate. I want to see which bitrates get what scores with different codecs.

Quote from: guruboolez on
• The latest version of adpcm-xq also includes 2-bit, 3-bit and 5-bit ADPCM (source: https://github.com/dbry/adpcm-xq)
Is it Shockwave Flash ADPCM?

Quote from: guruboolez on
• if you plan to test MP3 and to discard both LAME and HELIX, it won't interest many people here (and it will probably discredit the whole test). Don't shoot yourself in the foot!
I'll keep that in mind.

Quote from: guruboolez on
• there are two other AAC encoder you may consider: FHG and Nero (I won't recommend the latter but it supports HEv2)
I don't think I should include FhG and Nero AAC encoders because FDK is said to perform better than FhG all the time and you didn't recommend Nero.

Quote from: guruboolez on
• Discarding VBR won't interest many people here. If you force CBR with some encoders and keep VBR with other because VBR can't be disabled, be sure that most readers will consider it as unfair. If not worse.
You're right. I hope CBR mode gets added to Exhale.

Do you have any recommendations for bitrates? I want the lowest bitrates to be easily noticeable with almost all samples and the highest ones to be transparent for most samples (to people with untrained hearing).

I think I should decide to whether include the obsolete/exotic codecs or not depending on the number of encodings.

Thanks again.

Edit: What the last post on https://hydrogenaud.io/index.php/topic,127266.0.html says (mine) is still present, might be a problem.

Edit 2: This test will entirely be in stereo, I forgot to write this.

Re: Plans for an ABC/HR test

Reply #4 –
For samples, why not using the same set as Kamedo2?
https://hydrogenaud.io/index.php/topic,98003.msg815018.html
There's no working links for IgorC's samples list but I can post them if you're interested.

____

I wonder where you get this information that FHg AAC is always worse than FDK.
https://hydrogenaud.io/index.php/topic,120062.0.html


It is right that I don't recommend Nero AAC (but I don't recommend either Speex over Opus, Vorbis over Opus, Fhg MP3 over LAME, etc...). My conclusion was related to LC-AAC. If you want to check ultra-low bitrate with AAC, I would consider Nero as it supports HEv2 profile.

____

Quote
Do you have any recommendations for bitrates? I want the lowest bitrates to be easily noticeable with almost all samples and the highest ones to be transparent for most samples (to people with untrained hearing).
I'm still not sure to understand what you want. 8 kbps should be easily noticeable and 640 kbps should be transparent everywhere. But I'm quite sure its not the answer you're expecting.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #5 –
I think I should take a look at IgorC's samples.

Quote from: guruboolez on
I wonder where you get this information that FHg AAC is always worse than FDK.
I don't remember but apparently it's wrong.

Quote from: guruboolez on
Quote
Do you have any recommendations for bitrates? I want the lowest bitrates to be easily noticeable with almost all samples and the highest ones to be transparent for most samples (to people with untrained hearing).
I'm still not sure to understand what you want. 8 kbps should be easily noticeable and 640 kbps should be transparent everywhere. But I'm quite sure its not the answer you're expecting.
I want the lowest bitrate to be somewhere between the highest that is easily noticeable for almost all samples and the highest that is annoying for almost all samples, and the highest one to be about the lowest that is transparent for almost all samples.

I think I need specific and detailed informations and/or recommendations about encoders.

Edit: I forgot to say that I will certainly test Sonic Boom from Sonic CD (by Pastiche). Its leitmotif(s) apparently is a real killer sample for SBR. And I'm planning to test some other parts of it too (I almost didn't test it other than the first leitmotif). But I'm suspicious about it being really lossless (I got it from IA).

Re: Plans for an ABC/HR test

Reply #6 –
I think I should divide this test because I certainly want to test too many encodings. I'm planning to make a listening test of MP3 first. @guruboolez I'm preparing the samples, can you post IgorC's samples?

Re: Plans for an ABC/HR test

Reply #7 –
@Klymins : samples are here.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #8 –
Much thanks!

Re: Plans for an ABC/HR test

Reply #10 –
Quote from: Klymins on
  • Nellymoser Asao: I don't know a way to encode other than using Flash.
  • Speex: I don't know anything other than the reference encoder.
Afaik ASAO is supported in ffmpeg, but it's only mono anyways. Speex has a very rudimentary form of intensity stereo, that tends to create clipping on modern music and anything loud. Only way to avoid clipping with speex is lowering volume by ~5db. Anyway, its stereo is crap, and VBR/ABR is even worse. I wouldn't use it as it reaches too low bitrates most of the times ruining the audio quality. So it's just not worth trying it for a fair comparison. IMO these are too irrelevant for modern times, and as more codecs you add, more long, complex and boring is the test.

BTW I have my own plans for listening tests, but I could consider joining this attempt. :>

Re: Plans for an ABC/HR test

Reply #11 –
Quote from: a.ok.in on
Quote from: Klymins on
  • Nellymoser Asao: I don't know a way to encode other than using Flash.
  • Speex: I don't know anything other than the reference encoder.
Afaik ASAO is supported in ffmpeg, but it's only mono anyways. Speex has a very rudimentary form of intensity stereo, that tends to create clipping on modern music and anything loud. Only way to avoid clipping with speex is lowering volume by ~5db. Anyway, its stereo is crap, and VBR/ABR is even worse. I wouldn't use it as it reaches too low bitrates most of the times ruining the audio quality. So it's just not worth trying it for a fair comparison. IMO these are too irrelevant for modern times, and as more codecs you add, more long, complex and boring is the test.

BTW I have my own plans for listening tests, but I could consider joining this attempt. :>

Much thanks for information. Talking about clipping, I think it's worth saying I detected that USAC can go several dBFS above the input.

I'm currently confused by some samples of @Kamedo2 and IgorC being not homogenous. This can easily cause one encoder to perform better than another for the first part and the opposite for the second part. Is there a problem with cropping them?

Re: Plans for an ABC/HR test

Reply #12 –
I prepared the samples. For the test of MP3 (there will be multiple tests with this being the first), I thought these samples and orders would be good:

Erase/Rewind is by The Cardigans, and Sonic Heroes and This Machine are from Sonic Heroes. The others are from Kamedo2's and IgorC's sets. Feel free to write if you have something to recommend. (I'm planning to change the order and include many more samples too for tests involving Opus or profiles using SBR.)

And, about the encodings. I want to include Helix for [96,112,128]kbps, MP3Enc for [96,112,128]kbps, and MP3 Surround for 128kbps; but I'm not sure about should I include Helix for 160kbps, MP3Enc for 144kbps, MP3 Surround for 144kbps, and MP3 Surround for 160kbps (I'll be happy to see suggestions about that). The lower limit of 96kbps is also open to discuss.

Then, I still can't decide about the sampling rate (32000Hz or 44100Hz, for Helix and MP3Enc) so I'll be happy to see suggestions about that too (using the same sampling rate for all encoders is not required - I'd prefer one sampling rate per encoder to not have too many encodings to score, and I can consider making some pre-tests for that).

I also want to say that I'm planning to use 15500Hz as the cutoff point.

I'll be happy to see suggestions about especially the mentioned decisions. Thanks again.

Edit: I forgot to say that I'm planning to discard the first second for all samples and for all tests.

Re: Plans for an ABC/HR test

Reply #13 –
If no one answers for some more time, I'll go with Helix@[112,128,160]+MP3Enc@[96,112,128]+MP3S[128,144] with all of them being 44100Hz.

Re: Plans for an ABC/HR test

Reply #14 –
It's your test and your selections, so you are free to test whatever you choose. But to recap:
— MPE3enc: https://www.rarewares.org/rrw/mp3enc.php Last version released 28 years ago in 1998.
— MP3 Surround: https://en.wikipedia.org/wiki/MP3_Surround  Released in 2004 and apparently created for 5.1 channels
— Helix: recently maintained, so choice seems ok
— LAME MP3: any reason for not choosing the most popular MP3 encoder here?

Quote
I also want to say that I'm planning to use 15500Hz as the cutoff point.
Again, your choice, your rules. But any reason for this? Is it even possible to force MP3 Surround or MP3Enc to lowpass the audio signal?
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #15 –
Quote from: guruboolez on
— LAME MP3: any reason for not choosing the most popular MP3 encoder here?

I totally have a reason. One of my recent encodes from a clean flac file ended up having an audible "POP" sound in it. I am still trying to remember what track it was (i discarded it and cant remember). If i can remember i will share it in the forum.

Re: Plans for an ABC/HR test

Reply #16 –
Quote from: guruboolez on
It's your test and your selections, so you are free to test whatever you choose. But to recap:
— MPE3enc: https://www.rarewares.org/rrw/mp3enc.php Last version released 28 years ago in 1998.
— MP3 Surround: https://en.wikipedia.org/wiki/MP3_Surround  Released in 2004 and apparently created for 5.1 channels
— Helix: recently maintained, so choice seems ok
— LAME MP3: any reason for not choosing the most popular MP3 encoder here?

Quote
I also want to say that I'm planning to use 15500Hz as the cutoff point.
Again, your choice, your rules. But any reason for this? Is it even possible to force MP3 Surround or MP3Enc to lowpass the audio signal?
Thanks for informations. It's possible in MP3Enc with the -bw switch. It's not possible in MP3 Surround but I'll apply the cutoff in the conversion from MP3 to WAV. The reason is that keeping very high frequencies that can be masked easily and/or have to be loud to be heard is not fair as the degradation of these may not be heard. I'm also trying to not have too many encodings.

Quote from: lovecraft on
Quote from: guruboolez on
— LAME MP3: any reason for not choosing the most popular MP3 encoder here?

I totally have a reason. One of my recent encodes from a clean flac file ended up having an audible "POP" sound in it. I am still trying to remember what track it was (i discarded it and cant remember). If i can remember i will share it in the forum.
Thanks for information.

Re: Plans for an ABC/HR test

Reply #17 –
Quote from: lovecraft on
Quote from: guruboolez on
— LAME MP3: any reason for not choosing the most popular MP3 encoder here?
I totally have a reason. One of my recent encodes from a clean flac file ended up having an audible "POP" sound in it. I am still trying to remember what track it was (i discarded it and cant remember). If i can remember i will share it in the forum.

Your response might need to be taken with a grain of salt. An isolated issue does not warrant excluding a candidate. LAME has been used and partially developed by this community. Various intricate problems have been reported and addressed here. Additionally, I have purchased music from platforms like Amazon and Qobuz that was distributed in MP3 using LAME. This codec is a benchmark, used knowingly or unknowingly by millions of people worldwide for about fifteen years, and distributed by multinational companies. I would like to understand a good reason for not including this reference encoder and for digging up software abandoned for 30 years. Even if a major bug was identified with the current version of LAME, it would be conceivable to use a previous version. But here, I do not understand the point of the test. I would find it somewhat amusing to use only old software and conduct a test of codecs from the 90s. I would understand not including LAME or using the version from that era. But here, I admit my incomprehension and inability to grasp the interest and rationale behind this peculiar selection.


Quote
It's not possible in MP3 Surround but I'll apply the cutoff in the conversion from MP3 to WAV.
I just see another red flag in your upcoming test… You might consider using these tools at their default settings.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #18 –
@guruboolez Using which tools at their default settings? MP3 to WAV decoder (which is going to be foobar2000)?

Re: Plans for an ABC/HR test

Reply #19 –
Quote from: Klymins on
@guruboolez Using which tools at their default settings? MP3 to WAV decoder (which is going to be foobar2000)?
Use encoders at their default settings is probably a wise choice. If something is wrong with an encoder (remember: it's old stuff), it won't be your problem. And if you can't adjust the lowpass during the encoding, don't ever try to simulate it during the decoding process.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #20 –
Quote from: guruboolez on
Quote from: Klymins on
@guruboolez Using which tools at their default settings? MP3 to WAV decoder (which is going to be foobar2000)?
Use encoders at their default settings is probably a wise choice. If something is wrong with an encoder (remember: it's old stuff), it won't be your problem. And if you can't adjust the lowpass during the encoding, don't ever try to simulate it during the decoding process.
But I need to make sure that they have the same cutoff point. And, I really wonder, what is the problem with applying the cutoff in decoding process?

Re: Plans for an ABC/HR test

Reply #21 –
I have a feeling this will be so botched test that it will make sense as his table he posted some time ago.
"This encoder works so bad it sounds so good" type of test.

Klymins, what will be the purpose of your test, what exactly are you testing? I see you're mentioning weird sample rates, so I am assuming you're testing... what, exactly? What kind of audio is that? Not music albums, or higher res music, 44.1, 48 and up.
TAPE LOADING ERROR

Re: Plans for an ABC/HR test

Reply #22 –
Quote from: itisljar on
I have a feeling this will be so botched test that it will make sense as his table he posted some time ago.
"This encoder works so bad it sounds so good" type of test.

Klymins, what will be the purpose of your test, what exactly are you testing? I see you're mentioning weird sample rates, so I am assuming you're testing... what, exactly? What kind of audio is that? Not music albums, or higher res music, 44.1, 48 and up.

It's true that I was shocked by many Flash content using 11025Hz musics and left fullband for a time, but this is not the case anymore. I will continue using 11025Hz MP3's for Flash stuff (as creating or modifying) but I returned to fullband for standalone audio and this test will be in fullband too (per the 15500Hz cutoff point) (or super-wideband, depending on how you look at it). And the audio to be tested is some of Kamedo2's samples + some of IgorC's ones + some of mines.

Re: Plans for an ABC/HR test

Reply #23 –
Quote from: itisljar on
Klymins, what will be the purpose of your test, what exactly are you testing? 
Hmm... by the looks of 9 out of 10 of his ramblings, erm, posts, his own ego perhaps?

Also, given the timing, I can't help but see this as a Bizarro-vs-Superman type comparison, attempting to ride the momentum of that much-anticipated (and, unlike this one, genuinely useful) Wavpack test by @guruboolez



I'm sorry @Klymins, it's nothing personal. It's just that there's a limit to how much attention people will give to someone who comes across like that guy in the speaker's corner shouting that he's the new chosen one.
The older the 'lossier' - meaning: my hearing & my music collection.
After all, I listen to the music, not the media it's on.

Re: Plans for an ABC/HR test

Reply #24 –
Quote
there's a limit to how much attention
If it was necessary to post that, it surely constitutes "attention".
Seriously, if [arbitrary person] wants to set up a personal listening test and needs a forum to ask - why not HA?

(As for the suggested settings, I have a hunch they will be too close to transparent for someone who hasn't (yet) trained themself up on the test format. Wouldn't it be an idea to first do a "pilot" to get a gut feeling of what an 1.0 sounds like?)