Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Plans for an ABC/HR test (Read 21743 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Plans for an ABC/HR test

Reply #25 –
Quote from: Porcus on
Quote
there's a limit to how much attention
If it was necessary to post that, it surely constitutes "attention".
Don't you think that didn't cross my mind.

Edit: In retrospect, by 'attention,' I actually meant 'credence.'"
• Listen to the music, not the media it's on
• The older, the 'lossier'

Re: Plans for an ABC/HR test

Reply #26 –
Quote from: Porcus on
Quote
there's a limit to how much attention
If it was necessary to post that, it surely constitutes "attention".
Seriously, if [arbitrary person] wants to set up a personal listening test and needs a forum to ask - why not HA?

(As for the suggested settings, I have a hunch they will be too close to transparent for someone who hasn't (yet) trained themself up on the test format. Wouldn't it be an idea to first do a "pilot" to get a gut feeling of what an 1.0 sounds like?)

@Porcus Thank you. I have these high bitrates because of the killer samples, and the reason for the low-end limit is that I apparently confused something so you're right, but what do you recommend as the lowest MP3Enc bitrate? (I remembered, I decided to set the lowest to the lowest one that can be transparent for many samples, but I'm still not sure if 96kbps is that for MP3Enc, so I'll be waiting for recommendations about that.)

Re: Plans for an ABC/HR test

Reply #27 –
Now I think that the best lowest bitrate choice for MP3Enc is 80kbps. I also interpreted this post as a low anchor will be good and I think 20kbps Opus will do the job.

Re: Plans for an ABC/HR test

Reply #28 –
I prepared the encodings, but now should I open a single topic for all tests in the series or should I open a seperate topic for every test in the series? (What I mean by a test consists of 21 testconfig files, so currently only one test is ready (MP3).)

Re: Plans for an ABC/HR test

Reply #29 –
Quote from: Klymins on
And, I really wonder, what is the problem with applying the cutoff in decoding process?
It doesn't make any sense at all. You're going to test encoders, not decoders or DSP. Post-processing files can be useful for reasons such as ensuring the same sampling rate for all decoded files—because ABX tools might dislike two different sampling rates—or adjusting the volume level, which is a common post-processing task. These processing tasks aim to avoid bias: for instance, an encoder that lowers the volume could be easily identified in a blind test due to this reason (hence the file normalisation on the decoding side). Similarly, the ABC/HR tool might introduce artifacts when switching from 44100 Hz to 48000 Hz (hence the resampling).

However, you cannot filter the frequencies of an already encoded file. Doing so to improve or degrade the quality or sound signature is essentially cheating. If one encoder has a strength or weakness compared to others, it is totally unfair to negate this difference through the decoder.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #30 –
I still don't get why is it cheating. I think allowing an encoder to use a higher cutoff point than others do is cheating, am I wrong? (I applied the cutoffs using the SSRC resampler, so they are perfect.)

After all, this test aims for the compression artifacts, not the cutoffs. They must be the same to make the test work for my purpose. (But if an encoder asks for the cutoff point and gives lower than what I told to it, it's not my fault, of course.)

Also "What I mean by a test consists of 21 testconfig files" contains a mistake, the number may change for next tests.

Re: Plans for an ABC/HR test

Reply #32 –
Quote from: Porcus on
Quote from: Klymins on
I still don't get why is it cheating. I think allowing an encoder

In what @guruboolez quotes you on, you wrote "decoder". Did you mean to write "encoder"?

What I mean by allowing an encoder to use a higher cutoff point than others do is allowing an encoder to have its output go unmodified alongside of what I wrote. This was a word mistake.

Re: Plans for an ABC/HR test

Reply #33 –
You are in danger of creating an entirely artificial environment that has no relevance whatever in the real world. You have a fixation on low bitrate/low samplerate that is shared by no one else.

Re: Plans for an ABC/HR test

Reply #34 –
The sample rate will be 44100Hz for this test.

Re: Plans for an ABC/HR test

Reply #35 –
Quote from: Klymins on
I still don't get why is it cheating. I think allowing an encoder to use a higher cutoff point than others do is cheating, am I wrong?
o_O
You're going to compare the speed of a Pentium 3 and a Core i7. But wait a minute, it's unfair because the the core i7 has a much higher clock speed, and much more cores. So you disabled all innovative features because you think it's unfair. But what's the point of the test then? The cutoff point is a part of the audible quality. Period. Forcing one encoder to sound worse is a big failure on the methodology. It makes your test totally useless.

Quote
After all, this test aims for the compression artifacts, not the cutoffs.
Some artifacts are in direct relation with high frequencies (ringing) so no, you don't really have lowpass on one side and artifact on the other.
In the first message you said: "People with untrained hearing will be targeted". Untrained people are usually more sensitive to basic (analog) sound issues (like lowpassing, noise). Artifacts are more advanced. It makes your test even more curious.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #36 –
Quote from: Klymins on
(I applied the cutoffs using the SSRC resampler, so they are perfect.)
So you're resampling the files in order to wipe the higher frequencies out?
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #37 –
I would not be surprised if "weird" results come out anyway, given that the OP's individual preferences over compression artefacts.
Ultimately, if a personal listening test is an exercise to find what suits the OP - by using a reasonable test protocol - then whatever gets the OP's clock ticking is fair game, ... but:

With so many unknown unknowns, I don't think it is a good idea to pull all levers at once. Results will be hard to interpret on all fronts.
Say a codec/setting comes out surprising or maybe suspicious - what could that be due to? With too many deviations from what is "sorta-well-understood", nobody could tell whether it is in additional processing, codecs themselves, OP's hearing, OP's preferences or whatnot.

That's why I suggest a "pilot" test with known stuff, just do fewer for a first attempt. @guruboolez , I don't know if you agree (and besides, do you even remember how it was to be a beginner at this, that must be looong ago  ;D ) - and can input on  what is a sane number for a small initial test, you think? Low anchor, high anchor and two more?
See what you get out of it.

Re: Plans for an ABC/HR test

Reply #38 –
Quote from: guruboolez on
Quote from: Klymins on
I still don't get why is it cheating. I think allowing an encoder to use a higher cutoff point than others do is cheating, am I wrong?
o_O
You're going to compare the speed of a Pentium 3 and a Core i7. But wait a minute, it's unfair because the the core i7 has a much higher clock speed, and much more cores. So you disabled all innovative features because you think it's unfair. But what's the point of the test then? The cutoff point is a part of the audible quality. Period. Forcing one encoder to sound worse is a big failure on the methodology. It makes your test totally useless.

Quote
After all, this test aims for the compression artifacts, not the cutoffs.
Some artifacts are in direct relation with high frequencies (ringing) so no, you don't really have lowpass on one side and artifact on the other.
In the first message you said: "People with untrained hearing will be targeted". Untrained people are usually more sensitive to basic (analog) sound issues (like lowpassing, noise). Artifacts are more advanced. It makes your test even more curious.
I still can't understand you, or I'm not clear enough.

Quote from: guruboolez on
Quote from: Klymins on
(I applied the cutoffs using the SSRC resampler, so they are perfect.)
So you're resampling the files in order to wipe the higher frequencies out?
Yes, and this is because I don't know a lowpass DSP that can apply a sharp lowpass. But I'm aware that using 31000Hz sampling rate may cause the tool to attenuate the high end, so the resampling chain is this: 44100Hz => 62000Hz => 31000Hz => 44100Hz. (62000Hz is there because the resampler weirdly denies some resampling combinations in some situations.)

Quote from: Porcus on
I would not be surprised if "weird" results come out anyway, given that the OP's individual preferences over compression artefacts.
Ultimately, if a personal listening test is an exercise to find what suits the OP - by using a reasonable test protocol - then whatever gets the OP's clock ticking is fair game, ... but:

With so many unknown unknowns, I don't think it is a good idea to pull all levers at once. Results will be hard to interpret on all fronts.
Say a codec/setting comes out surprising or maybe suspicious - what could that be due to? With too many deviations from what is "sorta-well-understood", nobody could tell whether it is in additional processing, codecs themselves, OP's hearing, OP's preferences or whatnot.

That's why I suggest a "pilot" test with known stuff, just do fewer for a first attempt. @guruboolez , I don't know if you agree (and besides, do you even remember how it was to be a beginner at this, that must be looong ago  ;D ) - and can input on  what is a sane number for a small initial test, you think? Low anchor, high anchor and two more?
See what you get out of it.
@Porcus I don't think I fully understood what you said, which combinations should I use for the "pilot" test? And, excuse me but what is the point of high anchor?

Re: Plans for an ABC/HR test

Reply #39 –
Quote
44100Hz => 62000Hz => 31000Hz => 44100Hz

Just to be clear:
you decided to go for 44100 KHz for all MP3 encoders (which seems a fine decision), However, one competitor will be resampled three times after encoding, intentionally degrading the quality?

@Porcus @Klymins : I'm genuinely confused. I don't understand how this test will be organized or its overall purpose. From what I gather, as John33 succinctly put it, it seems artificial, and I doubt anyone here will gain anything meaningful from it.

@Klymins: Make things simple. Choose three or four MP3 encoders, including LAME for a useful comparison. Use default settings where available. Start with 96 kbps and a selection of samples (your current selection is likely fine). Once the test is complete, try another format (e.g., AAC) at a similar bitrate. If there are still contenders, create a third or fourth pool. After finishing, take the best encoder from each test and hold a final competition.

About twenty years ago, I undertook a similar project to form a well-informed opinion on which format and settings might be the best at 80 kbps (don't read the text, just try to get the structure):
https://web.archive.org/web/20160304090658/https://hydrogenaud.io/index.php/topic,35438.0.html
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #40 –
Quote from: guruboolez on
Quote
44100Hz => 62000Hz => 31000Hz => 44100Hz

Just to be clear:
you decided to go for 44100 KHz for all MP3 encoders (which seems a fine decision), However, one competitor will be resampled three times after encoding, intentionally degrading the quality?

@Porcus @Klymins : I'm genuinely confused. I don't understand how this test will be organized or its overall purpose. From what I gather, as John33 succinctly put it, it seems artificial, and I doubt anyone here will gain anything meaningful from it.

@Klymins: Make things simple. Choose three or four MP3 encoders, including LAME for a useful comparison. Use default settings where available. Start with 96 kbps and a selection of samples (your current selection is likely fine). Once the test is complete, try another format (e.g., AAC) at a similar bitrate. If there are still contenders, create a third or fourth pool. After finishing, take the best encoder from each test and hold a final competition.

About twenty years ago, I undertook a similar project to form a well-informed opinion on which format and settings might be the best at 80 kbps (don't read the text, just try to get the structure):
https://web.archive.org/web/20160304090658/https://hydrogenaud.io/index.php/topic,35438.0.html
All competitors are resampled three times. I didn't think that SSRC resampler would degrade the quality, the reason is that resampling all is easier for me.

I chose Helix instead of LAME because I think I read something in this forum saying Helix performs overally better than LAME (there was a problem with the high frequencies but that was sfb21 which is unused with this cutoff).

One of the goals of this test (and maybe the most important one) is finding which bitrate is acceptable for different codecs (and with which encoders).

Re: Plans for an ABC/HR test

Reply #41 –
Alright, if all contenders follow the same decoding pattern, that's an improvement. However, if you feel the need to cut high frequencies, it's likely because you can hear them, which means you're lowering (or changing) the quality. If you don't hear them, there's no need to remove them.
Again, it doesn't make any sense to me.

And if it still make sense to you, just a question: are you going to resample three time your future encodings before listening to them? If not, what's the practical need of your test?
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #42 –
Quote from: guruboolez on
However, if you feel the need to cut high frequencies, it's likely because you can hear them, which means you're lowering (or changing) the quality. If you don't hear them, there's no need to remove them.
I can hear above 15.5kHz but I generally can't detect the difference of 15.5kHz cutoff but I still can't risk keeping these frequencies.

Quote from: guruboolez on
And if it still make sense to you, just a question: are you going to resample three time your future encodings before listening to them? If not, what's the practical need of your test?
I'm not but I don't think these resamplings audibly degrade the quality.

Re: Plans for an ABC/HR test

Reply #43 –
Alright, go ahead and begin your test. You have everything you need.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #44 –
I'm waiting for answers to Reply#28 and Reply#38 for some more time.

Re: Plans for an ABC/HR test

Reply #45 –
28: I'd say "keep it here", because I have a hunch that after the first test you should evaluate. 

38: I know you summoned me, but I am not quite sure, so I asked others for help.
But since you have not done this before, I would first try to find settings where the codecs/settings score different to your ears. Putting up a BIG test to risk that everything gets the same, is ... in that scenario, too boring to stay committed to it? Maybe?

Re: Plans for an ABC/HR test

Reply #46 –
#28: single topic
#38: I second Porcus suggestion.
Wavpack Hybrid -c4hx6

Re: Plans for an ABC/HR test

Reply #47 –
Okay, so I'm starting this test as a personal test with a plan of making it public if I see I can succesfully do it.

Re: Plans for an ABC/HR test

Reply #48 –
The tool has tons of bugs. It created empty config files multiple times. I'm trying again.