HydrogenAudio

Lossy Audio Compression => MP3 => MP3 - Tech => Topic started by: timcupery on 2010-01-23 20:08:12

Title: Lame 3.98 wastes bits, esp. at V0
Post by: timcupery on 2010-01-23 20:08:12
So when I discovered Mp3packer awhile ago, and tested it on many of my mp3's, I discovered that mp3's encoded with Lame did not meaningfully compress. Lame mp3's, whether VBR, ABR or even most CBR files would decrease in size by 0% or 0.1%.

This has changed with Lame 3.98. I've noticed a few high-bitrate ABR encodes (from Amazon) will compress somewhat using MP3packer, but I've especially noticed with -V0 encodes (from eMusic). Albums encoded using Lame 3.98 -V0 will compress 2-3%, sometimes more, when run through Mp3packer.
Which means that Lame 3.98 has significant wasted bits upon encoding.

I did some quick testing, encoding a bunch of FLAC files with Lame 3.98, using V4 through V0, and then testing how much the filesize decreased when run through Mp3packer. All of the FLAC files were relatively short (to speed up my testing process), and all were ripped from CD and have full-frequency range up to 22.05 khz. I picked a range of sounds - some applause at concerts, some voice, some metal, some quirky instrumental.

Here are my results, showing the % decrease resulting from Mp3packer, for each file at each VBR setting.
I also give the average initial bitrate (pre-Mp3packer) for each VBR setting, as well as the average filesize decrease (with each file counting equally) and the total filesize decrease (in which longer and otherwise larger files count more).
edit: Note that the files with the greatest decrease in size by Mp3packer processing, tend to be the ones that start out with the highest bitrates.

(http://www.unc.edu/~cupery/pics/audio/Lame_3.98_shrinking.gif)

I will continue to run Lame 3.98 -V0 encodes (from eMusic) through Mp3packer, but probably won't worry about it for my own encodes (usually V2 or V3).
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-23 20:27:33
The --strictly-enforce-iso switch got forced on with LAME 3.98. Since they found out that a horrid and widely used software player had problems with 320kbps without LAME being 100% ISO.

The switch will limit the frame size of the codec which makes 320kbps Mp3 barely store and use bits from the bit reservoir. Also the switch will very likely to cause the extra wasted bloat with V0 since that uses a alot of 320 frames.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-01-23 20:36:34
Inefficiency results from the way 320 kbps frames are used. That's why the very high quality settings are effected most.
With 320 kbps frames the mp3 specs aren't very clear.
There is the practical problem that the FhG mp3 decoder that comes with WMP interprets this unclear mp3 spec in a pretty unpleasant way, so the Lame devs have chosen a very defensive way of using 320 kbps frames which looses efficiency (and the very last bits of audio data space).
From own test with mp3packed mp3s and WMP however the Lame devs have done a bit more than necessary for obeying to the FhG decoder.
I never ran upon a problem with mp3packed files (okay: I used WMP only for testing).

I use a batch script from within foobar that generally uses mp3packer after Lame.
As mp3packer is so fast it's a no-issue to me.
For downloaded mp3s I have a context menu entry for folders which uses mp3packer for all mp3 files in the folder.

OOps, /mnt was faster.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-01-23 20:50:30
I tested LAME 3.98.2 in --abr mode. Here's the graph: resulting bitrate vs. requested, and bitrate after mp3packer.

(http://img715.imageshack.us/img715/229/clipboard01a.png)
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-23 21:02:23
Sadly the ISO frame size limit on LAME 3.98.2 causes regression with 320kbps CBR.

An good example with a problem track:

LAME 3.97 vs LAME 3.98.2 320kbps CBR

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 20:48:47

File A: C:\Temp\The Robots 320 CBR (LAME 3.97).mp3
File B: C:\Temp\The Robots 320 CBR (LAME 3.98.2).mp3

20:48:47 : Test started.
20:49:16 : 01/01  50.0%
20:49:25 : 02/02  25.0%
20:49:38 : 03/03  12.5%
20:49:54 : 04/04  6.3%
20:50:11 : 05/05  3.1%
20:50:23 : 06/06  1.6%
20:50:43 : 07/07  0.8%
20:50:57 : 08/08  0.4%
20:51:09 : 09/09  0.2%
20:51:26 : 10/10  0.1%
20:51:31 : Test finished.

 ----------
Total: 10/10 (0.1%)

LAME 3.98 sounds the worst, due to its harsher pre-echo artifacts. LAME 3.97 sorta cheats since it can take advantage of it's larger frame size by storeing more bits on the bit reservoir.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: timcupery on 2010-01-23 21:13:48
Thanks for the explanation. Makes sense why higher-bitrate files are disproportionately affected.

The whole situation is a bit disappointing. I take it WMP is the "widely used player" that has trouble with Lame 320kbps frames that aren't perfectly ISO? or is iTunes a culprit also? Did the problem exist anywhere else also? (e.g., hardware - if WMP is affected, then in Zune affected also?)

Is there any way in Lame 3.98 to turn off the ISO compliance switch?
Or plans in 3.99 to do things differently?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-01-23 21:16:22
Sadly the ISO frame size limit on LAME 3.98.2 causes regression with 320kbps CBR.
...LAME 3.98 sounds the worst, due to its harsher pre-echo artifacts. ...

Do you mind trying CBR 256 and ABR 270 (or similar)?
As bit reservoir is used with these settings there may be a chance that audio data amount at the critical spots is higher than when using CBR 320.

Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-23 21:27:19
I take it WMP is the "widely used player" that has trouble with Lame 320kbps frames that aren't perfectly ISO? or is iTunes a culprit also?


Yep, WMP uses a FhG decoder that had the problem with LAME's 320 frames. Am not sure about iTunes though, since i've never used 320kbps since i find that bitrate to be inefficient.

Is there any way in Lame 3.98 to turn off the ISO compliance switch?
Or plans in 3.99 to do things differently?


Sadly codeded to be forced on .

At least the LAME devs just limit the frame size on 320 frames in VBR. While FhG decided to not use 320 frames in VBR with their encoder (32 - 256 frames only).
Title: Lame 3.98 wastes bits, esp. at V0
Post by: timcupery on 2010-01-23 21:46:21
In what way was the ISO compatibility of 320kbps frames "forced" on Lame? Was it a decision that Lame devs made for the sake of general compatibility "out there" or was there pressure from outside sources (e.g., Microsoft, FhG)?

It's surprising because Lame has been Mp3packer-efficient (and therefore presumably not enforcing ISO compatibility on 320kbps frames) prior to 3.98.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-01-23 21:55:15
The most restrictive way of interpreting the mp3 specs for 320 kbps frames is not to use bit reservoir (IIRC the exact limitation is towards buffer size used for 320 kbps frames, but it ends up having bit reservoir forbidden. Buffer size limitation doesn't make any sense at all, especially as it addresses implementation, not format [only as a consequence], and it's only about small buffers anyway, but that's the way it is).
This is what current Lame does.

I can't imagine there was pressure on the Lame devs from outside sources. I think they just want to make sure that Lame encoded files are guaranteed to be usable with WMP because an mp3 file produced by someone who knows what he's doing can make its way to average Joe.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-23 21:58:55
Sadly the ISO frame size limit on LAME 3.98.2 causes regression with 320kbps CBR.
...LAME 3.98 sounds the worst, due to its harsher pre-echo artifacts. ...
Do you mind trying CBR 256 and ABR 270 (or similar)?
As bit reservoir is used with these settings there may be a chance that audio data amount at the critical spots is higher than when using CBR 320.

I've tried out 256 and 270 ABR (I got a 285 encode) with a more critical sample:

LAME 3.98.2 256 vs 320


Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 21:31:45

File A: C:\Temp\musique_non_stop 256 CBR.mp3
File B: C:\Temp\musique_non_stop 320 CBR.mp3

21:31:45 : Test started.
21:32:26 : 00/01  100.0%
21:32:52 : 01/02  75.0%
21:33:13 : 02/03  50.0%
21:33:52 : 03/04  31.3%
21:34:21 : 04/05  18.8%
21:34:44 : 04/06  34.4%
21:35:00 : 05/07  22.7%
21:35:25 : 06/08  14.5%
21:35:47 : 07/09  9.0%
21:36:02 : 08/10  5.5%
21:36:24 : 09/11  3.3%
21:36:47 : 10/12  1.9%
21:36:50 : Test finished.

 ----------
Total: 10/12 (1.9%)

The 256 encode sounds the worse, but the 320 version sounds only a little bit better.

LAME 3.98.2 270 ABR vs 320 CBR

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 21:38:52

File A: C:\Temp\musique_non_stop 270 ABR.mp3
File B: C:\Temp\musique_non_stop 320 CBR.mp3

21:38:52 : Test started.
21:39:33 : 00/01  100.0%
21:40:03 : 01/02  75.0%
21:40:14 : 02/03  50.0%
21:40:23 : 03/04  31.3%
21:40:41 : 04/05  18.8%
21:40:55 : 05/06  10.9%
21:41:08 : 06/07  6.3%
21:41:34 : 07/08  3.5%
21:41:44 : 08/09  2.0%
21:41:58 : 09/10  1.1%
21:42:23 : 10/11  0.6%
21:42:31 : 11/12  0.3%
21:42:39 : 12/13  0.2%
21:42:49 : 13/14  0.1%
21:42:58 : 14/15  0.0%
21:43:01 : Test finished.

 ----------
Total: 14/15 (0.0%)

The 270 ABR versions is at 285kbps, sadly it sounds worse then 320kbps.

LAME 3.98.2 256 CBR vs 270 ABR

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 21:53:36

File A: C:\Temp\musique_non_stop 256 CBR.mp3
File B: C:\Temp\musique_non_stop 270 ABR.mp3

21:53:36 : Test started.
21:53:53 : 01/01  50.0%
21:53:59 : 02/02  25.0%
21:54:05 : 03/03  12.5%
21:54:14 : 04/04  6.3%
21:54:21 : 05/05  3.1%
21:54:29 : 06/06  1.6%
21:54:37 : 07/07  0.8%
21:54:50 : 08/08  0.4%
21:54:59 : 09/09  0.2%
21:55:12 : 10/10  0.1%
21:55:20 : 11/11  0.0%
21:55:29 : 12/12  0.0%
21:55:30 : Test finished.

 ----------
Total: 12/12 (0.0%)

The 256 encode has horrid precho at 0:04, while the 270 version is more clearer but still very bad.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-01-23 22:02:31
That's pretty difficult for me to ABX...
/mnt, can you test this encoding vs. 3.97?

[attachment=5667:the_robots.mp3]
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-01-23 22:13:23
I've tried out 256 and 270 ABR (I got a 285 encode) with a more critical sample: ...

Thank you, /mnt, for testing.
So there's no hope for improving the situation with settings like that.

We should keep in mind however that mp3 isn't perfect with serious pre-echo samples (and you are probably more sensitive to pre-echo than most of the HA members).
Anyway I'd also prefer if Lame had at least a switch not to use this limitation for 320 kbps frames.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: robert on 2010-01-23 22:47:34
The following thread tells the reason for the changed behaviour:
http://www.hydrogenaudio.org/forums/index....showtopic=40308 (http://www.hydrogenaudio.org/forums/index.php?showtopic=40308)
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-23 22:58:03
That's pretty difficult for me to ABX...
/mnt, can you test this encoding vs. 3.97?

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 22:24:33

File A: C:\Temp\the_robots 320 CBR LAME 3.97.mp3
File B: C:\Downloads\the_robots.mp3

22:24:33 : Test started.
22:24:59 : 00/01  100.0%
22:25:04 : 01/02  75.0%
22:25:12 : 01/03  87.5%
22:25:22 : 02/04  68.8%
22:26:02 : 03/05  50.0%
22:26:08 : 04/06  34.4%
22:26:14 : 05/07  22.7%
22:26:25 : 06/08  14.5%
22:26:30 : 07/09  9.0%
22:26:38 : 08/10  5.5%
22:26:46 : 09/11  3.3%
22:26:52 : 10/12  1.9%
22:27:00 : 11/13  1.1%
22:27:06 : 12/14  0.6%
22:27:20 : 13/15  0.4%
22:27:38 : 14/16  0.2%
22:27:51 : 15/17  0.1%
22:28:05 : 16/18  0.1%
22:28:15 : 17/19  0.0%
22:28:25 : 18/20  0.0%
22:28:34 : Test finished.

 ----------
Total: 18/20 (0.0%)

Your encoded version sounds flat at 0:17.5 - 0:20, which is due to smearing artifacts.

I've tried out 256 and 270 ABR (I got a 285 encode) with a more critical sample: ...
Thank you, /mnt, for testing.
So there's no hope for improving the situation with settings like that.


I don't think theres much hope on those situations. At least those situations are rare, unless the user is a huge fan of: electronic music, castanets, drums / cymbals (at low bitrates) and Kraftwerk.

The timing resolution on the Mp3 specs limit completely shrinks the "room for improvement" for Mp3. IMO it's the best you can ever get out of Mp3 with critical pre-echo samples, while AAC still has some room for improvement.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-01-23 23:27:34
Your encoded version sounds flat at 0:17.5 - 0:20, which is due to smearing artifacts.

Thank you.
Basically, it was encoded with LAME 3.98.2 with buffer restriction removed (it's just my test compile).
So buffer restriction isn't the root of all evil. 
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-01-23 23:28:02
The following thread tells the reason for the changed behaviour:
http://www.hydrogenaudio.org/forums/index....showtopic=40308 (http://www.hydrogenaudio.org/forums/index.php?showtopic=40308)

IIRC: this restricts buffer size, and as a consequence bit reservoir usage was already restricted in 3.97. But it was restricted to a minor extent than it's done with 3.98.
But as lvqcl has shown there's also not too much hope in improving things by removing the restriction. Maybe it's more important to have a playback guarantee even with the most restrictive decoders.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-23 23:56:51
Your encoded version sounds flat at 0:17.5 - 0:20, which is due to smearing artifacts.
Thank you.
Basically, it was encoded with LAME 3.98.2 with buffer restriction removed (it's just my test compile).
So buffer restriction isn't the root of all evil. 

Hmm, i wounder what it is causing the regression on 3.98 then.

I've done a re-test of that sample and got the same results:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 23:37:48

File A: C:\Temp\the_robots 320 CBR LAME 3.97.mp3
File B: C:\Downloads\the_robots.mp3

23:37:48 : Test started.
23:38:22 : 01/01  50.0%
23:38:35 : 02/02  25.0%
23:38:52 : 03/03  12.5%
23:39:08 : 04/04  6.3%
23:39:19 : 05/05  3.1%
23:39:26 : 06/06  1.6%
23:39:33 : 07/07  0.8%
23:40:09 : 07/08  3.5%
23:40:17 : 08/09  2.0%
23:40:40 : 09/10  1.1%
23:41:01 : 10/11  0.6%
23:41:10 : 11/12  0.3%
23:41:23 : 12/13  0.2%
23:41:37 : 13/14  0.1%
23:42:00 : 14/15  0.0%
23:42:02 : Test finished.

 ----------
Total: 14/15 (0.0%)

Anyway there is another sample that sounds worse with 3.98 at 320kbps:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/23 23:47:28

File A: C:\Temp\musique_non_stop 320 CBR LAME 3.97.mp3
File B: C:\Temp\musique_non_stop 320 CBR LAME 3.98.mp3

23:47:28 : Test started.
23:47:39 : 01/01  50.0%
23:47:46 : 02/02  25.0%
23:47:51 : 03/03  12.5%
23:47:56 : 04/04  6.3%
23:48:00 : 05/05  3.1%
23:48:03 : 06/06  1.6%
23:48:06 : 07/07  0.8%
23:48:11 : 08/08  0.4%
23:48:17 : 09/09  0.2%
23:48:21 : 10/10  0.1%
23:48:22 : Test finished.

 ----------
Total: 10/10 (0.1%)

Pre-echo sounds really bad at 0:02 on the 3.98.2 encode.

lvqcl, can you encode and upload this sample (http://www.hydrogenaudio.org/forums/index.php?act=attach&type=post&id=5656) with your modified version of LAME 3.98.2?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-01-24 00:34:43
Sure.
[attachment=5669:musique_non_stop.mp3]
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-24 01:01:37
Thanks, lvqcl.

I've tried out the --strictly-enforce-iso switch with LAME 3.97:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/24 00:17:20

File A: C:\Temp\musique_non_stop 320 CBR LAME 3.97.mp3
File B: C:\Temp\musique_non_stop 320 CBR LAME 3.97 ISO.mp3

00:17:20 : Test started.
00:17:56 : 01/01  50.0%
00:18:02 : 02/02  25.0%
00:18:09 : 03/03  12.5%
00:18:33 : 04/04  6.3%
00:18:40 : 05/05  3.1%
00:18:49 : 06/06  1.6%
00:19:04 : 07/07  0.8%
00:19:15 : 08/08  0.4%
00:19:25 : 09/09  0.2%
00:19:38 : 10/10  0.1%
00:20:05 : 11/11  0.0%
00:20:17 : 12/12  0.0%
00:20:32 : 13/13  0.0%
00:20:58 : 14/14  0.0%
00:21:13 : 15/15  0.0%
00:21:23 : 16/16  0.0%
00:21:25 : Test finished.

 ----------
Total: 16/16 (0.0%)

The strict ISO encoded version has really horrid artifacts starting at 0:02, just like with LAME 3.98.2. While the default 3.97 encode sounds better with less annoying artifacts appearing later.

LAME 3.98.2 vs LAME 3.98.2 without the ISO frame size limit:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/24 00:40:53

File A: C:\Downloads\musique_non_stop.mp3
File B: C:\Temp\musique_non_stop 320 CBR LAME 3.98.mp3

00:40:53 : Test started.
00:41:11 : 01/01  50.0%
00:41:16 : 02/02  25.0%
00:41:26 : 03/03  12.5%
00:41:35 : 04/04  6.3%
00:41:41 : 05/05  3.1%
00:41:52 : 06/06  1.6%
00:41:58 : 07/07  0.8%
00:42:04 : 08/08  0.4%
00:42:10 : 09/09  0.2%
00:42:19 : 10/10  0.1%
00:42:27 : 11/11  0.0%
00:42:38 : 12/12  0.0%
00:42:45 : 13/13  0.0%
00:42:53 : 14/14  0.0%
00:43:00 : 15/15  0.0%
00:43:06 : 16/16  0.0%
00:43:14 : 17/17  0.0%
00:43:21 : 18/18  0.0%
00:43:29 : 19/19  0.0%
00:43:43 : 20/20  0.0%
00:43:50 : Test finished.

 ----------
Total: 20/20 (0.0%)

My LAME 3.98 encoded version has horrid artifact starting from 0:02, just like the LAME 3.97 strict ISO encode does. While lvqcl's encode sounds better.

LAME 3.97 vs LAME 3.98.2 without the ISO frame size limit:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/24 00:50:11

File A: C:\Downloads\musique_non_stop.mp3
File B: C:\Temp\musique_non_stop 320 CBR LAME 3.97.mp3

00:50:11 : Test started.
00:51:07 : 01/01  50.0%
00:51:16 : 01/02  75.0%
00:51:36 : 01/03  87.5%
00:52:02 : 01/04  93.8%
00:52:28 : 01/05  96.9%
00:52:48 : 01/06  98.4%
00:52:57 : 02/07  93.8%
00:53:03 : 02/08  96.5%
00:53:09 : 02/09  98.0%
00:53:15 : 03/10  94.5%
00:53:32 : 03/11  96.7%
00:53:37 : 04/12  92.7%
00:53:46 : 05/13  86.7%
00:53:56 : 06/14  78.8%
00:54:05 : 06/15  84.9%
00:54:11 : 07/16  77.3%
00:54:17 : 07/17  83.4%
00:54:22 : 07/18  88.1%
00:54:27 : 08/19  82.0%
00:54:29 : Test finished.

 ----------
Total: 8/19 (82.0%)

I can't really tell the difference, they both sound equally bad.

I didn't use ReplayGain on lvqcl's first sample "the_robots.mp3", because it was not scanned and i decided to do the test without applying track gain. Am just wondering if that would have effected the test?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-01-24 16:24:51
I have another test with the --strictly-enforce-iso switch, and it does degrade the sound quality.

LAME 3.97 320kbps CBR --strictly-enforce-iso Vs LAME 3.97 320kbps CBR

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/24 15:59:42

File A: C:\Temp\The Robots 320 CBR (LAME 3.97).mp3
File B: C:\Temp\The Robots 320 CBR LAME 3.97 ISO.mp3

15:59:42 : Test started.
16:00:15 : 01/01  50.0%
16:00:21 : 02/02  25.0%
16:00:35 : 03/03  12.5%
16:00:42 : 04/04  6.3%
16:00:55 : 05/05  3.1%
16:01:00 : 06/06  1.6%
16:01:05 : 07/07  0.8%
16:01:19 : 08/08  0.4%
16:01:30 : 09/09  0.2%
16:01:41 : 10/10  0.1%
16:01:52 : 11/11  0.0%
16:02:05 : 12/12  0.0%
16:02:06 : Test finished.

 ----------
Total: 12/12 (0.0%)
The pre-echo is alot worse on the ISO encode.

I have also done the test with the old kraftwerk sample from ff123.net: (http://ff123.net/samples/kraftwerk.flac)

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/24 16:11:37

File A: C:\Temp\kraftwerk 320 CBR LAME 3.97 ISO.mp3
File B: C:\Temp\kraftwerk 320 CBR LAME 3.97.mp3

16:11:37 : Test started.
16:12:02 : 01/01  50.0%
16:12:08 : 02/02  25.0%
16:12:15 : 03/03  12.5%
16:12:22 : 04/04  6.3%
16:12:30 : 05/05  3.1%
16:12:36 : 06/06  1.6%
16:12:46 : 07/07  0.8%
16:12:56 : 08/08  0.4%
16:13:00 : 09/09  0.2%
16:13:08 : 10/10  0.1%
16:13:12 : 11/11  0.0%
16:13:20 : 12/12  0.0%
16:13:30 : 13/13  0.0%
16:13:37 : 14/14  0.0%
16:13:38 : Test finished.

 ----------
Total: 14/14 (0.0%)
An obvious artifact appears at 0:10 - 0:12 on the strict ISO encoded version.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: timcupery on 2010-01-24 16:45:53
I'd love to hear from a Lame dev on the actual difference(s) between 3.97 and 3.98 with regards to Mp3packer-compressible boat. (I assume no one who's responded to this thread is an actual Lame dev, or if they are wasn't privy to the decision in-question, because otherwise it wouldn't be necessary to reverse-engineer Lame to figure out what has changed.)

Secondly, it appears that Mp3packer optimizing may create files that don't work with some versions of WMP?

Third, I tried playing some large (~260 kbps) mp3 files in WMP to see if it would fail. I've tried both Lame 3.96 -V0 files and Mp3packer-compressed Lame 3.98 files. Both worked fine on my computers. I have WMP 11 (on XP SP3) on my desktop (although I've installed a bunch of audio software that may have changed the default mp3 codecs). I have WMP 12 (on Windows 7) on my laptop, and the only audio program I've installed there is foobar2000.

So I'm wondering - on what versions of WMP does this error happen?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-01-24 16:59:01
Quote
I assume no one who's responded to this thread is an actual Lame dev

See post #14

Quote
because otherwise it wouldn't be necessary to reverse-engineer Lame to figure out what has changed

This change was mentioned in LAME changelog:
Code: [Select]
LAME 3.98 beta 1   May 16 2007
...
    * Restricted bitreservoir size for 320 kbps frames to the size used for sideinfo, because of decoding problems with FhG decoders installed on almost every Windows system


Quote
So I'm wondering - on what versions of WMP does this error happen?

The problem sample is "BlackBird_lame3.97.mp3" from this post (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=40308&view=findpost&p=354533).
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-02-21 14:24:38
According to this post (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=40308&view=findpost&p=687889),  LAME now has different workaround for buggy FhG decoder that doesn't cause bitrate bloat and doesn't limit frame size.

Both 3.98 and 3.99alpha versions were changed.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: timcupery on 2010-02-21 16:32:38
@lvqcl - thanks for pointing out the Lame-dev thing that I'd missed earlier in the thread.

The problem doesn't come with modern versions of WMP (I tested 11 and 12), but rather with mplayer2 (the "hidden" WMP v.6.4 in XP) which uses the l3codecx.ax DirectShow decoder.

However, the problem appears a lot more limited than the solution. While mplayer2 on my XP machine does mess up with the original blackbird 3.97 sample (http://www.hydrogenaudio.org/forums/index.php?showtopic=40308&st=0&p=354533&#entry354533), it has no problem with other files that theoretically should cause problems. I have not been able to produce any problem using mplayer2 to play Lame 3.96 or 3.97 high-bitrate V0 files (that contain lots of 320kbps frames), or with 3.98 high-bitrate V0 files which I have substantially compressed using MP3packer (so basically removing the bloat that results from --strictly-enforce-iso switch, as V0 files from 3.96 and 3.97 can almost never achieve any compression with MP3packer.
So I've wondered if the higher-bitrate bloat (and/or inefficiency with problem samples that are already maxed out at 320kbps for some frames) is extended further than is necessary for player compatibility.

So, I'm glad to know about the new development with Lame 3.99 alpha 2. Thanks to robert and the lame devs for continuing to work on this!
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-02-21 18:02:55
So, I'm glad to know about the new development with Lame 3.99 alpha 2.

In fact, LAME 3.98.x branch was changed too.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: WonderSlug on 2010-02-21 18:39:27
@lvqcl - thanks for pointing out the Lame-dev thing that I'd missed earlier in the thread.

The problem doesn't come with modern versions of WMP (I tested 11 and 12), but rather with mplayer2 (the "hidden" WMP v.6.4 in XP) which uses the l3codecx.ax DirectShow decoder.

However, the problem appears a lot more limited than the solution. While mplayer2 on my XP machine does mess up with the original blackbird 3.97 sample (http://www.hydrogenaudio.org/forums/index.php?showtopic=40308&st=0&p=354533&#entry354533), it has no problem with other files that theoretically should cause problems. I have not been able to produce any problem using mplayer2 to play Lame 3.96 or 3.97 high-bitrate V0 files (that contain lots of 320kbps frames), or with 3.98 high-bitrate V0 files which I have substantially compressed using MP3packer (so basically removing the bloat that results from --strictly-enforce-iso switch, as V0 files from 3.96 and 3.97 can almost never achieve any compression with MP3packer.
So I've wondered if the higher-bitrate bloat (and/or inefficiency with problem samples that are already maxed out at 320kbps for some frames) is extended further than is necessary for player compatibility.

So, I'm glad to know about the new development with Lame 3.99 alpha 2. Thanks to robert and the lame devs for continuing to work on this!


Interesting.

I have a Windows 2000 machine, and bringing up mplayer2 on it (verifying it was version 6.4.09.1129), it plays that blackbird397 sample perfectly fine.

However, I did install the latest K-Lite Codec Pack on that machine, so that likely replaced the l3codecx DirectShow filter with either a newer version, or is using a 3rd party one that doesn't have the problem.

Title: Lame 3.98 wastes bits, esp. at V0
Post by: robert on 2010-02-21 19:23:43
If you want to reproduce the problem, simply use Graphedit (http://www.digital-digest.com/dvd/downloads/showsoftware_graphedit_141.html) to build a DirectShow filter graph, using Fhg's l3codecx.ax.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-02-21 19:37:25
I have 2 FhG decoders in my system:
1) c:\Windows\System32\l3codecx.ax (ver. 1.5.0.50)
2) c:\Program Files\K-Lite Codec Pack\Filters\l3codecx.ax (ver. 1.9.0.311)

The former doesn't play blackbird397, the latter does.

WonderSlug, if your system uses l3codecx from k-lite folder, run "regsvr32.exe c:\Windows\System32\l3codecx.ax" and try again.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: timcupery on 2010-02-21 20:09:58
So, I'm glad to know about the new development with Lame 3.99 alpha 2.

In fact, LAME 3.98.x branch was changed too.

Excellent. I guess if it's a relatively simple tweak, might there be a version 3.98.3 official release?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: robert on 2010-02-21 20:19:08
That's the plan.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-02-21 21:44:18
Wonderful.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: pflinden on 2010-02-23 17:56:10
I don't think theres much hope on those situations. At least those situations are rare, unless the user is a huge fan of: electronic music, castanets, drums / cymbals (at low bitrates) and Kraftwerk.

The timing resolution on the Mp3 specs limit completely shrinks the "room for improvement" for Mp3. IMO it's the best you can ever get out of Mp3 with critical pre-echo samples, while AAC still has some room for improvement.


I have plenty of electronic music in my collection like Kraftwerk, Aphex Twin, Autechre etc. What would be the best way to encode these?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: dv1989 on 2010-02-23 19:48:42
Assuming you want to use LAME, you seem to have two options A: wait for the next version, which will address the issue; or B: encode some examples with 3.98, and test whether or not you hear anything undesirable.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: /mnt on 2010-02-23 20:50:45
I have plenty of electronic music in my collection like Kraftwerk, Aphex Twin, Autechre etc. What would be the best way to encode these?


Sadly a lot of electronic music contains a lot of sharp transients, which can produce artifacts know as pre-echo. Which is a major problem with Mp3.

I would recommend doing an ABX (http://www.hydrogenaudio.org/forums/index.php?showtopic=16295) test with some of Kraftwerk's tracks such as: The Robots, Spacelab, The Man Machine, (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77128&view=findpost&p=674479) Computer World 2, Tour De France (1983 version), Boing Boom Tschak and the most hard to encode Kraftwerk track IMO would be Musique Non Stop. If you are sensitive to pre-echo you will likely to be able to ABX most of those tracks effortlessly with any Mp3 encoder at around 192kbps - 320kbps.

If you managed pass a ABX test with most of those tracks, i then would recommend trying out AAC if you have an iPod or a player that supports AAC. From my personal ABX tests, i find most of those tracks sound acceptable or better then Mp3 with Nero's AAC encoder at -q 0.50 (175kbps) while sounding horrid with LAME, even at 320kbps.

Also Musepack is worth trying out with electronic music.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-02-27 18:14:52
http://lame.cvs.sourceforge.net/viewvc/*ch...athrev=lame3_98 (http://lame.cvs.sourceforge.net/viewvc/*checkout*/lame/lame/doc/html/history.html?pathrev=lame3_98)

LAME 3.98.3 was released.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: pflinden on 2010-02-27 21:36:37
I have plenty of electronic music in my collection like Kraftwerk, Aphex Twin, Autechre etc. What would be the best way to encode these?


Sadly a lot of electronic music contains a lot of sharp transients, which can produce artifacts know as pre-echo. Which is a major problem with Mp3.

I would recommend doing an ABX (http://www.hydrogenaudio.org/forums/index.php?showtopic=16295) test with some of Kraftwerk's tracks such as: The Robots, Spacelab, The Man Machine, (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77128&view=findpost&p=674479) Computer World 2, Tour De France (1983 version), Boing Boom Tschak and the most hard to encode Kraftwerk track IMO would be Musique Non Stop. If you are sensitive to pre-echo you will likely to be able to ABX most of those tracks effortlessly with any Mp3 encoder at around 192kbps - 320kbps.

If you managed pass a ABX test with most of those tracks, i then would recommend trying out AAC if you have an iPod or a player that supports AAC. From my personal ABX tests, i find most of those tracks sound acceptable or better then Mp3 with Nero's AAC encoder at -q 0.50 (175kbps) while sounding horrid with LAME, even at 320kbps.

Also Musepack is worth trying out with electronic music.


I have iPod/iTunes so I could encode electronic stuff as AAC. Is this a problem only with purely electronic music, or do for example hip-hop beats, some of the more electronic Radiohead songs or the quick silent/loud dynamics of the Elbow track "Starlings" cause the same kind of artifacts? And is hand clapping of an audience still a problem for MP3?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: dv1989 on 2010-02-27 21:44:52
Surely you won't know until you test for yourself? No one else has your ears.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: greynol on 2010-02-27 21:46:06
This discussion about encoding at AAC is not on-topic.  Please start a new topic if you wish to discuss this further.  Continuation of this conversation in this thread will result in binned posts and warnings issued.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: john33 on 2010-02-27 23:54:49
http://lame.cvs.sourceforge.net/viewvc/*ch...athrev=lame3_98 (http://lame.cvs.sourceforge.net/viewvc/*checkout*/lame/lame/doc/html/history.html?pathrev=lame3_98)

LAME 3.98.3 was released.

Compiles are at Rarewares now.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-02-28 16:28:41
Thank you, robert and john33 (and whoever else contributed), for the new version.

sfb21 bitrate bloat has decreased (for the samples I tested) - very welcome. No need any more for my usual lowpass at 16.7 kHz.
What changed for sfb21 behavior?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: robert on 2010-02-28 16:35:55
What changed for sfb21 behavior?
Actually nothing. The reason for the bloating was solely the limitation of audio-data-size / bitreservoir-usage, as an attempt to circumvent the FhG decoder problem.

@timcupery: let's discuss your topic in the other thread, started by NIN9.

Moderation: Posts moved to said discussion:
http://www.hydrogenaudio.org/forums/index....showtopic=79059 (http://www.hydrogenaudio.org/forums/index.php?showtopic=79059)
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-02-28 17:50:24
What changed for sfb21 behavior?

Actually nothing. ...

You're right of course. I had a wrong bitrate in mind for 3.98.2's plain -V0 - I just rechecked.
Title: Lame 3.98 wastes bits, esp. at V0
Post by: halb27 on 2010-02-28 19:03:49
As I'm going to reencode that part of my collection for which I use mp3:

I don't care about 30 kbps or so, but like the best quality for 250 kbps or so on average even in worst case scenerios.
-V0 is the best VBR quality level, and to have an even more demanding setting I think using a negative value for the --ns-xxxxx options can do the job.
After some trials I ended up with -V0 --ns-alto -6 --ns-treble -3 -Y which yields 256 kbps for my standard test set.
--ns-treble has a more serious impact on bitrate than --ns-alto. That's why there are different values for the two options.  --ns-treble unfortunately covers sfb21, that's why I also use -Y.

Any experience with settings like this making -V0 more defensive?
Title: Lame 3.98 wastes bits, esp. at V0
Post by: lvqcl on 2010-08-14 17:57:04
http://support.microsoft.com/kb/2115168/en-us (http://support.microsoft.com/kb/2115168/en-us)
MS10-052 Vulnerability in Microsoft MPEG Layer-3 codecs could allow remote code execution

L3codecx.ax was updated to ver. 1.6.0.52 (date=15-Jun-2010). This version is able to play BlackBird_lame3.97.mp3