Make aoTuV the new recommended build?

Topic: Make aoTuV the new recommended build? (Read 43370 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Make aoTuV the new recommended build?

Reply #75 – 2004-06-03 14:40:58

Quote

Phong, what kind of music do you listen (maybe quiet or classical)? Results for GT3 are a little bit strange for me. I almost always get bigger difference in average bitrate between AoTuV and GT3.

As you know Garf even changed average bitrate nominals for q>=5. But looking on stats you have provided I can't see that difference.

I listen to a pretty wide mix (probably the majority being rock and electronic), classical might even be underrepresented (I think I only have a couple tracks from Star Wars scores for that). For doing these measurements I tried to choose a fairly wide mix that had a similar bitrate distribution as my whole collection. I'll post a song list when I post the complete stats later today.

My methodology for picking songs probably could be questioned... I started out just throwing together a bunch of songs I liked, but the average was a lot higher than my collection as a whole. So, I did some analysis of my whole collection (which is encoded with gt3b2 at q6) to see how bitrates are distributed (i.e. how many songs are in the 150-160 range, how many in the 160-170, etc.) I excluded a few things (non-music albums, e.g. Bill Cosby, or the collection of sound effects at the end of the Star Trek: Generations soundtrack). I then added/removed a few songs to make the sample population more similar.

Of course, since I didn't choose a totally random sample, the selection is gonna be biased. One obvious bias is that there will sometimes be two or three songs from one album - this is largely because ripping from more albums was more of a nusiance (I don't have any space to keep a lossless collection anywhere). I did try choosing a totally random sample, but that was even more problematic - like 1/5 of the songs would have been They Might be Giants. Since the bias is consistent though (I'm looking for relative bitrates, not true averages over all music), I didn't think it would be a problem for the purproses of choosing similar -q settings. It's entirely possible though that I've somehow underrepresented tracks where gt3b2 bloats a lot compared to aotuv. I'm open to criticism and suggestions on how to improve my sample set (obviously bound by the restriction that I don't have lossless copies of every song on Earth.)

Quote

Now, I rather think most serious problem of Vorbis at high bitrate is some kind of noise boost problem appears on Brahms3

Do you feel this might be a general problem with vorbis (like the HF boost problem), or something peculiar to this particular sample/sound?

Make aoTuV the new recommended build?

Reply #76 – 2004-06-03 14:53:46

Quote

Do you feel this might be a general problem with vorbis (like the HF boost problem), or something peculiar to this particular sample/sound?

This problem could matter for those who tend to listen to old recordings regardless of musical styles. And I like to listen to the historical recordings. (e.g. performed by Horowitz, Furtwängler) I'm not sure if this problem can be found from decent recordings of nowadays.

Make aoTuV the new recommended build?

Reply #77 – 2004-06-03 18:54:56

Okey dokey. I've compiled all the data now for the whole -q range from -1 to 10. My computer isn't the fastest thing on earth, and it's usually doing three things at once (it runs a go playing robot on KGS as well as being my regular desktop system), so it took a little time. Fortunately I have a system of scripts set up to make compiling all this data easy, and I can just leave it crunching in the background. Adding new encoders or samples or quality settings would be pretty easy.

You can download all the raw data and neat stuff in spreadsheet form in your choice of formats:
vorbis_bitrates.sxc (OpenOffice)
vorbis_bitrates.xls (MS Excel)

Here are the bitrates relative to stock vorbis (excluding quality levels > 9 because they go all wonky):

These are -q settings that should average to about the same bitrate. I've excluded aqk2 from these compuations for 2 <= q < 5 because its bitrate is a distant outlier in that range. Likewise for qk32 and gt3b2 for q > 9. If you want to test one of those forks in one of those bitrate ranges, the spreadsheet files include both these numbers and a version that includes the outliers.

Code: [Select]

              Average bitrates                   Equivalent -q levels (excluding outliers)
 -q   xiph.org   aotuv    aqk2   gt3b2    qk32   xiph.org   aotuv    aqk2   gt3b2    qk32
-1.00   46.92   47.67   47.68   46.92   43.23      -0.94   -1.00   -1.00   -0.94   -0.62
-0.50   52.89   52.86   52.86   52.89   49.06      -0.56   -0.56   -0.56   -0.56   -0.28
-0.01   59.79   58.70   58.70   59.79   55.83
0.00    62.16   61.81   61.81   62.17   58.19       0.00    0.02    0.02    0.00    0.27
0.50    70.19   69.22   69.22   70.19   65.61       0.42    0.48    0.48    0.42    0.73
0.99    77.82   76.86   76.86   77.82   72.66
1.00    77.31   76.25   76.25   77.31   72.95       1.00    1.11    1.05    1.00    1.31
1.50    80.55   81.30   85.99   80.55   80.07       1.68    1.54    1.23    1.68    1.66
1.99    83.69   86.43   93.71   83.69   84.97
2.00    87.88   91.39   98.76   87.88   88.95       2.30    2.00    --      2.30    2.20
2.50    93.78   96.25  104.20   93.78   95.04       2.58    2.33    --      2.58    2.47
2.99    99.45  100.64  108.80   99.45  100.80
3.00   107.31  108.50  117.64  107.31  110.22       3.22    3.11    --      3.22    3.00
3.50   113.94  116.18  126.16  113.94  117.39       3.61    3.44    --      3.61    3.35
3.99   120.18  123.14  133.78  120.18  124.05
4.00   120.19  121.45  132.05  120.19  124.10       4.18    4.12    --      4.11    4.00
4.50   131.32  132.79  143.91  138.47  136.46       4.65    4.58    --      4.37    4.43
4.99   142.58  144.24  154.84  152.78  148.25
5.00   151.13  150.45  160.89  161.02  157.55       5.49    5.45    5.01    5.00    5.26
5.50   161.24  162.27  168.44  170.73  164.14       5.69    5.62    5.26    5.25    5.64
5.99   172.02  174.64  174.77  181.04  168.43
6.00   180.15  182.24  182.25  188.26  179.56       6.36    6.22    6.22    6.00    6.38
6.50   191.42  195.83  195.83  200.20  191.10       6.64    6.47    6.47    6.29    6.65
6.99   203.14  209.59  209.59  212.52  203.13
7.00   205.65  214.89  214.89  218.51  205.65       7.41    7.10    7.10    7.00    7.41
7.50   221.25  232.84  232.84  232.76  221.25       7.71    7.39    7.39    7.35    7.71
7.99   237.65  252.78  252.78  247.88  237.64
8.00   236.77  251.83  251.83  246.63  236.77       8.24    8.00    8.00    8.08    8.24
8.50   267.84  281.32  281.32  277.34  267.84       8.61    8.40    8.40    8.47    8.61
8.99   301.55  312.20  312.20  310.61  301.55
9.00   307.99  321.28  321.28  319.26  308.02       9.08    9.00    9.00    9.01    9.04
9.50   396.08  407.85  407.86  492.09  471.17       9.59    9.46    9.46    --      --
9.99   440.95  457.45  457.45  546.63  510.48
10.00  441.50  458.15  458.15  547.46  511.03

And, as promised, the list of songs. I promise I have a totally legitimate excuse for using those Best of Led Zeppelin CDs.

Quote

Aphex Twin - Come to Daddy / 01 - Come to Daddy (Pappy Mix)
Aphex Twin - Come to Daddy / 04 - Bucephalus Bouncing Ball
Aphex Twin - Come to Daddy / 05 - To Cure a Weakling Child (Contour Regard)
Beck - Odelay / 08 - Where It's At
Ben Harper - Welcome to the Cruel World / 02 - Whipping Boy
Ben Harper and the Innocent Criminals - Burn to Shine / 11 - Beloved One
Bjork - Homogenic / 02 - Joga
Bjork - Post / 02 - Hyper-ballad
Bjork Gudmundsdottir & trio Gudmundar Ingolfssonar - Gling-Glo / 01 - Gling Glo
Clutch - The Elephant Riders / 04 - The Soapmakers
Clutch - The Elephant Riders / 10 - The Dragonfly
Daft Punk - Discovery / 04 - Harder, Better, Faster, Stronger
David Bowie - Earthling / 01 - Little Wonder
Dick Dale - Better Shred than Dead (Disc 2) / 11 - Bandito
Geggy Tah - Sacred Cow / 08 - Such a Beautiful Night
Incubus - S.C.I.E.N.C.E / 07 - A Certain Shade of Green
Isao Tomita - Snowflakes Are Dancing / 05 - Arabesque No. 1
John Linnell - State Songs / 01 - Illinois
John Williams - Star Wars Episode I: The Phantom Menace/02 - Duel of the Fates
John Williams - Star Wars Trilogy: The Original Soundtrack Anthology (Disc 2) / 05 - The Imperial March (Darth Vader's Theme)
Kenji Yamamoto - Metroid Prime [OST] / 08 - Ancient Chozo Ruins
Lamb - Fear of Fours / 05 - All in Your Hands
Lamb - Fear of Fours / 07 - Bonfire
Led Zeppelin - Early Days: The Best of Led Zeppelin Volume 1 / 02 - Babe I'm Gonna Leave You
Led Zeppelin - Early Days: The Best of Led Zeppelin Volume 1 / 12 - When the Levee Breaks
Norah Jones - Come away with Me / 01 - Don't Know Why
Portishead - Portishead / 09 - Only You
Skunk Anansie - Paranoid & Sunburnt / 03 - I Can Dream
Stereolab - Emperor Tomato Ketchup / 04 - Yper-Sound
The Cardigans - First Band on the Moon / 07 - Lovefool
They Might Be Giants - Apollo 18 / 08 - The Guitar
They Might Be Giants - Apollo 18 / 13 - See the Constellation
They Might Be Giants - Factory Showroom / 06 - New York City
They Might Be Giants - Live!! New York City 10-14-94 (Promo) / 07 - Why Does the Sun Shine
They Might Be Giants - No! / 06 - In the Middle, in the Middle, in the Middle
Tom Petty - Wildflowers / 08 - Don't Fade on Me
Tom Petty - Wildflowers / 12 - A Higher Place
Various Artists - The Big Lebowski [Soundtrack] / 03 - Elvis Costello - My Mood Swings
Various Artists - The Big Lebowski [Soundtrack] / 08 - Kenny Rogers & The First Edition - Just Dropped In (To See What Condition My Condition Was In)
Various Artists - The Big Lebowski [Soundtrack] / 11 - Henri Mancini - Lujon

Make aoTuV the new recommended build?

Reply #78 – 2004-06-03 23:42:05

It seems my computer is that much slower than yours Phong, here is a bitrate comparison of 12 quality levels over the three encoders across 80 randomly picked songs. Less work and yet it still took me longer

Make aoTuV the new recommended build?

Reply #79 – 2004-06-04 01:47:29

Quote

I've compared three encoders on my favourite samples with following settings.

aoTuV b2 -q6
LAME 3.90.3 --alt-preset standard
Musepack 1.14 --standard --xlevel (I'm personally interested in this setting since I've been using this parameter for my archiving.)

Bartok_strings2
Castanets
Chanchan
Hosokawa___Atem_lied
Rosemary
Trust
Waiting

Thanks for the results, Harashin. They do correlate in some ways to the 128 kbps multiformat listening test I think.

Well, I'm very close to making aoTuV the recommended encoder for all bitrates.

Make aoTuV the new recommended build?

Reply #80 – 2004-06-04 06:14:44

Quote

Well, I'm very close to making aoTuV the recommended encoder for all bitrates.

My results aren't good enough to conclude anything for everyone, even though they're good for me.

Anyway, I did some more test.
Caramel_Vorbis_MPC_test

Make aoTuV the new recommended build?

Reply #81 – 2004-06-04 21:37:03

Quote

It seems my computer is that much slower than yours Phong, here is a bitrate comparison of 12 quality levels over the three encoders across 80 randomly picked songs. Less work and yet it still took me longer

Are you sure? Mine took 5 days...

Anyhow, based on your numbers and de Mon's comments, I'm thinking my numbers don't reflect the typical bitrate difference between aotuv and gt3b2, but they're probably good enough, since it looks like aotuv might be the winner a the same -q level (and lower bitrate). Looking through my whole collection (again, gt3b2 at -q 6), I see several outliers with bitrates more than 70kbps over the average (a few even over 300kbps, the highest being Ions (-) off of Tool - AEnima at a whopping 377kbps), and none at all more than 60kbps below the average. Most of these bitrate hogs are electronic with lots of impulse type sounds. I think these sorts of sounds are probably underrepresented in my sample set.

I agree with harashin that we can't call a winner yet, but indeed, it's not looking good for gt3b2 (espescially considering that aotuv is getting by with fewer bits). I'm hoping I'll have time to do some tests myself this weekend, which I'm hoping will be telling (I'm extremely sensetive to preecho, but otherwise average-eared).

Make aoTuV the new recommended build?

Reply #82 – 2004-06-05 00:19:09

Quote

Quote
It seems my computer is that much slower than yours Phong, here is a bitrate comparison of 12 quality levels over the three encoders across 80 randomly picked songs. Less work and yet it still took me longer

Are you sure? Mine took 5 days...

Oh heh I thought you started the -1 to 10 encode in the morning and had the results that evening

I'm trying to produce a neat little 'munger' program to construct a single wave to represent an entire sound collection. Suffice it to say I'm not quite at your level of coding yet though

Make aoTuV the new recommended build?

Reply #83 – 2004-06-05 17:32:51

I've tried to perform a similar test than Harashin. Some files were not tested:
- bigyellow: I can't bear the sound of this sample, because it sounds 'artefacted' even without encoding.
- newyorkcity, ordinaryworld, rosemary, sincealways: short in time

I've also tested the two last files, because Harashin's results were interesting.

The listening conditions of my tests may differs from harashin's. I've only rated and ABXed a short part of each sample. Notation is linked to the tested range. The range is described in log files, also available:
http://www.foobar2000.net/divers/tests/vorbis200/

The test confirmed the theory: aoTuV is cleaner, but sometimes imprecise compared to GT3. More details about notation (like Debussy.wav) are present on log files.

I can't resume the test (but is it needed?) now.

Make aoTuV the new recommended build?

Reply #84 – 2004-06-06 07:35:31

Quote

I've tried to perform a similar test than Harashin. Some files were not tested:
- bigyellow: I can't bear the sound of this sample, because it sounds 'artefacted' even without encoding.
- newyorkcity, ordinaryworld, rosemary, sincealways: short in time

I've also tested the two last files, because Harashin's results were interesting.

Your bitrate values are different from mine, I suppose you did test aoTuVb2 -q6 with GT3b2 -q6, didn't you?

However, it's indeed interesting to me that aoTuV tied(slightly better?) with GT3 at same -q value. (with lower average bitrate)

Make aoTuV the new recommended build?

Reply #85 – 2004-06-06 08:42:42

Quote

The test confirmed the theory: aoTuV is cleaner, but sometimes imprecise compared to GT3. More details about notation (like Debussy.wav) are present on log files.

The problem worries me very much. If I can continue development of aoTuV, I will want to correct someday.

Thanks.

Make aoTuV the new recommended build?

Reply #86 – 2004-06-06 10:21:20

Quote

The problem worries me very much. If I can continue development of aoTuV, I will want to correct someday.

Thanks.

Oh no! Do you want to say you can't continue your tunings on Vorbis now? Pity news.

Make aoTuV the new recommended build?

Reply #87 – 2004-06-06 11:08:29

Quote from: harashin,Jun 6 2004, 07:35 AM

Quote from: guruboolez,Jun 6 2004, 01:32 AM

Your bitrate values are different from mine, I suppose you did test aoTuVb2 -q6 with GT3b2 -q6, didn't you?

Yes... I don't really know. There are now many version/sub-version, and it's a bit confusing. I've used an ogdropXp for my encodings, labelled as 1.01 + GT3b2.

Which version of GT tuning was recommanded for tests?

Make aoTuV the new recommended build?

Reply #88 – 2004-06-06 11:37:15

Quote

Yes... I don't really know. There are now many version/sub-version, and it's a bit confusing. I've used an ogdropXp for my encodings, labelled as 1.01 + GT3b2.

Which version of GT tuning was recommanded for tests?

There's no problem with GT3b2 since it gave us exactly same averaged files. I meant about aoTuVb2's -q value. I used both oggdropXPd V.1.7.11 using GT3b2 and V.1.7.11 using aoTuVb2 at RW for my test. I usually use oggenc by Aoyumi, but I was too lazy to encode 18 files with it.

Make aoTuV the new recommended build?

Reply #89 – 2004-06-06 12:43:24

Oups, I've just understand the meaning of your question. Sorry.
For aoTuV, I've used an oggenc.exe of beta 2 at 6,22 (and not 6.00). I was maybe fooled by the "dot" problem. I need to verify it (but I can't yet).

Make aoTuV the new recommended build?

Reply #90 – 2004-06-06 21:17:55

Harashin. Thank you very much for your test.

I calculated average results:

[average rating] [average bitrate]
Vorbis aoTuVb2 -q6
4,14 208,43

Musepack 1.14b --xlevel
4,05 187,00 (missed rating for MPC in Rosemary sample)

LAME 3.90.3 --alt-preset standard
3,64 209,86

Make aoTuV the new recommended build?

Reply #91 – 2004-06-07 00:59:31

Quote

Musepack 1.14b --xlevel
4,05 187,00 (missed rating for MPC in Rosemary sample)

It should be counted as 5.00 because it was transparent to me. Hence, my average score for mpc is calculated as 4.19. But that quick test isn't good for any conclusion, either. (even for me)

Make aoTuV the new recommended build?

Reply #92 – 2004-06-07 15:19:51

Quote

Oh no! Do you want to say you can't continue your tunings on Vorbis now? Pity news.

There is an intention that I continue development of aoTuV. However, I cannot promise continued development simultaneously. Please guess.

Make aoTuV the new recommended build?

Reply #93 – 2004-06-07 16:03:00

I tried to do a few tests this weekend but made the mistake of trying to find my own samples. Since I couldn't find many I could ABX at these settings, I only was able to finish one in the time I had. When I have time I'll try with some of the samples others have been using.

Code: [Select]

Aphex Twin - Come to Daddy / 04 - Bucephalus Bouncing Ball
                  | bitrate | score
-----------------+---------+-------
xiph.org -q 6.36 |  207.9  |  3.3
aotuv    -q 6.22 |  228.2  |  3.3
gt3b2    -q 6.00 |  237.0  |  3.7

Comments, ABX results, etc.:
BucephalusBouncingBall_result.txt
BucephalusBouncingBall.flac

Notice