ReplayGain tagging is setting off NTFS dirty bit

Topic: ReplayGain tagging is setting off NTFS dirty bit (Read 6765 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

ReplayGain tagging is setting off NTFS dirty bit

2009-12-23 21:51:08

On my Windows 7 system, running a ReplayGain scan on a set of files with 0.9.6.9 will frequently trip the dirty bit of whichever volume that the files are on, prompting a chkdsk scan on the next restart. This happens much more often if I set foobar to automatically RG scan files after conversion, but also happens on straight RG scans. This seems to be happening when the tags are being written. Whenever this happens, I get an error message saying that the source file is corrupt the first time I try to apply the RG settings, and the tagging process aborts. This is in the form:

Code: [Select]

Could not update tags (The file is corrupted) on:
"F:\User Files\Datana\Documents\Output\Lossy\Foreigner - The Definitive Collection\Disc 2\07 - Midnight Blue.mp3"

If I run the scan a second time, this message goes away and it proceeds normally. chkdsk on an affected volume reveals no errors, and SMART data indicate that the drive has no issues, so I don't think that the physical drives are an issue (I have a four HD configuration).

To confirm that it was foobar causing this, I ran fsutil immediately before and immediately after a RG scan, and confirmed a clean state before and a dirty state after. I repeated this a few times, running chkdsk in-between in order to unset the dirty bit.

At first, I thought that this was due to mishandling of Unicode filenames, as it was happening primarily to Japanese language rips, but I've confirmed that it happens with fully ASCII filenames as well.

I've managed to reproduce this behavior on both my normal configuration as well as a completely wiped, fresh foobar configuration, and I'm not sure what could be causing it. Pasting my configuration files over to my Vista and XP machines results in neither one exhibiting this behavior.

Am I overlooking something here?

As for installed components, I have:

Code: [Select]

Core (2009-08-22 02:25:22)
	foobar2000 core 0.9.6.9
foo_abx.dll (2009-06-07 04:25:26)
	ABX Comparator 1.3.4
foo_ac3.dll (2009-05-09 16:27:36)
	AC3 decoder 0.9.3
foo_adpcm.dll (2009-05-09 16:28:04)
	ADX decoder 1.8
	BRR decoder and converter 0.7
	GCN DSP decoder 1.3
	Interplay ACM decoder 1.0
	kode's ADPCM decoders 1.2
	OKI-ADPCM decoder 0.14
	RAC decoder 1.0
	XA ADPCM decoder 1.3
foo_albumlist.dll (2009-08-22 02:23:44)
	Album List 4.3.1
foo_cdda.dll (2009-08-22 02:23:34)
	CD Audio Decoder 2.1.4
foo_channel_mixer.dll (2008-03-12 01:37:47)
	Channel Mixer 0.9.6.5
foo_converter.dll (2009-08-22 02:23:26)
	Converter 1.2.1
foo_dsp_fsurround.dll (2009-01-24 21:40:28)
	FreeSurround 0.3.5
foo_dsp_std.dll (2009-08-22 02:23:48)
	Standard DSP Array 1.0
foo_fileops.dll (2009-08-22 02:22:36)
	File Operations 2.1.2
foo_freedb2.dll (2009-08-22 02:22:52)
	freedb Tagger 0.6.1
foo_gep.dll (2009-05-09 16:29:16)
	Game Emu Player 1.64
foo_input_alac.dll (2009-03-22 14:15:46)
	ALAC Decoder 1.0.3
foo_input_dts.dll (2009-12-13 23:01:38)
	DTS decoder 0.2.7
foo_input_monkey.dll (2009-05-01 13:40:52)
	Monkey's Audio Decoder 2.1.4
foo_input_std.dll (2009-08-22 02:23:28)
	Standard Input Array 1.0
foo_input_tak.dll (2008-04-08 12:16:06)
	TAK Decoder 0.4.2
foo_input_tta.dll (2008-12-02 14:03:40)
	TTA Audio Decoder (unofficial) 2.4.2
foo_masstag.dll (2009-03-29 19:53:12)
	Masstagger 1.8
foo_midi.dll (2009-08-31 19:13:32)
	MIDI synthesizer host 1.94
foo_out_asio.dll (2009-03-22 14:15:46)
	ASIO support 1.2.7
foo_out_wasapi.dll (2009-05-19 21:45:18)
	WASAPI output support 2.1
foo_playcount.dll (2009-04-29 19:09:32)
	Playback Statistics 2.1.9
foo_psf.dll (2009-02-07 17:04:32)
	Highly Experimental 2.0.6
foo_rgscan.dll (2009-08-22 02:23:20)
	ReplayGain Scanner 2.0.9
foo_texttools.dll (2009-01-31 13:23:38)
	Text Tools 1.0.3
foo_ui_columns.dll (2007-07-22 14:36:02)
	Columns UI 0.2.0 beta 1
foo_ui_std.dll (2009-08-22 02:23:54)
	Default User Interface 0.9.5
foo_unpack.dll (2009-08-22 02:22:20)
	RAR reader 1.2
	ZIP/GZIP reader 1.0

EDIT1: List means of isolating foobar as the culprit.
EDIT2: List an example error message.

[!--sizeo:1--][span style=\"font-size:8pt;line-height:100%\"][!--/sizeo--]Moderation: Codeboxed the components list.[/size]

ReplayGain tagging is setting off NTFS dirty bit

Reply #1 – 2009-12-23 23:17:43

"The file is corrupted" is on a filesystem level, so it's not foobar2000's fault. Check if anything relevant is displayed under 'Custom Views/Administrative Events' in the Event Viewer (and it might be worthwhile looking in other areas, e.g. 'Applications and Services/Microsoft/Windows/...', they don't all seem to be part of the other filtered view here..).

Does it happen on all of your drives?

It would be worthwhile eliminating hardware issues first:

-Win 7 has a memory tester built in. If you press space whilst booting up, it will display the boot screen and you can tab across to the memory testing tool.
-Your hard drive manufacturer probably has some tool to test your hard drive, most probably with short and long tests. It would be worth checking if CrystalDiskInfo flags anything up as well.
-Some stress testing with Prime95.

That's not an exhaustive list, but a start. Otherwise perhaps some kind of OS issue, or even a motherboard issue, though probably unlikely..

ReplayGain tagging is setting off NTFS dirty bit

Reply #2 – 2009-12-24 06:49:57

I can confirm this behaviour. I've been using Win7 and 0.9.6.9 since about October this year, but it's only in the past couple of weeks I've noticed this issue happen. I'm unsure if this relates to fb2k/ReplayGain, but it's an interesting coincidence. To me it seems more probable the problem is caused by a Windows update, because I didn't see the problem for the first 2 months.

I seem to get the problem whenever I hibernate my machine (ASUS G1S laptop), then later bring it out of hibernate, and reboot it at some stage after that. I have had cases where ReplayGain will fail to write the tags for some files, but running it again works fine (I didn't think to check the logs though).

Datana, have you seen this behaviour with the foobar2000 1.0 betas? What hardware are you running?

ReplayGain tagging is setting off NTFS dirty bit

Reply #3 – 2009-12-24 11:57:53

As musicmusic said, this kind of error cannot be caused by an application which is simply using the standard OS functions to access files.
Because ReplayGain scanning is quite a resource exhaustive operation (it's done in several threads at once to fully utilize the CPU), it appears it might behave like a nice stress test for hardware and/or driver problems.

ReplayGain tagging is setting off NTFS dirty bit

Reply #4 – 2009-12-24 12:08:25

Strange.

I'm using Windows 7 Ultimate x64 with foobar2000 v1.0 beta 5.
Creative Sound Blaster Fatal1ty Champion, latest drivers.

If I'm using the command correctly it seems to work good here:

Code: [Select]

C:\Users\Andreasvb>fsutil dirty query C:
Volume - C: is NOT Dirty

C:\Users\Andreasvb>fsutil dirty query D:
Volume - D: is NOT Dirty

C:\Users\Andreasvb>fsutil dirty query I:
Volume - I: is NOT Dirty

ReplayGain tagging is setting off NTFS dirty bit

Reply #5 – 2009-12-24 20:59:48

Quote from: fraganator on 2009-12-24 06:49:57

I seem to get the problem whenever I hibernate my machine (ASUS G1S laptop), then later bring it out of hibernate, and reboot it at some stage after that. I have had cases where ReplayGain will fail to write the tags for some files, but running it again works fine (I didn't think to check the logs though).

Generally Windows 7 seems to have some problems with certain hard disks and the Sleep state or the Hibernate state.

Both a detailed description and a hotfix are available.

ReplayGain tagging is setting off NTFS dirty bit

Reply #6 – 2009-12-25 02:18:29

Thanks for the info Robertina. My laptop will come out of hibernate mode no worries. It has a 200GB SATA harddrive, so it's not approaching the 1TB drive mentioned in the KB article.

I came across the issue again this morning. I had left my foo_bpm component running overnight to scan ~1000 files. When the component finishes scanning, it will automatically update the file tags. During this update operation I got the file corrupted message:

Code: [Select]

Could not update tags (Sharing violation) on:
"D:\Music\CDs\Battles\B EP\01_Battles - SZ2.mp3"
Could not update tags (Sharing violation) on:
"D:\Music\CDs\Battles\EP C\07_Battles - Fantasy II.mp3"
Could not update tags (The file is corrupted) on:
"D:\Music\CDs\Harmonic 313\When Machines Exceed Human Intelligence\10_Harmonic 313 - Call To Arms.mp3"
Could not update tags (Sharing violation) on:
"D:\Music\CDs\Hot Chip\Made In The Dark\01_Hot Chip - Out At The Pictures.mp3"
[b]Could not update tags (The file is corrupted) on:
"D:\Music\CDs\Prefuse 73\Everything She Touched Turned Ampexian\18_Prefuse 73 - Fingertip Trajectories.mp3"[/b]
Could not update tags (The file is corrupted) on:
"D:\Music\CDs\Two Fingers\Instrumentals\12_Two Fingers - Twelvses (Two Fingers Remix).mp3"
Could not update tags (Sharing violation) on:
"D:\Music\CDs\Venetian Snares\Filth\01_Venetian Snares - Deep Dicking.mp3"
Could not update tags (Sharing violation) on:
"D:\Music\CDs\Venetian Snares\Higgins Ultra Low Track Glue Funk Hits 1972-2006\05_Venetian Snares - Vokeheads.mp3"

The 'corrupted file' is the only tag which doesn't contain a BPM value after the tagging operation.

Running fsutil shows that the D: partition now has the dirty bit set:

Code: [Select]

C:\Windows\system32>fsutil dirty query C:
Volume - C: is NOT Dirty

C:\Windows\system32>fsutil dirty query D:
Volume - D: is Dirty

So it's not just ReplayGain which can cause the problem, but a tag update in general. My component uses the update_info_async_simple function if that matters, though as musicmusic pointed out the error is being generated at the filesystem level. So this issue is either OS/driver or hardware related. As an aside, I had hibernated my PC a couple of times during the day before letting foo_bpm scan overnight, so that may have been the root cause of the problem.

Time to do some in depth stress tests...

ReplayGain tagging is setting off NTFS dirty bit

Reply #7 – 2009-12-25 02:27:04

Quote from: fraganator on 2009-12-25 02:18:29

Thanks for the info Robertina. My laptop will come out of hibernate mode no worries. It has a 200GB SATA harddrive, so it's not approaching the 1TB drive mentioned in the KB article.

Thank you for your feedback, fraganator.

I am no expert in these things, but do you think the situation could be caused by something like this?

ReplayGain tagging is setting off NTFS dirty bit

Reply #8 – 2009-12-25 02:36:06

I'd say the issue you linked to is simply a bug in Windows Explorer's tag writing scheme, which foobar2000 doesn't use (AFAIK). Thanks for looking into things though

ReplayGain tagging is setting off NTFS dirty bit

Reply #9 – 2009-12-27 08:34:53

Quote from: fraganator on 2009-12-24 06:49:57

I can confirm this behaviour. I've been using Win7 and 0.9.6.9 since about October this year, but it's only in the past couple of weeks I've noticed this issue happen. I'm unsure if this relates to fb2k/ReplayGain, but it's an interesting coincidence. To me it seems more probable the problem is caused by a Windows update, because I didn't see the problem for the first 2 months.

I seem to get the problem whenever I hibernate my machine (ASUS G1S laptop), then later bring it out of hibernate, and reboot it at some stage after that. I have had cases where ReplayGain will fail to write the tags for some files, but running it again works fine (I didn't think to check the logs though).

Datana, have you seen this behaviour with the foobar2000 1.0 betas? What hardware are you running?

For reference, my system is a stock clocked Q6600 with 8GB RAM and four Seagate 500 GB ST3500630AS drives.

After building it, I had stress tested it for 36 hours under Prime95 and run Memtest86+ for 8 hours. It's been over a year since I put it together (it was originally running Vista; the drive with the OS received a complete reformat before 7 was installed), so I went ahead and re-Memtested it for 3 hours and ran Prime95 for 8. Both yielded no errors. Using SeaTools for DOS, I performed the extended drive check and all parameters came back green. I can't call it completely certain I've ruled out all of the hardware as the source of the problem, but I can say that it's stable under harsher conditions than foobar would put it under and the hard drives are definitely not the issue.

I have tried Beta 5, and it exhibits the same behavior 0.9.6.9. Based on what you said, I went and tried to intentionally invoke the dirty bit flip before and after hibernation. It can still be done before hibernation, but only on the drive with the pagefile and only sometimes. After hibernation and reawakening the system, this starts happening with every drive and is, as far as I can tell, always invoked. It's definitely something with the writing of the tags that causes this; I can run an RG scan, have the results displayed and fsutil shows no errors, but it gets flipped once I write changes (and the "Could not update tags (The file is corrupted)" error appears, of course).

I suppose this might be some strange interaction between the Windows 7 NTFS.sys and the non-system drives (which were initially formatted under Vista). That doesn't explain why only the drive with the pagefile shows this before hibernation, or why the system drive shows this after. I might try moving stuff off of the pagefile drive, reformatting under 7, and moving it back just to see if I can exclude the Vista-formatted filesystem as a factor.

ReplayGain tagging is setting off NTFS dirty bit

Reply #10 – 2009-12-27 10:34:57

foobar2000 seems to group three errors (which I've annotated) under the "The file is corrupted" message:

Code: [Select]

[...]
    case ERROR_CRC: //Data error (cyclic redundancy check).
    case ERROR_FILE_CORRUPT: //The file or directory is corrupted and unreadable.
    case ERROR_DISK_CORRUPT: //The disk structure is corrupted and unreadable.
        throw exception_io_file_corrupted();
[...]

It might be interesting to check with Process Monitor which one it is.

Can you reproduce it just by doing a large tagging operation then (no RG scan)? How many files are you usually doing this on to trigger it?

I tried a large RG scan previously just to see if I could trigger anything (~3300 files), but it worked OK. I'll try again after enabling/using hibernation though (if only for interest's sake..). (Win 7 64-bit, Intel P55 chipset here). Most probably my drives were formatted under Vista.

[edit] OK, I tried a few large operations, mixed in with few hibernations and all went OK. So not sure where that points the finger. For some ideas - SATA controller, SATA controller driver, BIOS, ... ? I would have suggested trying a second clean install of Windows 7 on that PC (or even Vista) to eliminate more things, but since we don't know the actual cause of the issue it might not be a good idea unless done right..

ReplayGain tagging is setting off NTFS dirty bit

Reply #11 – 2009-12-27 10:56:18

After doing a large tag update to a list of approx 1000 files (removing a tag field), the dirty bit was set. So RG doesn't affect things, only the tag updating. If it matters, I have id3v2 compatibility mode set when writing tags.

ReplayGain tagging is setting off NTFS dirty bit

Reply #12 – 2009-12-27 11:53:03

Quote from: fraganator on 2009-12-27 10:56:18

If it matters, I have id3v2 compatibility mode set when writing tags.

No, after all you already confirmed that the problem is on the filesystem level, not on the file contents level. I suspect any (sufficiently large) write operation on the partition could trigger the problem, maybe even saving a text file in Notepad.

ReplayGain tagging is setting off NTFS dirty bit

Reply #13 – 2010-02-18 22:39:30

This is a bug in Windows 7, see [a href='index.php?act=findpost&pid=688762']here[/a].

Notice