NTFS FS Performance

Feb 24, 2011 at 4:04 PM

I am noticing some performance issues using NTFS when copying large numbers of files (20000 indivdual tif images) multiple times

I create a vhd disk as below

 long diskSize = int.Parse(textSize.Text) * 1024 * 1024;
                using (Stream vhdStream = File.Create(textNewDisk.Text))
                {
                    Disk disk = Disk.InitializeDynamic(vhdStream, DiscUtils.Ownership.Dispose, diskSize);                                                
                    BiosPartitionTable bpt = BiosPartitionTable.Initialize(disk);

                    int i = radioNTFS.Checked ? bpt.Create(WellKnownPartitionType.WindowsNtfs, true) : bpt.Create(WellKnownPartitionType.WindowsFat, true); 
                        
                    VolumeManager vm = new VolumeManager(disk);
                    LogicalVolumeInfo[] lvi= vm.GetLogicalVolumes();

   using (NtfsFileSystem fs = NtfsFileSystem.Format(lvi[i], "Test"))
   {
       fs.CreateDirectory("PODS");
   }
}
I then try and write the 20000 files into the PODS directory i created using the following
 disk = Disk.OpenDisk(textExistDisk.Text, FileAccess.ReadWrite);
 VolumeManager vm = new VolumeManager(disk);

 ss= vm.GetLogicalVolumes()[0].Open();

if (NtfsFileSystem.Detect(ss))
{
     ntfsOpenedDisk = new NtfsFileSystem(ss);
}

foreach (string f in files)
{
using (Stream stream = ntfsOpenedDisk.OpenFile(destfile, FileMode.Create))
{
   if (stream.CanWrite)
   {
        byte[] fileContent = File.ReadAllBytes(f);
        stream.Write(fileContent, 0, fileContent.GetLength(0));
    }
}
}
The first time it passes through the performance is good and it writes all the files in roughly 45 secs. 
If i then create a new directory on the root of the disk and try copying the same files in to that directory about 3/4 of the way through performance
drops of considerably and the last 25% takes an age! It occainsonally errors too and doesnt complete at all.
The error it gives is - Attempt to get an MFT record with an old reference.
Could this be an issue with the implementation of NTFS or the way I am using it?

 

Coordinator
Feb 25, 2011 at 7:57 AM

There are a couple of things that could cause slow-down.

Creating short names for files can be quite slow, especially if you have a lot of files with similar names.  You can disable short name creation:

ntfsOpenedDisk.NtfsOptions.ShortNameCreation = ShortFileNameOption.Disabled;

The other cause can be if the disk is getting full - DiscUtils starts getting pretty inefficient when there's only approx 12.5% of the disk left - finding free clusters can get quite slow.

 

The error you're seeing is quite worrying - especially if it doesn't happen every time.  Do you always use the same size of disk, and add the same files?  It would be useful to get the full stack trace of the error, if you can reproduce it.

 

Cheers,

Ken

 

Feb 25, 2011 at 8:19 AM

Thanks Ken

I already have the short filename option disabled.

I always use the same disk size and files when testing. I create a 500mb NTFS disk. The 20000 image files take up about 150mb on disk. The first time i copy them all its fine, the second time it gets 75% of the way through and then generates the error. So its not getting close to being full, only just over half.

This is the stack trace of the error

at DiscUtils.Ntfs.MasterFileTable.GetRecord(FileRecordReference fileReference)
   at DiscUtils.Ntfs.File.RemoveAttributeExtent(AttributeReference extentRef)
   at DiscUtils.Ntfs.File.RemoveAttributeExtents(NtfsAttribute attr)
   at DiscUtils.Ntfs.File.MakeAttributeResident(AttributeReference attrRef, Int32 maxData)
   at DiscUtils.Ntfs.File.FileStream.ChangeAttributeResidencyByLength(Int64 value)
   at DiscUtils.Ntfs.File.FileStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at DiscUtils.SparseStream.SparseWrapperStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at DiscUtils.BlockCacheStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at DiscUtils.Ntfs.Bitmap.SetByte(Int64 index, Byte value)
   at DiscUtils.Ntfs.Bitmap.MarkPresent(Int64 index)
   at DiscUtils.Ntfs.Bitmap.AllocateFirstAvailable(Int64 minValue)
   at DiscUtils.Ntfs.Index.AllocateBlock(IndexNode parentNode, IndexEntry parentEntry)
   at DiscUtils.Ntfs.IndexNode.Divide()
   at DiscUtils.Ntfs.IndexNode.AddEntry(IndexEntry newEntry, Boolean promoting)
   at DiscUtils.Ntfs.IndexNode.AddEntry(IndexEntry newEntry, Boolean promoting)
   at DiscUtils.Ntfs.IndexNode.AddEntry(IndexEntry newEntry, Boolean promoting)
   at DiscUtils.Ntfs.IndexNode.AddEntry(IndexEntry newEntry, Boolean promoting)
   at DiscUtils.Ntfs.IndexNode.AddEntry(Byte[] key, Byte[] data)
   at DiscUtils.Ntfs.Index.set_Item(Byte[] key, Byte[] value)
   at DiscUtils.Ntfs.IndexView`2.set_Item(K key, D value)
   at DiscUtils.Ntfs.ObjectIds.Add(Guid objId, FileRecordReference mftRef, Guid birthId, Guid birthVolumeId, Guid birthDomainId)
   at DiscUtils.Ntfs.File.CreateNew(INtfsContext context, FileRecordFlags flags)
   at DiscUtils.Ntfs.File.CreateNew(INtfsContext context)
   at DiscUtils.Ntfs.NtfsFileSystem.OpenFile(String path, FileMode mode, FileAccess access)
   at DiscUtils.DiscFileSystem.OpenFile(String path, FileMode mode)
   at VHDTestApp.Form1.btnAddFiles_Click(Object sender, EventArgs e)
I can send you my little test app and files if you need it?
Coordinator
Feb 25, 2011 at 9:50 PM

Thanks.

With the info above, I've managed to recreate the slow-down, and the error.

Cheers,

Ken

Coordinator
Feb 26, 2011 at 11:16 AM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Mar 1, 2011 at 3:12 PM

 

Hi Ken

I have downloaded the fix and tried it. It does indeed improve the performance and fix the error i was getting, however I now get another error on creating a 3rd directory in the root.

1. Create a 500MB NTFS disk
2. Create a directory called Test and copy in 20000 files (150mb)
3. Create another directory called Test1 and copy in 20000 files (150mb)
4. Create another called Test2 and copy in 20000 files (150mb). At this point (about 20% of the way through) I get the error - Corrupt record

Here is the stack trace:

at DiscUtils.Ntfs.FixupRecordBase.FromBytes(Byte[] buffer, Int32 offset, Boolean ignoreMagic)
at DiscUtils.Ntfs.FixupRecordBase.FromBytes(Byte[] buffer, Int32 offset)
at DiscUtils.Ntfs.IndexBlock..ctor(Index index, IndexNode parentNode, IndexEntry parentEntry, BiosParameterBlock bpb)
at DiscUtils.Ntfs.Index.GetSubBlock(IndexNode parentNode, IndexEntry parentEntry)
at DiscUtils.Ntfs.IndexNode.TryFindEntry(Byte[] key, IndexEntry& entry, IndexNode& node)
at DiscUtils.Ntfs.IndexNode.TryFindEntry(Byte[] key, IndexEntry& entry, IndexNode& node)
at DiscUtils.Ntfs.Index.set_Item(Byte[] key, Byte[] value)
at DiscUtils.Ntfs.IndexView`2.set_Item(K key, D value)
at DiscUtils.Ntfs.ObjectIds.Add(Guid objId, FileRecordReference mftRef, Guid birthId, Guid birthVolumeId, Guid birthDomainId)
at DiscUtils.Ntfs.File.CreateNew(INtfsContext context, FileRecordFlags flags)
at DiscUtils.Ntfs.File.CreateNew(INtfsContext context)
at DiscUtils.Ntfs.NtfsFileSystem.OpenFile(String path, FileMode mode, FileAccess access)
at DiscUtils.DiscFileSystem.OpenFile(String path, FileMode mode)
at VHDTestApp.Form1.btnAddFiles_Click(Object sender, EventArgs e) in D:\Develop\Tools\DiscUtils\TestApp\VHDTestApp\Form1.cs:line 239