Using NTFSFileSystem with physical NTFS Partition

Jul 6, 2010 at 1:16 PM

Hi,

I am looking for a way to quickly copy (physical) Files to a VHD (NTFSFileSystem Object).

I tried to open the partition as FileStream and create a new NTFSFileSystem based on it, but that does not work.

Do you have an idea, if and how to do it?

 

Actually I am copying with reading all file into memory and putting it into NTFSFileSystem-file. But that is quite slow.

I hope a direct stream copy can be faster...

 

 

Regards,

Jango

Jul 6, 2010 at 2:10 PM

Hey Jangoman,

I would suggest you, to have a look into the example osclone. It does a copy of all files and stuff from one virtual disk into another.
It includes creating a new virtual disk, partitioning it and formating the partition. Copying the files streambased from the one disk into the other and migrating the
relevant stuff which is needed to get it bootable.

I hope that helps, else some example code what you already tried would be easier to help you.

Ciao Ephraim

Jul 7, 2010 at 10:04 AM

Hi Ephraim,

thanks for the suggestion. OSClone was the example that gave me the idea. The obstacle with bootable VHD is already solved. (At least XP)

My problem is the performance at copying from pyhsical to virtual. Actual code is like this, but its quite slow. (I also tried "smaller pieces" instead of ReadAllBytes, but it was not faster.)


        static bool CopyDirToVHD(NtfsFileSystem fs, System.IO.DirectoryInfo sourceDir)
        {
[..]

                            FileAttributes fileAttributes = File.GetAttributes(fi.FullName);
[..]

                            using (Stream stream = fs.OpenFile(fileNameDest, FileMode.Create))
                            {
                                byte[] fileContent = File.ReadAllBytes(fi.FullName);
                                stream.Write(fileContent, 0, fileContent.GetLength(0));
                                
                                stream.Close();

                                fs.SetAttributes(fileNameDest, fileAttributes);
[..]


What I tried now was "putting the physical NTFS to the NtfsFileSystem class", with the hope that that kind of copy action would be faster. (CopyFile function from OSClone example, copying streams directly.)

But the "conversion" from real NTFS to sparsestream must be wrong, as I get an "unkown error" when interpreting it as NtfsFileSystem (new NtfsFileSystem(sourcePartStream);)

 

[..]
                                handle = CreateFile(@"\\?\GLOBALROOT\Device\Harddisk0\Partition" + iPartition.ToString(), GENERIC_READ, FILE_SHARE_READ, IntPtr.Zero, OPEN_EXISTING, 0, IntPtr.Zero);
                               
                                FileStream partitionStream = new FileStream(handle, FileAccess.Read);
                                partitionStream.Read(bufVBR, 00, iVBRSize);
                                partitionStream.Position = 0;
                                

                                // Format partition with NTFS
                                NtfsFileSystem fs =
                                    NtfsFileSystem.Format(pvi[iPartition],sPartName[iPartition]);

                                
                                // Write VBR 
                                SparseStream vbrStream = pvi[iPartition].Open();
                                vbrStream.Seek(0x54, 0);
                                vbrStream.Write(bufVBR, 0x54, (iVBRSize - 0x54));
                                

[..]                                
                                SparseStream sourcePartStream = SparseStream.FromStream(
                                    partitionStream, Ownership.None);

                                sourcePartStream = new BlockCacheStream(sourcePartStream, Ownership.None);

                                NtfsFileSystem sourceNtfs = new NtfsFileSystem(sourcePartStream);

                                CopyFile(sourceNtfs, fs, "\\\\Somebigtestfile.txt");

                                partitionStream.Close();

Maybe you have some inspiration for me...

 

Ciao,

Jango

Jul 7, 2010 at 10:22 AM
Hey Jango, copying the whole stream of the disk would take more time as copying the files itself, as you copy the not-used parts too and so it takes longer and the virtualdisk gets bigger than it needs to be. IMHO, ReadAllBytes is not good, as when it comes to big files, you will need e.g. 2 GB memory as you read all bytes. I would suggest, try it with copying 10MB junks in a static buffer, so the buffer doesn't need to be created everytime. I think it's done that way in the OSClone example. Another thing what might bring some more performance is to copy not from the disk itself, rather than do a volume shadow copy of the disk and read the stuff from the "virtual" physicaldevice you get then. For that, have a look at AlphaVSS and AlphaFS projects here by codeplex. If you are going to use AlphaVSS you should make sure to dispose it in any situation on XP correctly, as not disposing it will end in a situation where you can't create new shadow copies. And if you want to copy the whole disk, make sure to exclude hibernate.sys and pagefile.sys, as they are big and unneeded! Ciao Ephraim
Jul 7, 2010 at 11:02 AM

Hi Ephraim,

no it was not my plan to copy the whole "partition stream". I just thought, maybe it is faster if I abstract the real NTFS to a "virtual" using the NtfsFileSystem class.

But guess what? You suggested things that I did up to now, too. That is good, as it shows me that my direction seems to be the right one :-)

- chunks instead of readallbytes

- using of (Alpha) VSS and the XP limitation (I remember the discussion in the group of alpha VSS)

- excluding pagefile and hibernate (and maybe recycle.bin)

 

Thanks,

jango

 

 

Coordinator
Jul 14, 2010 at 11:34 PM

Hi Folks,

You might want to take a look at the new DiskClone sample I've just uploaded.  It copies the partition stream (which is more efficient that file copying because it's all linear reads and writes), but with some twists:

  • Uses VSS to create a volume snapshot - doesn't use Alpha VSS, so limited to Win7 (and maybe Vista)
  • Files like the page file, hibernation file, and some large VSS files are removed from the image as it's copied (a snapshot stream holds the modify parts of the stream in memory)
  • The NTFS cluster bitmap is used to only copy clusters that are in use (only works with NTFS partitions)
  • The VHD file is generated dynamically (so in theory you could transmit the volume contents directly to a web server or similar as a dynamic VHD without needing to write anything to a local disk)

Cheers,

Ken

Jul 15, 2010 at 9:41 AM

Hey Ken,

nice stuff!! .... But why is VSS limited to Win7 ? ... On WinXp VSS is supported since SP2 but only one Snapshot at the same time.

I think I will have a look at it :).

Ciao Ephraim

Coordinator
Jul 15, 2010 at 10:39 PM
I used p/invoke to call into VSS. It appears there's a bug in VSS in XP that means it won't work with p/invoke. Specifically, CreateVssBackupComponents fails with a COM interop problem - can't call QueryInterface to get a reference to the IVssBackupComponents interface. Seems to work fine on Windows 7 though. Cheers, Ken