Writing to compressed files

Aug 6, 2011 at 6:06 PM

Ken,

Trying to work out the rules writing to compressed and sparse files (seeking and writing got me exceptions as writing over a partially written sparse area is not supported).  Is it safe to presume you can write anywhere in the file as long as it's aligned on a 16*clustersize boundary and is a full 16*clustersize write?  With the last write in the file being allowed to be less?  Or do all writes need to be 16*clustersize sequentially from the start of the file on?

 

Thanks for any guidance!

Bill

Coordinator
Aug 6, 2011 at 6:54 PM

I think this is really a bug - DiscUtils should just do the right thing.  Can you post a stack trace?

 

Cheers,

Ken

Aug 7, 2011 at 6:14 PM

It happens when I write over a portion of a run that was already partially written.  The NotImplimentedException below fires in that case.  The run is there and was already properly compressed/written (based on the data previous written to that portion of the file)  I presumed the exception is because you didn't support reading in the current data, merging, and writing it out (hence the question).  If this is supposed to be working I'll reset the scenario and get a full stack trace (in Chicago today so will have to be after I get back home on Monday).

 

public override void WriteClusters(long startVcn, int count, byte[] buffer, int offset)

        {

            if (buffer.Length < (count * _bytesPerCluster) + offset)

            {

                throw new ArgumentException("Cluster buffer too small", "buffer");

            }

 

            int runIdx = 0;

            int totalWritten = 0;

            while (totalWritten < count)

            {

                runIdx = _cookedRuns.FindDataRun(startVcn + totalWritten, runIdx);

                CookedDataRun run = _cookedRuns[runIdx];

 

                if (run.IsSparse)

                {

                    throw new NotImplementedException("Writing to sparse datarun");

                }

 

                long focusVcn = startVcn + totalWritten;

                int toWrite = (int)Math.Min(count - totalWritten, run.Length - (focusVcn - run.StartVcn));

 

                long lcn = _cookedRuns[runIdx].StartLcn + (focusVcn - run.StartVcn);

                _fsStream.Position = lcn * _bytesPerCluster;

                _fsStream.Write(buffer, offset + (totalWritten * _bytesPerCluster), toWrite * _bytesPerCluster);

 

                totalWritten += toWrite;

            }

        }

Coordinator
Aug 7, 2011 at 6:24 PM

Ah - something else is going wrong then.  For sparse / compressed streams, the intent is that RawClusterStream.WriteClusters is wrapped by either CompressedClusterStream.WriteClusters or SparseClusterStream.WriteClusters.

Both of those prefix calls to the raw stream with calls like this:

            _rawStream.AllocateClusters(startVcn, count);
            _rawStream.WriteClusters(startVcn, count, buffer, offset);

That AllocateClusters call should ensure the exception you're seeing doesn't fire.

Cheers,

Ken

Coordinator
Aug 8, 2011 at 6:40 AM

Bill,

I found a bug in the code to allocate clusters before they're written.  Hopefully this is fixed now.

See change 0798cefd408e

Cheers,

Ken

Aug 9, 2011 at 11:44 PM

Thanks, that fixed problem #1.  Just opened an issue on #2 (think I spotted the error, but would like you to confirm it)

Bill

 

Aug 18, 2011 at 10:49 PM

The checked in fix worked.  Iterated over a very compressed disk 100 times and all is working well.