StackOverflowException

Feb 3, 2011 at 8:31 PM

When copying files and folders to a filesystem from a local drive i keep getting a StackOverflowException. It is doing it when i use code to copy files and when explorer copies alot using my bridge. (The bridge logs the reads and writes then crashes with StackOverflowException.

Coordinator
Feb 5, 2011 at 11:10 AM

Can you provide a stack trace of where the crash is occuring?

 

Cheers,

Ken

Feb 5, 2011 at 2:52 PM
Edited Feb 5, 2011 at 3:42 PM

This is the only details vs gives:

System.StackOverflowException was unhandled

An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll

 

{Cannot evaluate expression because the current thread is in a stack overflow state.}

Heres the code im using to copy: It processes around 3857 directorys. (some empty and some with 3-5 files.)

fs is a NtfsFilesystem that is loaded.

    Sub UpdateDB()
        fs.CreateDirectory("\data")
        'fs.CreateDirectory("\data\herbs")
        'fs.CreateDirectory("\data\glossary")
        'fs.CreateDirectory("\data\undo")
        'fs.CreateDirectory("\data\linking")
        Dim root As String = "C:\"
        log = File.Open("log.log", FileMode.Append, FileAccess.Write)
        doDir(root & "data", root)
        'doDir("data\glossary", root)
        'doDir("data\undo", root)
        'doDir("data\linking", root)
    End Sub
    Dim log As FileStream
    Sub doDir(ByVal dir As String, ByVal root As String)
        Console.WriteLine(dir)
        For Each f In My.Computer.FileSystem.GetFiles(dir)
            Dim outF As Stream = fs.OpenFile(f.Replace(root, ""), FileMode.Create, FileAccess.ReadWrite)
            Dim inF As FileStream = File.Open(f, FileMode.Open, FileAccess.ReadWrite)
            inF.Seek(0, SeekOrigin.Begin)
            outF.Seek(0, SeekOrigin.Begin)
            CopyStreamContents(inF, outF)
            inF.Close()
            outF.Close()
        Next
        Dim dirs = My.Computer.FileSystem.GetDirectories(dir)
        For I = 0 To dirs.Count - 1
            Dim bytes As Byte() = ASCIIEncoding.ASCII.GetBytes(dirs(I) & vbCrLf)
            log.Write(bytes, 0, bytes.Length)
            fs.CreateDirectory(dirs(I).Replace(root, ""))
            doDir(dirs(I), root)
        Next
    End Sub
Coordinator
Feb 5, 2011 at 4:02 PM

So 'root' is on a Dokan drive, and that bridge crashes?

Looking at your bridge, noticed a few things:

  • There's no locking - think you need to add SyncLock(p1) ... End SyncLock around all use of DiscUtils in Class1.vb
  • The write function doesn't return the number of bytes written, set writtenBytes to buffer.Length
  • ReadFile and WriteFile are not closing the file stream, just leave it dangling

WriteFile something more like this?

    Public Function WriteFile(ByVal filename As String, ByVal buffer() As Byte, ByRef writtenBytes As UInteger, ByVal offset As Long, ByVal info As Dokan.DokanFileInfo) As Integer Implements Dokan.DokanOperations.WriteFile
        log("WRITEFILE " & filename & " offset=" & offset & " buffer.Length=" & buffer.Length)
        Try
            SyncLock (p1)

                Using fs As Stream = p1.OpenFile(filename, FileMode.OpenOrCreate, FileAccess.ReadWrite)
                    If Not fs.CanWrite Then Return -DokanNet.ERROR_ACCESS_DENIED
                    fs.Seek(offset, SeekOrigin.Begin)
                    fs.Write(buffer, 0, buffer.Length)
                    writtenBytes = buffer.Length
                    fs.Flush()
                End Using

                Return 0
            End SyncLock

        Catch generatedExceptionName As Exception
            Debug.Write(String.Format("WRITEFILE: {0} {1} {2}", filename, generatedExceptionName.Message, generatedExceptionName.StackTrace))
            Return 0
        End Try
    End Function


May be worth fixing these as a first step.

Feb 5, 2011 at 4:42 PM

Ok ill try thatt. Although root is the C: drive (Physical) copying to a discutils drive. This was tried without mounting in dokan.

Root was used to convert paths to relative paths. 

Feb 5, 2011 at 6:32 PM

Still getting an stackOverflowexception. 

I tried SyncLock on the bridge and now it wont create files. :S

Coordinator
Feb 6, 2011 at 10:27 AM

I'm struggling to reproduce - just tried copying a folder structure with 55,777 files in 11,097 folders, with no exception.

Can you post the content of CopyStreamContents - may give a hint as to what's going on.

Cheers,

Ken

Feb 6, 2011 at 3:24 PM
Edited Feb 6, 2011 at 3:46 PM

 

Strange... Im only copying about 8000 files total.... My folder structures is 8 levels deep max.
Heres a link to the entire module used for testing. (Its a modified version of the tester module in the bridge project.) http://dl.dropbox.com/u/10471884/Module1.vb

 

Heres the code:

 


 Public Sub CopyStreamContents(ByVal objInput As Stream, ByVal objOutput As Stream)

        ' assert these are the right kind of streams
        If objInput Is Nothing Then Throw New ArgumentNullException("input")
        If objOutput Is Nothing Then Throw New ArgumentNullException("output")
        If Not objInput.CanRead Then Throw New ArgumentException("Input stream must support CanRead")
        If Not objOutput.CanWrite Then Throw New ArgumentException("Output stream must support CanWrite")

        ' skip if the input stream is empty (if seeking is supported)
        If objInput.CanSeek Then If objInput.Length = 0 Then Exit Sub

        ' allocate buffer (if all pre-conditions are met)
        Dim buffer(1023) As Byte
        Dim count As Integer = buffer.Length

        ' iterate read/writes between streams
        Do
            count = objInput.Read(buffer, 0, count)
            If count = 0 Then Exit Do
            objOutput.Write(buffer, 0, count)
        Loop

    End Sub

 

Coordinator
Feb 9, 2011 at 12:18 AM

Sadly, I'm still failing to reproduce this - I have tried exceeding the disk space, and using directory hierarchies 10 deep.

I suspect there's a bug in DiscUtils that your particular data set is exposing.  Can you try identifying the last file that copies correctly for you, and cutting down the set of files you're copying to see if you can narrow the problem down?

Would be useful if you could post a (relatively) small set of files that causes the problem for you.

 

I've also posted a new build of DiscUtils 0.10 (http://discutils.codeplex.com/releases/view/50826), based on the latest changes, if you could try using that - just in case we're chasing ghosts...

Cheers,

Ken

Feb 9, 2011 at 1:03 PM

I tried removing the file that it was stopping on and it would stop on the next. File Index 2532 in array of files.

Heres my revised code that im using:

 Dim cnt = 0
    Dim fcnt = 0
    Dim fs As DiscFileSystem
    Sub UpdateDB()
        fs.CreateDirectory("\data")
        'fs.CreateDirectory("\data\herbs")
        'fs.CreateDirectory("\data\glossary")
        'fs.CreateDirectory("\data\undo")
        'fs.CreateDirectory("\data\linking")
        Dim root As String = "C:\"
        log = File.Open("log.log", FileMode.Append, FileAccess.Write)
        doDir(root & "data", root)
        'doDir("data\glossary", root)
        'doDir("data\undo", root)
        'doDir("data\linking", root)
    End Sub
    Dim log As FileStream
    Sub doDir(ByVal dir As String, ByVal root As String)
        Dim dirs = My.Computer.FileSystem.GetDirectories(dir, FileIO.SearchOption.SearchAllSubDirectories, "*")
        For I = 0 To dirs.Count - 1
            cnt += 1
            Dim bytes As Byte() = ASCIIEncoding.ASCII.GetBytes(cnt & dirs(I) & vbCrLf)
            Console.WriteLine(cnt & dirs(I))
            log.Write(bytes, 0, bytes.Length)
            SyncLock fs
                If Not dirs(I).Contains("refsref") And Not dirs(I).Contains("refref") Then fs.CreateDirectory(dirs(I).Replace(root, ""))
            End SyncLock
            'doDir(dirs(I), root)
        Next
        SyncLock fs
            Console.WriteLine(dir)
            Dim files = My.Computer.FileSystem.GetFiles(dir, FileIO.SearchOption.SearchAllSubDirectories)
            For I = 0 To files.Count - 1
                Dim f As String = files(I)
                cnt += 1
                fcnt += 1
                Dim bytes As Byte() = ASCIIEncoding.ASCII.GetBytes(fcnt & " " & cnt & " " & f & vbCrLf)
                Console.WriteLine(fcnt & " " & cnt & " " & f)
                If fcnt = 2532 Then
                    Console.Write("FAIL?")
                End If
                log.Write(bytes, 0, bytes.Length)
                log.Flush(True)
                Dim outF As Stream = fs.OpenFile(f.Replace(root, ""), FileMode.Create, FileAccess.ReadWrite)
                Dim inF As FileStream = File.Open(f, FileMode.Open, FileAccess.ReadWrite)
                inF.Seek(0, SeekOrigin.Begin)
                outF.Seek(0, SeekOrigin.Begin)
                Dim data(1024 * 1024) As Byte
                Dim count As Integer = inF.Read(data, 0, inF.Length)
                outF.Write(data, 0, count) 'CopyStreamContents(inF, outF)
                inF.Close()
                outF.Close()
            Next
        End SyncLock
    End Sub
its stopping on fs.openfile and with sourcecode it goes to line 482 in NonResidentAttributeBuffer.cs . (This is source for your 0.10 release)

Coordinator
Feb 9, 2011 at 11:09 PM

Thanks - that points the finger at the code that adds new entries into a directory.  I'll start digging into that code.

Cheers,

Ken

Feb 20, 2011 at 11:46 PM

Any fixes yet? I would help but im not quite sure how DiscUtils works. :S

Coordinator
Feb 25, 2011 at 8:59 AM

I haven't managed to find/reproduce the problem yet.  If I'm right, and it's a problem in the directory code it's proving elusive.

Could you post the set of files (+ size of each file) you're copying in?  Hopefully with that info, I can exactly reproduce the behaviour you're seeing.

 

Cheers,

Ken

Coordinator
Feb 26, 2011 at 5:13 PM

I've just uploaded some changes that may fix issues that are contributing to what you're seeing - in cases with large numbers of (small) files, directories and other indexes were getting corrupted.

Can you try the latest changes out please, and let me know if you're still seeing the problem.

Cheers,

Ken

Mar 10, 2011 at 12:32 AM

Still happening. I will try to debug it tomorrow. Copied 4000-5000 files before the error

Mar 10, 2011 at 4:14 PM
Edited Mar 10, 2011 at 7:13 PM

Now stopping at 486 in same file.

EDIT: Full stack trace here: http://dl.dropbox.com/u/10471884/Stack.txt

Its repeatedly doing this:

DiscUtils.dll!DiscUtils.Ntfs.File.UpdateRecordInMft() Line 346 + 0xb bytes C#

  DiscUtils.dll!DiscUtils.Ntfs.MasterFileTable.WriteRecord(DiscUtils.Ntfs.FileRecord record = {DiscUtils.Ntfs.FileRecord}) Line 366 + 0xd bytes C#

  DiscUtils.dll!DiscUtils.Ntfs.MasterFileTable.AllocateRecord(DiscUtils.Ntfs.FileRecordFlags flags = None) Line 247 + 0xb bytes C#

  DiscUtils.dll!DiscUtils.Ntfs.File.ExpelAttribute(DiscUtils.Ntfs.FileRecord record = {DiscUtils.Ntfs.FileRecord}) Line 855 + 0xf bytes C#

 

EDIT: I just started the bridge after deleting the fs file and its throwing at a different location: Not sure if its related or not. Did not recur when i restarted the bridge. Im doing a checkin of the bridge now. 

Line 1774 of ntfsfilesystem.cs

Null Refernce Exception

   at DiscUtils.Ntfs.NtfsFileSystem.GetFile(Int64 index) in C:\Users\Adam\Downloads\DiscUtilsSrc-0.10\src\Ntfs\NtfsFileSystem.cs:line 1774

   at DiscUtils.Ntfs.NtfsFileSystem.get_VolumeLabel() in C:\Users\Adam\Downloads\DiscUtilsSrc-0.10\src\Ntfs\NtfsFileSystem.cs:line 158

   at Tester.Module1.Main() in C:\Users\Adam\Documents\Visual Studio 2010\Projects\DokanDiscUtilsBridge\Tester\Module1.vb:line 124

   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

Coordinator
Mar 11, 2011 at 8:46 AM

Thanks - the full stack trace is very helpful.  I guessed wrong when I thought it was a directory issue.

The Null Reference Exception is probably unrelated.

Cheers,

Ken

Coordinator
Mar 12, 2011 at 6:08 PM

I think I've now fixed this - the latest set of changes (up to 05d3819ce88e) should prevent a lot of file fragmentation that was occuring, and also cope properly with a fragmented Master File Table.

Regards,

Ken

Mar 25, 2011 at 7:22 AM

Stilll happening. Hmmmmm...... Widh i knew more about filesystems and DiscUtils...

Coordinator
Apr 2, 2011 at 10:31 AM

Could you grab a new stack trace - I thought I'd caught the recursion case that was causing the problems.

Not sure if that's still the problem, or there's something new going on.

 

Cheers,

Ken

Apr 2, 2011 at 5:06 PM
Edited Apr 3, 2011 at 2:35 AM

The stack trace was nearly identical. the only difference was where i
changed my code

On 4/2/11, festive_ken <notifications@codeplex.com> wrote:
> From: festive_ken
>
> Could you grab a new stack trace - I thought I'd caught the recursion case
> that was causing the problems.Not sure if that's still the problem, or
> there's something new going on. Cheers,Ken
>
>