How to ensure that a file has unique line in if the size of the file is very big

Go To


Language: File size: 1GB, and stuff.

Encoding of the text file: UTF8 (so each character is represented by different numbers of bytes).

Collation: UnicodeCI (when several characters are essentially the same, the most popular version will be the one unique.). I think I know how to handle t his one.

Because each character is represented by different numbers of bytes and each line has different numbers of characters, the number of bytes in each line also vary.

I suppose we have to compute hash for each line. We also need to store buffers location where the line each. Then we have to compare buffers. Then we will check whether the same line shows up or not.

Is there special functions best for that?

2012-04-04 04:32
by user4951


Depending on how long the lines are, you may be able to compute an MD5 hash value for each line and store than in a HashMap:

Using sr As New StreamReader("myFile")
    Dim lines As New HashSet(Of String)
    Dim md5 As New Security.Cryptography.MD5Cng()

    While sr.BaseStream.Position < sr.BaseStream.Length
        Dim l As String = sr.ReadLine()
        Dim hash As String = String.Join(String.Empty, md5.ComputeHash(System.Text.Encoding.UTF8.GetBytes(l)).Select(Function(x) x.ToString("x2")))

        If lines.Contains(hash) Then
            'Lines are not unique
            Exit While
        End If
    End While
End Using

Untested, but this may be fast enough for your needs. I can't think of something much faster that still maintains some semblance of conciseness :)

2012-04-04 04:37
by Ry-
What's the size of the md5? One imperfect method is to use an integer hash that compacts memory. But then again, several different lines must have the same hash. That's why we need to store buffer locations too - user4951 2012-04-04 04:46
+1 also I modified your code heavily. That exit while is a bug, etc - user4951 2012-04-04 05:57
sr.BaseStream.Position < sr.BaseStream.Length need to be changed. Basically sr.BaseStream.Position often advance a lot more than the string length. It always increase by 1024 at a tim - user4951 2012-04-04 06:40
@JimThio: Sorry. MD5 is 32 bytes, Exit While is not a bug, and if you know a better way to check for the end of the stream I'd be happy to hear it - Ry- 2012-04-04 13:37
yes it's end of strea - user4951 2012-04-06 06:43


This is the contemporary answer

Public Sub makeUniqueForLargeFiles(ByVal strFileSource As String)
    Using sr As New System.IO.StreamReader(strFileSource)
        Dim changeFileName = reserveFileName(strFileSource, False, True)
        Using sw As New System.IO.StreamWriter(reserveFileName(strFileSource, False, True), False, defaultEncoding)
            Dim lines As New Generic.Dictionary(Of Integer, System.Collections.Generic.List(Of Long))
            While sr.BaseStream.Position < sr.BaseStream.Length
                Dim offset = sr.BaseStream.Position
                Dim l As String = sr.ReadLine()
                Dim nextOffset = sr.BaseStream.Position
                Dim hash = l.GetHashCode
                Do ' a trick to put the for each in a "nest" that we can exit from
                    If lines.ContainsKey(hash) Then
                        Using sr2 = New System.IO.StreamReader(strFileSource)
                            For Each offset1 In lines.Item(hash)
                                sr2.BaseStream.Position = offset1
                                Dim l2 = sr2.ReadLine
                                If l = l2 Then
                                    Exit Do 'will sr2.dispose be called here?
                                End If
                        End Using
                        lines.Add(hash, New Generic.List(Of Long))
                    End If
                Loop While False
                sr.BaseStream.Position = nextOffset
            End While
        End Using
    End Using
End Sub
2012-04-04 05:58
by user4951