Python string comparison similarity - 【StackMirror】|python

I am trying to compare two lists of data which has some free text denoting the same object. example

List 1 ['abc LLC','xyz, LLC']
List 2 ['abc , LLC','xyz LLC']

It is a simple example but the problem is there can be many changes like changes in case or adding some "." in between. Is there any python package that can do the comparison and give a measure of similarity?

python

2012-04-04 07:48
by Raman Narayanan

What do you mean by "probability" - Oliver Charlesworth 2012-04-04 07:50

@OliCharlesworth I think the author wants to find a percentage of similarity between two strings. Like if the strings are 85% similar - bezmax 2012-04-04 07:52

You don't want "probability", you want "similarity".

http://stackoverflow.com/questions/682367/good-python-modules-for-fuzzy-string-compariso - Joe 2012-04-04 07:52

@Max, yes that's what I meant. Thank - Raman Narayanan 2012-04-04 08:54

@all thanks for the hel - Raman Narayanan 2012-04-04 08:55

I believe you've got to define your problem more precisely: what kind of similarity are you detecting? What is the mathematical definition of your similarity? Otherwise people can only guess what you want.

Or maybe that is actually your question: you want people to suggest you a similarity definition (like Levenshtein Distance) - HongboZhu 2012-04-04 09:31

You could use an implementation of the Levenshtein Distance algorithm for non-precise string matching, for instance this one from Wikibooks.

Another option would be to, for instance, fold everything to lower case, remove spaces, etc. prior to raw comparison -- this of course depends on your use case:

import string, unicodedata
allowed = string.letters + string.digits
def fold(s):
  s = unicodedata.normalize("NFKD", unicode(s).lower()).encode("ascii", "ignore")
  s = "".join(c for c in s if c in allowed)
  return s

for example in ['abc LLC','xyz, LLC', 'abc , LLC','xyz LLC']:
  print "%r -> %r" % (example, fold(example))

would print

'abc LLC' -> 'abcllc'
'xyz, LLC' -> 'xyzllc'
'abc , LLC' -> 'abcllc'
'xyz LLC' -> 'xyzllc'

2012-04-04 07:54
by AKX

there's an excellent binary library that uses levenshtein distance (edit distance) between strings to estimate similarity. Give it a try:

https://github.com/miohtama/python-Levenshtein

2012-04-04 08:21
by Not_a_Golfer