Python NLTK snowball stemmer UnicodeDecodeError in terminal but not Eclipse PyDev -


i'm using snowball stemmer stem words in documents shown in below code snippet. 

    stemmer = englishstemmer()     # stem, lowercase, substitute punctuations, remove stopwords.     attribute_names = [stemmer.stem(token.lower()) token in wordpunct_tokenize(re.sub('[%s]' % re.escape(string.punctuation), '', doc)) if token.lower() not in stopwords.words('english')]

when run on documents using pydev in eclipse, receive no errors. when run in terminal (mac osx) receive below error. can please help?

file "data_processing.py", line 171, in __filter__ attribute_names = [stemmer.stem(token.lower()) token in   wordpunct_tokenize(re.sub('[%s]' % re.escape(string.punctuation), '', doc)) if token.lower()     not in stopwords.words('english')]  file "7.3/lib/python2.7/site-packages/nltk-2.0.4-py2.7.egg/nltk/stem/snowball.py", line   694, in stem word = (word.replace(u"\u2019", u"\x27")  unicodedecodeerror: 'ascii' codec can't decode byte 0xc2 in position 7: ordinal not in range(128)

this works in pydev because configures python work in encoding of console (which utf-8).

you can reproduce same error in pydev if go run configuration (run > run configurations) on 'common' tab want encoding ascii.

this happens because word string , you're replacing unicode chars.

i hope code below sheds light you:

this considering ascii default encoding:

>>> 'íã'.replace(u"\u2019", u"\x27") traceback (most recent call last):   file "<stdin>", line 1, in <module> unicodedecodeerror: 'ascii' codec can't decode byte 0xa1 in position 0: ordinal not in range(128)

but if in unicode, works (you may need encode afterwards encoding expect if expect deal strings , not unicode).

>>> u'íã'.replace(u"\u2019", u"\x27") u'\xed\xe3'

so, can make string unicode before replace

>>> 'íã'.decode('cp850').replace(u"\u2019", u"\x27") u'\xed\xe3'

or can encode replace chars

>>> 'íã'.replace(u"\u2019".encode('utf-8'), u"\x27".encode('utf-8')) '\xa1\xc6'

note must know what's actual encoding you're working on in place (so, although i'm using cp850 or utf-8 in examples, may different encodings have use)


Comments

Post a Comment

Popular posts from this blog

python - How to create a legend for 3D bar in matplotlib? -

Image
given ax = plt.subplot() : ax.bar()[0] can passed plt.legend() . however, ax.bar3d() returns none . how create legend displayed bars? update: passing legend="stuff" ax.bar3d() , calling ax.legend() raises /usr/lib/python2.6/site-packages/matplotlib/axes.py:4368: userwarning: no labeled objects found. use label='...' kwarg on individual plots. warnings.warn("no labeled objects found. " you need use proxy artist legends not supported. this code: from mpl_toolkits.mplot3d import axes3d import matplotlib.pyplot plt import numpy np fig = plt.figure() ax = fig.add_subplot(111, projection='3d') x, y = np.random.rand(2, 100) * 4 hist, xedges, yedges = np.histogram2d(x, y, bins=4) elements = (len(xedges) - 1) * (len(yedges) - 1) xpos, ypos = np.meshgrid(xedges[:-1]+0.25, yedges[:-1]+0.25) xpos = xpos.flatten() ypos = ypos.flatten() zpos = np.zeros(elements) dx = 0.5 * np.ones_like(zpos) dy = dx.copy() dz = hist.flatten() ax...
Read more

java - Multi-Label Document Classification -

i have database in store data based upon following 3 fields: id, text, {labels}. note each text has been assigned more 1 label \ tag \ class. want build model (weka \ rapidminer \ mahout) able recommend \ classify bunch of labels \ tags \ classes given text. i have heard svm , naive bayes classifier, not sure whether support multi-label classification or not. guides me right direction more welcome! the basic multilabel classification method one-vs.-the-rest (ovr), called binary relevance (br). basic idea take off-the-shelf binary classifier, such naive bayes or svm, create k instances of solve k independent classification problems. in python-like pseudocode: for each class k: learner = svm(settings) # example labels = [class_of(x) == k x in samples] learner.learn(samples, labels) then @ prediction time, run each of binary classifiers on sample , collect labels predict positive. (both training , prediction can done in parallel, since problems assumed ...
Read more

php - Dynamic url re-writing using htaccess -

how can rewrite http://mydomain.com/test/master.php?catid=2&subcatid=8 http://mydomain.com/test/master.php/catname/subcat using .htaccess file. once rewritten, how can access these parameters in master.php? note: untested, let me know if doesn't work. rewriterule ^master.php/([a-za-z0-9_-]+)/([a-za-z0-9_-]+)$ /master.php?catid$1&subcatid=$2 [l]
Read more