Mikaela/Limnoria
1
0
Fork 0
mirror of https://github.com/Mikaela/Limnoria.git synced 2024-12-23 19:22:45 +01:00
Code Issues Projects Releases Wiki Activity

utils.str: Prevent issue with tokens like '\x80' in Python 3.

Browse Source
This commit is contained in:
Valentin Lorentz 2013-01-23 15:48:24 +01:00
parent b421120bad
commit 2ace534bdb
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
src/callbacks.py
View File

@ -307,7 +307,10 @@ class Tokenizer(object):
else:
token = codecs.getencoder('utf8')(token)[0]
token = codecs.getdecoder('unicode_escape')(token)[0]
token = token.encode('iso-8859-1').decode()
try:
token = token.encode('iso-8859-1').decode()
except: # Prevent issue with tokens like '"\\x80"'.
pass
return token
def _insideBrackets(self, lexer):