Unicode encoding error python

Ошибки при конвертации#

При конвертации между строками и байтами очень важно точно знать, какая кодировка используется, а также знать о возможностях разных кодировок.

Например, кодировка ASCII не может преобразовать в байты кириллицу:

In [32]: hi_unicode = 'привет' In [33]: hi_unicode.encode('ascii') --------------------------------------------------------------------------- UnicodeEncodeError Traceback (most recent call last) ipython-input-33-ec69c9fd2dae> in module>() ----> 1 hi_unicode.encode('ascii') UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128) 

Аналогично, если строка «привет» преобразована в байты, и попробовать преобразовать ее в строку с помощью ascii, тоже получим ошибку:

In [34]: hi_unicode = 'привет' In [35]: hi_bytes = hi_unicode.encode('utf-8') In [36]: hi_bytes.decode('ascii') --------------------------------------------------------------------------- UnicodeDecodeError Traceback (most recent call last) ipython-input-36-aa0ada5e44e9> in module>() ----> 1 hi_bytes.decode('ascii') UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128) 

Еще один вариант ошибки, когда используются разные кодировки для преобразований:

In [37]: de_hi_unicode = 'grüezi' In [38]: utf_16 = de_hi_unicode.encode('utf-16') In [39]: utf_16.decode('utf-8') --------------------------------------------------------------------------- UnicodeDecodeError Traceback (most recent call last) ipython-input-39-4b4c731e69e4> in module>() ----> 1 utf_16.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte 

Наличие ошибок — это хорошо. Они явно говорят, в чем проблема. Хуже, когда получается так:

In [40]: hi_unicode = 'привет' In [41]: hi_bytes = hi_unicode.encode('utf-8') In [42]: hi_bytes Out[42]: b'\xd0\xbf\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82' In [43]: hi_bytes.decode('utf-16') Out[43]: '뿐胑룐닐뗐苑' 

Обработка ошибок#

У методов encode и decode есть режимы обработки ошибок, которые указывают, как реагировать на ошибку преобразования.

Параметр errors в encode#

По умолчанию encode использует режим strict — при возникновении ошибок кодировки генерируется исключение UnicodeError. Примеры такого поведения были выше.

Вместо этого режима можно использовать replace, чтобы заменить символ знаком вопроса:

In [44]: de_hi_unicode = 'grüezi' In [45]: de_hi_unicode.encode('ascii', 'replace') Out[45]: b'gr?ezi' 

Или namereplace, чтобы заменить символ именем:

In [46]: de_hi_unicode = 'grüezi' In [47]: de_hi_unicode.encode('ascii', 'namereplace') Out[47]: b'gr\\Nezi' 

Кроме того, можно полностью игнорировать символы, которые нельзя закодировать:

In [48]: de_hi_unicode = 'grüezi' In [49]: de_hi_unicode.encode('ascii', 'ignore') Out[49]: b'grezi' 

Параметр errors в decode#

В методе decode по умолчанию тоже используется режим strict и генерируется исключение UnicodeDecodeError.

Если изменить режим на ignore, как и в encode, символы будут просто игнорироваться:

In [50]: de_hi_unicode = 'grüezi' In [51]: de_hi_utf8 = de_hi_unicode.encode('utf-8') In [52]: de_hi_utf8 Out[52]: b'gr\xc3\xbcezi' In [53]: de_hi_utf8.decode('ascii', 'ignore') Out[53]: 'grezi' 

Режим replace заменит символы:

In [54]: de_hi_unicode = 'grüezi' In [55]: de_hi_utf8 = de_hi_unicode.encode('utf-8') In [56]: de_hi_utf8.decode('ascii', 'replace') Out[56]: 'gr��ezi' 

Источник

Python Unicode Encode Error

Summary: The UnicodeEncodeError generally occurs while encoding a Unicode string into a certain coding. Only a limited number of Unicode characters are mapped to strings. Thus, any character that is not-represented / mapped will cause the encoding to fail and raise UnicodeEncodeError. To avoid this error use the encode( utf-8 ) and decode( utf-8 ) functions accordingly in your code.

You might be using handling an application code that needs to deal with multilingual data or web content that has plenty of emojis and special symbols. In such situations, you will possibly come across numerous problems relating to Unicode data. But python has well-defined options to deal with Unicode characters and we shall be discussing them in this article.

What is Unicode ?

Unicode is a standard that facilitates character encoding using variable bit encoding. I am sure, you must have heard of ASCII if you are into the world of computer programming. ASCII represents 128 characters while Unicode defines 2 21 characters. Thus, Unicode can be regarded as a superset of ASCII. If you are interested in having an in-depth look at Unicode, please follow this link.
Click on Unicode:- U+1F40D to find out what it represents! (Try it. )

What is a UnicodeEncodeError ?

The best way to grasp any concept is to visualize it with an example. So let us have a look at an example of the UnicodeEncodeError .

u = 'é' print("Integer value for é: ", ord(u)) print("Converting the encoded value of é to Integer Equivalent: ", chr(233)) print("UNICODE Representation of é: ", u.encode('utf-8')) print("ASCII Representation of é: ", u.encode('ascii'))

Integer value for é: 233 Converting the encoded value of é to Integer Equivalent: é UNICODE Representation of é: b'\xc3\xa9' Traceback (most recent call last): File "main.py", line 5, in print("ASCII Representation of é: ",u.encode('ascii')) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

In the above code, when we tried to encode the character é to its Unicode value we got an output but while trying to convert it to the ASCII equivalent we encountered an error. The error occurred because ASCII only allows 7-bit encoding and it cannot represent characters outside the range of [0..128].

You now have an essence of what the UnicodeEncodeError looks like. Before discussing how we can avoid such errors, I feel that there is a dire need to discuss the following concepts:

Encoding and Decoding

The process of converting human-readable data into a specified format, for the secured transmission of data is known as encoding. Decoding is the opposite of encoding that is to convert the encoded information to normal text (human-readable form).

• encode() is an inbuilt method used for encoding. Incase no encoding is specified, UTF-8 is used as default.
• decode() is an inbuilt method used for decoding.

u = 'Πύθωνος' print("UNICODE Representation of é: ", u.encode('utf-8'))

UNICODE Representation of é: b'\xce\xa0\xcf\x8d\xce\xb8\xcf\x89\xce\xbd\xce\xbf\xcf\x82'

The following diagram should make things a little easier:

Codepoint

Unicode maps the codepoint to their respective characters. So, what do we mean by a codepoint?

• Codepoints are numerical values or integers used to represent a character.
• The Unicode code point for é is U+00E9 which is integer 233. When you encode a character and print it, you will generally get its hexadecimal representation as an output instead of its binary equivalent (as seen in the examples above).
• The byte sequence of a code point is different in different encoding schemes. For eg: the byte sequence for é in UTF-8 is \xc3\xa9 while in UTF-16 is \xff\xfe\xe9\x00.

Please have a look at the following program to get a better grip on this concept:

u = 'é' print("INTEGER value for é: ", ord(u)) print("ENCODED Representation of é in UTF-8: ", u.encode('utf-8')) print("ENCODED Representation of é in UTF-16: ", u.encode('utf-16'))

INTEGER value for é: 233 ENCODED Representation of é in UTF-8: b'\xc3\xa9' ENCODED Representation of é in UTF-16: b'\xff\xfe\xe9\x00'

Now that we have an overview of Unicode and UnicodeEncodeError , let us discuss how we can deal with the error and avoid it in our program.

➥ Problem: Given a string/text to be written in a text File; how to avoid the UnicodeEncodeError and write given text in the text file.

f = open('demo.txt', 'w') f.write('να έχεις μια όμορφη μέρα') f.close()

Traceback (most recent call last): File «uniError.py», line 2, in f.write(‘να έχεις μια όμορφη μέρα’) File «C:\Users\Shubham-PC\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py», line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: ‘charmap’ codec can’t encode characters in position 0-1: character maps to

✨ Solution 1: Encode String Before Writing To File And Decode While Reading

You cannot write Unicode to a file directly. This will raise an UnicodeEncodeError . To avoid this you must encode the Unicode string using the encode() function and then write it to the file as shown in the program below:

text = u'να έχεις μια όμορφη μέρα' # write in binary mode to avoid TypeError f = open('demo.txt', 'wb') f.write(text.encode('utf8')) f.close() f = open('demo.txt', 'rb') print(f.read().decode('utf8'))

✨ Solution 2: Open File In utf-8

If you are using Python 3 or higher, all you need to do is open the file in utf-8 , as Unicode string handling is already standardized in Python 3.

text = 'να έχεις μια όμορφη μέρα' f = open('demo2.txt', 'w', encoding="utf-8") f.write(text) f.close()

✨ Solution 3: Using The Codecs Module

Another approach to deal with the UnicodeEncodeError is using the codecs module.

Let us have a look at the following code to understand how we can use the codecs module:

import codecs f = codecs.open("demo3.txt", "w", encoding='utf-8') f.write("να έχεις μια όμορφη μέρα") f.close()

✨ Solution 4: Using Python’s unicodecsv Module

If you are dealing with Unicode data and using a csv file for managing your data, then the unicodecsv module can be really helpful. It is an extended version of Python 2’s csv module and helps the user to handle Unicode data without any hassle.

Since the unicodecsv module is not a part of Python’s standard library, you have to install it before using it. Use the following command to install this module:

Let us have a look at the following example to get a better grip on the unicodecsv module:

import unicodecsv as csv with open('example.csv', 'wb') as f: writer = csv.writer(f, encoding='utf-8') writer.writerow(('English', 'Japanese')) writer.writerow((u'Hello', u'こんにちは'))

Conclusion

In this article, we discussed some of the important concepts regarding Unicode character and then went on to learn about the UnicodeEncodeError and finally discussed the methods that we can use to avoid it. I hope by the end of this article you can handle Unicode characters in your python code with ease.

Please subscribe and stay tuned for more interesting articles!

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

🚀 If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

I am a professional Python Blogger and Content creator. I have published numerous articles and created courses over a period of time. Presently I am working as a full-time freelancer and I have experience in domains like Python, AWS, DevOps, and Networking.

Be on the Right Side of Change 🚀

• The world is changing exponentially. Disruptive technologies such as AI, crypto, and automation eliminate entire industries. 🤖
• Do you feel uncertain and afraid of being replaced by machines, leaving you without money, purpose, or value? Fear not! There a way to not merely survive but thrive in this new world!
• Finxter is here to help you stay ahead of the curve, so you can keep winning as paradigms shift.

Learning Resources 🧑‍💻

⭐ Boost your skills. Join our free email academy with daily emails teaching exponential with 1000+ tutorials on AI, data science, Python, freelancing, and Blockchain development!

Join the Finxter Academy and unlock access to premium courses 👑 to certify your skills in exponential technologies and programming.

New Finxter Tutorials:

Finxter Categories:

Источник