Python utf invalid continuation byte

How to fix unicodedecodeerror, invalid continuation byte in Python?

The «UnicodeDecodeError: invalid continuation byte» error in Python is usually raised when a string of text being processed is not properly encoded as Unicode. This error can occur when reading data from a file or from a database, or when processing data from an external source. To resolve this error, it’s important to understand how the data is being encoded and to make sure that it’s properly decoded before being processed in Python.

Method 1: Use the correct encoding

When you encounter the UnicodeDecodeError with the message «invalid continuation byte», it means that Python is trying to decode a byte sequence that is not valid for the specified encoding. This error can be fixed by using the correct encoding.

Here are the steps to fix this error using the correct encoding:

Step 1: Determine the Encoding

The first step is to determine the encoding of the byte sequence. You can use the chardet library to automatically detect the encoding:

import chardet with open('file.txt', 'rb') as f: data = f.read() encoding = chardet.detect(data)['encoding']

Step 2: Decode the Byte Sequence

Once you have determined the encoding, you can decode the byte sequence using the correct encoding:

with open('file.txt', 'r', encoding=encoding) as f: data = f.read()

Step 3: Handle Errors

If the byte sequence contains invalid characters that cannot be decoded using the specified encoding, you can handle the errors using the errors parameter:

with open('file.txt', 'r', encoding=encoding, errors='replace') as f: data = f.read()

The errors parameter can take the following values:

• ‘strict’ : raise a UnicodeDecodeError if the byte sequence contains invalid characters
• ‘ignore’ : ignore the invalid characters and continue decoding
• ‘replace’ : replace the invalid characters with the Unicode replacement character U+FFFD

Step 4: Encode the Unicode String

If you need to encode the Unicode string back to bytes, you can use the encode() method:

data = 'Hello, world!' encoded_data = data.encode(encoding)

Here, encoding is the encoding used to decode the byte sequence.

That’s it! By following these steps, you should be able to fix the UnicodeDecodeError with the message «invalid continuation byte» in Python by using the correct encoding.

Method 2: Check the data for invalid characters

If you are working with text data in Python, you may encounter the UnicodeDecodeError: invalid continuation byte error. This error occurs when you try to decode a string that contains invalid characters or bytes. In this tutorial, we will show you how to fix this error by checking the data for invalid characters.

Step 1: Read the File in Binary Mode

The first step is to read the file in binary mode using the rb mode instead of the r mode. This will ensure that the file is read as bytes and not as text.

with open('file.txt', 'rb') as file: data = file.read()

Step 2: Decode the Data

The next step is to decode the data using the appropriate encoding. In this example, we will use the utf-8 encoding.

try: text = data.decode('utf-8') except UnicodeDecodeError: pass

Step 3: Check for Invalid Characters

Now that we have decoded the data, we can check for invalid characters using the isprintable() method. This method returns True if all the characters in the string are printable, otherwise it returns False .

invalid_chars = [] for char in text: if not char.isprintable(): invalid_chars.append(char)

Step 4: Replace Invalid Characters

Finally, we can replace the invalid characters with a valid character using the replace() method.

for char in invalid_chars: text = text.replace(char, '')

Full Example

with open('file.txt', 'rb') as file: data = file.read() try: text = data.decode('utf-8') except UnicodeDecodeError: pass invalid_chars = [] for char in text: if not char.isprintable(): invalid_chars.append(char) for char in invalid_chars: text = text.replace(char, '')

This code will read the file in binary mode, decode the data using the utf-8 encoding, check for invalid characters, and replace them with a valid character. This should fix the UnicodeDecodeError: invalid continuation byte error.

Method 3: Use a try-except block to handle the error

To fix the UnicodeDecodeError: ‘utf-8’ codec can’t decode byte. error in Python, you can use a try-except block to handle the error. Here’s an example code snippet:

try: with open('file.txt', 'r', encoding='utf-8') as f: text = f.read() except UnicodeDecodeError: with open('file.txt', 'r', encoding='ISO-8859-1') as f: text = f.read()

In this code, we try to open the file with UTF-8 encoding. If there’s a UnicodeDecodeError , we catch it with the except block and try to open the file again with ISO-8859-1 encoding.

You can also wrap the file reading code in a function to make it more reusable:

def read_file(filename): try: with open(filename, 'r', encoding='utf-8') as f: text = f.read() except UnicodeDecodeError: with open(filename, 'r', encoding='ISO-8859-1') as f: text = f.read() return text

This function takes a filename as an argument and returns the file’s contents. If there’s a UnicodeDecodeError , it tries to open the file again with ISO-8859-1 encoding.

In summary, using a try-except block to handle the UnicodeDecodeError in Python involves trying to open the file with UTF-8 encoding, catching the error if it occurs, and trying to open the file again with another encoding (such as ISO-8859-1). This approach allows you to handle the error gracefully and continue with your program’s execution.

Method 4: Force decode using the «ignore» option

To fix the UnicodeDecodeError with the invalid continuation byte error in Python, you can force decode the string using the «ignore» option. Here’s how you can do it in Python:

with open('filename.txt', 'rb') as f: data = f.read() try: decoded_data = data.decode('utf-8', 'ignore') except UnicodeDecodeError as e: print(f"Error: e>") with open('new_filename.txt', 'w') as f: f.write(decoded_data)

In this example, we first read the file in binary mode using rb . This is necessary because the file contains invalid bytes that can’t be decoded directly. Then, we use the decode() method to decode the data using the «ignore» option. This option tells Python to ignore any invalid bytes and continue decoding the rest of the string. If there are still invalid bytes left after decoding, they will be replaced with the «replacement character» (U+FFFD). Finally, we write the decoded data to a new file in text mode using w .

Note that this method may result in some data loss, as any invalid bytes will be ignored or replaced with the «replacement character». If you want to preserve all the data in the file, you may need to use a different method, such as manually fixing the invalid bytes or using a different encoding.

Источник

How to fix UnicodeDecodeError: invalid continuation byte

One error that you might encounter when working with Python is:

This error occurs when you try to decode a bytes object with an encoding that doesn’t support that character.

This tutorial shows an example that causes this error and how to fix it.

How to reproduce this error

Suppose you have a bytes object in your Python code as follows:

Next, you want to decode the bytes character using the utf-8 encoding like this:

You get an error because the character \xe1 in the bytes object is the á character encoded using latin-1 encoding.

How to fix this error

To resolve this error, you need to change the encoding used in the decode() method to latin-1 as follows:

Note that this time the decode() method runs without any error.

You can also get this error when running other methods such as pandas read_csv() method.

You need to specify the encoding used by the method as follows:

The same also works when you use the open() function to work with files:

 If you only want to read the files without modifying the content, you can use the open() function in rb read binary mode.

Here’s an example when you parse an HTML file using Beautiful Soup:

 When you decode the bytes object, you need to use the encoding that supports the object.

If you don’t want to encode the object when opening a file, you need to specify the open mode as rb or wb to read and write in binary mode.

I hope this tutorial helps. See you in other tutorials! 👍

Take your skills to the next level ⚡️

I’m sending out an occasional email with the latest tutorials on programming, web development, and statistics. Drop your email in the box below and I’ll send new stuff straight into your inbox!

About

Hello! This website is dedicated to help you learn tech and data science skills with its step-by-step, beginner-friendly tutorials.
Learn statistics, JavaScript and other programming languages using clear examples written for people.

Search

Type the keyword below and hit enter

Tags

Click to see all tutorials tagged with:

Источник

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte in position: invalid continuation byte – How to fix this error?

Usually, there should be no problem working with Latin characters. Except when interacting with special characters, we can see the “UnicodeDecodeError: ‘utf-8’ codec can’t decode byte in position: invalid continuation byte”.

Why does the “UnicodeDecodeError: ‘utf-8’ codec can’t decode byte in position: invalid continuation byte” appear? And how to solve it?

Encode and decode 2 different character sets

The error appears when we encode with one character set and try to use a different character set when we want to decode an object. See the example for a better understanding.

encoding = 'LearnShäreIT'.encode('latin-1') decoding = encoding.decode('utf-8') print(decoding) # UnicodeDecodeError

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 7

To solve this error, you must use the character set that was previously used for encoding when you decode the string you want, like the code sample below.

encoding = 'LearnShäreIT'.encode('utf-8') # Using the same character set decoding = encoding.decode('utf-8') print(decoding)

The charset is inconsistent when saving files and reading files

When we create and save a CSV file, we choose the UTF-16 BE charset, as shown below.

But when reading the file with pandas.read_csv(), we use the default character set of read_csv() which is utf-8. See the code below for a better understanding.

import pandas as pd # Using encoding = 'utf-8' but charset of data.csv = 'utf-16' data = pd.read_csv('data.csv') print(data)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0

We have to set the encoding=’utf-16′ for consistency between encoding and decoding. Like this:

import pandas as pd # Using encoding='utf-16' data = pd.read_csv('data.csv', encoding='utf-16') print(data)

 Name Website 0 LearnShareIT learnshareit.com 1 Facebook facebook.com 2 Google google.com 3 Udemy udemy.com

Using detect() function in the chardet package

You can use chardet to detect the character encoding of a file. This library is handy when working with a large pile of text. But it can also be used when working with downloaded data you don’t know its charset.

The detect() function detects what charset a non-Unicode string is using. It returns a dictionary containing the automatically detected charset and confidence level.

Before using the detect() function, we need to install the chardet with the following command line:

Then we will import the chardet at the top of the python file. Next, we pass the data into the detect() function to detect its charset. After getting the charset, pass it to the read_csv() . Like this:

import chardet import pandas as pd # Detect character encoding of data.csv enc = chardet.detect(open('data.csv', 'rb').read()) print(enc['encoding']) # UTF-16 # Use pandas to read data.csv data = pd.read_csv('data.csv', encoding=enc['encoding']) print(data)

UTF-16 Name Website 0 LearnShareIT learnshareit.com 1 Facebook facebook.com 2 Google google.com 3 Udemy udemy.com

Change character encoding manually

This way is very simple. Just open the file you need to read with notepad++. On the menu bar, select Encoding -> Convert to UTF-8. Like this:

import pandas as pd # Using pandas to read data.csv with charset = UTF-8 data = pd.read_csv('data.csv') print(data)

 Name Website 0 LearnShareIT learnshareit.com 1 Facebook facebook.com 2 Google google.com 3 Udemy udemy.com

Summary

Basically, the error “UnicodeDecodeError: ‘utf-8’ codec can’t decode byte in position: invalid continuation byte” comes from the inconsistency between the encoding and decoding processes. As long as you make sure to use a character set for encoding and decoding (such as UTF-8), you won’t get this error again.

Maybe you are interested:

Hi, I’m Cora Lopez. I have a passion for teaching programming languages such as Python, Java, Php, Javascript … I’m creating the free python course online. I hope this helps you in your learning journey.

Name of the university: HCMUE
Major: IT
Programming Languages: HTML/CSS/Javascript, PHP/sql/laravel, Python, Java

Источник