Python CSV Reader Encoding
Pandas and csv libraries are popular names when handling CSV files. Python comes pre-installed with csv, but for pandas, you must install it before using it.
When it comes to data manipulation and analysis, pandas reign supreme because it possesses many functions and attributes that can perform such tasks. This article focuses on how these libraries implement encoding when reading CSV files.
Reading CSV data
When reading CSV files (using pandas or csv), the following processes are conducted: decoding, parsing, data conversions (optional), and data fetching.
Decoding: To read a file, the library must first convert a series of bytes into characters from a particular charset. Sometimes, this section is challenging since the library might not be aware of the file’s encoding. The library may raise an exception at this moment. For instance, if it cannot recognize the encoding or runs into byte sequences that it cannot decode, it may produce an error message.
With Python 3 and local systems getting better at encoding, the encoding process mostly happens seamlessly without us having to explicitly define the encoding system when loading CSV files. However, encoding is still a vital issue when we want to filter out some unwanted characters in our CSV file or some cases, get data in the needed view.
We will save the following simple data into a UTF-8 encoded CSV file named “streets10.csv” and use it for our examples considering two encodings – ASCII and UTF-8. ASCII encoding is the most common character encoding format for English text, whereas UTF-8 contains much more characters.
| Name | Streets |
| Bob | NazarethkirtchStraße |
| Alex | St Äbràhâm |
Table 1: Example data set that we will use in our example. It is saved with UTF-8 encoding as “streets10.csv”.
In the above data, the following characters are none ASCII characters: ß, Ä, à, and â. Any attempt to read the CSV file with ASCII encoding will result in encoding errors because of these characters.
[Example code]-How to read bytes object from csv?
If your input file really contains strings with Python syntax b prefixes on them, one way to workaround it (even though it’s not really a valid format for csv data to contain) would be to use Python’s ast.literal_eval() function as @Ry suggested — although I would use it in a slightly different manner, as shown below.
This will provide a safe way to parse strings in the file which are prefixed with a b indicating they are byte-strings. The rest will be passed through unchanged.
Note that this doesn’t require reading the entire CSV file into memory.
import ast import csv def _parse_bytes(field): """Convert string represented in Python byte-string literal b'' syntax into a decoded character string - otherwise return it unchanged. """ result = field try: result = ast.literal_eval(field) finally: return result.decode() if isinstance(result, bytes) else result def my_csv_reader(filename, /, **kwargs): with open(filename, 'r', newline='') as file: for row in csv.reader(file, **kwargs): yield [_parse_bytes(field) for field in row] reader = my_csv_reader('bytes_data.csv', delimiter=',') for row in reader: print(row) 
martineau 114934
You can use ast.literal_eval to convert the incorrect fields back to bytes safely:
import ast def _parse_bytes(bytes_repr): result = ast.literal_eval(bytes_repr) if not isinstance(result, bytes): raise ValueError("Malformed bytes repr") return result 
Ry- 211017
The easiest way is as below. Try it out.
import csv from io import StringIO byte_content = b"iam byte content" content = byte_content.decode() file = StringIO(content) csv_data = csv.reader(file, delimiter=",") 
Rakeshkumar Taninki 142
Related Query
- How to read bytes object from csv?
- Python3 How to make a bytes object from a list of integers
- How to remove a range of bytes from a bytes object in python?
- How to get data from csv into a python object
- In python 3, how can I put individual bytes from a bytes object into a list without them being converted to integers?
- How to read bytes from file
- How to read data from CSV into nested key-value pairs for future retrieval?
- How to add checkbuttons to every row of a table read from csv in tkinter?
- How to read zipfile from bytes in Python 3.x?
- How to convert a fetch return object to csv from a fetch API in C3.ai COVID-19 datalake?
- How can I read a csv file using panda dataframe from GPU?
- python how to read bytes type data from file and convert it to utf-8?
- How to read CSV File dwnloaded from NSE
- How can i write a byte object file from integers of different length of bytes
- How to read a csv file and convert commas from the numbers to dot?
- How to read a certain column from csv file?
- how to read csv file from dropbox as a dictionary (as read from csv.DictReader() )?
- How to read numbers from file in Python?
- How to read bytes as stream in python 3
- Read .csv file from URL into Python 3.x — _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
- How to construct a TarFile object in memory from byte buffer in Python 3?
- How to differentiate a file like object from a file path like object
- How can i read a PDF file from inline raw_bytes (not from file)?
- How can I convert bytes object to decimal or binary representation in python?
- How to read input() from a text file in Python
- How to Read .json file in python code from google cloud storage bucket
- How do I print only the first 10 lines from a csv file using Python?
- Python: How to read and load an excel file from AWS S3?
- How to convert the arff object loaded from a .arff file into a dataframe format?
- How to read simple text from a PDF file with Python?
- Python bytes object from generator
- How to import object from builtins affecting just one class?
- How to read first 1000 entries in a csv file
- How to shift bits in a 2-5 byte long bytes object in python?
- how to read a csv into a dictionary in python?
- How can I select a random object from a class?
- How to read image using OpenCV got from S3 using Python 3?
- How to read the file contents from a file?
- How to read only a specific range of lines out of a csv file with python?
- How do I create a Python bytes object in the C API
- Read an video file object from S3 and use it for further processing through Opencv
- Read contents of .tar.gz file from website into a python 3.x object
- How to plot a graph from csv in python
- How to read n files from directory in Python?
- How to remove an object by value from a set
- How to download zip file and parse csv file from it in python
- How to read config file from module and from main file?
- How to clear objects from the object store in ray?
- How to convert the bytes to the manual output when i read a PE file
- How to update one element of a csv with information from another csv?
More Query from same tag
- Is there an equivalent of PyMongo for Python 3.2?
- I am trying to use python and selenium to create a driver scraper, I need to grab particular data from the web page and put it in csv row and column
- Check for valid file extension
- How to make list active in Kivymd list
- Set tkinter filedialog to open only executable files
- How to reorder coordinate values using the euclidian distance in Python?
- python override getLogger()
- Why is the staticmethod decorator necessary?
- Get a list of string as input from user in loop python
- Tornado difference between run in executor and defining async methods
- atexit runs after objects are already freed?
- is there a way to send multiple images to an API at the same time fastapi
- KeyError: 0 — loc
- How can I quickly change pixels in a image from a color dictionary?
- Confusion regarding UTF8 substring length
- PyQt Signals and Slots: «new style» emit?
- Python: Reverse words in each paragraph of a file without changing the order of paragraph?
- Mocking Popen to return different results depending on call_args
- Can’t install matplotlib in a Python 3.10 venv
- Solve google foobar staircase problem with python
- python 3 calculator with tkinter
- How to download folder by checking item inside it from remote server to local dir using Python?
- Why does my script that uses twilio run perfectly locally, but throw an error on Python Anywhere?
- Why won’t this ctypes code work with Python 3.3 but will work with Python 2.7?
- How to print a collection of characters next to each other
- Cannot install python3.6 on Debian 4.9 VM
- Python — Waiting untill a number of operations are done in concurrent.futures
- How to type hint an instance-level function (i.e. not a method)?
- A module to profile peak memory usage of Python code
- How to wrap a third party decorator with my own in python
- X and Y Ticks on a 4×4 multiplot using matplotlib in Python
- How can I using yield list element infinitely
- How can i mock configparser.ConfigParser read method in python
- Write a basic HTTP Server using socket API
- Swap slices of indexes using a function