Read hex from file and convert to decimal
I made a small script to read text file that contain hexadecimal and convert to another file in decimal by using command-line as a filename input.
#this is content from hex.txt 0x5B03FA01 0x42018360 0x9FF943B3 #this is content from hex_1.txt 1526987265 1107395424 2683913139 #this was copied from Learn Python the Hard way exercise. from sys import argv from os.path import exists #this is the main function to convert hexadecimal to decimal def hex2dec (hex): result_dec = int(hex, 0) return result_dec #this to accept parameter from command-line script, from_file = argv #this to make sure the output generate as a new file with _1 to_file = from_file[:-4] + "_1.txt" out_file = open(to_file, 'w').close() #this took me a few hour to figure out a bug. Thanks to stackoverflow with open(from_file) as in_file: lines = in_file.read().splitlines() for i in lines: converted = str(hex2dec (i)) out_file = open(to_file, 'a+') out_file.write(converted + "\n") out_file.close() #this to print the result for reference on-screen. print "converted =" + str(converted) print "done." Even though this is done but I feel there are plenty area I could improve to this script e.g. bugs handling in hex2dec function. However, may I ask you what would you do to enhance this further? and if you could suggest, could anyone suggest me which topic or area I can study further to improve this code or any other about Python?
\$\begingroup\$ or (more efficiently, since it doesn’t invoke the printf command once per line): awk ‘< printf "%d\n", $1 >‘ < hex.txt \$\endgroup\$
3 Answers 3
Python 3
The script currently does not work in Python 3. For example, print «converted lang-py prettyprint-override»> def hex2dec(hex): «»»Convert a hexadecimal string to a decimal number»»» result_dec = int(hex, 0) return result_dec
Some other things about the above function:
- Rather than hex2dec , perhaps use hex_to_decimal . In this case it’s not too big of a deal though.
- You’re storing the result in a variable and then returning the variable without doing anything else to it. This means you could also return result immediately: return int(hex, 0)
- Since it’s using a built in function without any other operation, is the function necessary?
Running it!
As said above, using Python 3 failed to run it. Changing the print statements to functions fixed this issue. Python 2 also supports the print() function, so it should have no issue running on Python 2 as well.
The program ran as expected with a correct file, and produced an output file with the converted values.
Running it with an invalid file, or no arguments at all, crashes the program with either a FileNotFoundError or ValueError respectively. If a file was given which exists but has an invalid hexadecimal value in it (for instance a random string), the program also crashes:
Traceback (most recent call last): File "hex2dec.py", line 22, in converted = str(hex2dec (i)) File "hex2dec.py", line 7, in hex2dec result_dec = int(hex, 0) ValueError: invalid literal for int() with base 0: 'random word' The above errors tells us we need some error checking.
The first error to solve would be the ValueError of not giving a filename as argument when running the program. This can be solved by using a CLI module such as argparse , click , or perhaps another one you like. I personally use argparse, since it’s pretty simple and it’s part of the standard library.
import argparse def get_cli_parser(): parser = argparse.ArgumentParser(description='Describe what your program does') parser.add_argument('input_file', type=str, help='The file to read the hexadecimal values from' return parser if __name__ == '__main__': parser = get_cli_parser() args = vars(parser.parse_args()) At this point you can access the command line arguments args as a dictionary.
The if __name__ == ‘__main__’: part of the code is only applicable if it’s run as a script. This means that if the file is imported as a module, this will not run. See also this answer for more info.
When we run the above code without any arguments, we get the following:
usage: hex2dec2.py [-h] input_file hex2dec2.py: error: the following arguments are required: input_file Next up is the FileNotFoundError . This one is quite easy to solve: before opening the file, check if it exists using the already imported os module:
if not os.path.exists(args['input_file']): print('Input file does not exist, please try again') return The last ValueError can be solved in a number of ways. One would be to use regex on the current line of the file to find any hexadecimal values. Assuming only the hexadecimal values should be on a line, it becomes a lot simpler though. We could just check if the line starts with 0x (e.g. a hexadecimal value).
Reading the file should also be done line by line as an iterator, to prevent huge files from crashing your program by too high memory consumption. Right now the program reads all the text in the file into memory. This can be done as described in the following question :
with open(input_file) as input_file: for line in input_file: # rest of code here. Placing this in a function of its own also prevents you from opening the file continuously.
The final product
import argparse import os def write_and_print_result(args, result): """Write every result to the output file and print each converted value """ out_file = args['input_file'][:-4] + "_1.txt" with open(out_file, 'a+') as output: for converted in result: output.write(converted + "\n") print('Converted: <>'.format(converted)) def read_and_convert_file(args): """Read the given input file and convert each hexadecimal value to decimal value """ result = [] with open(args['input_file']) as input_file: for line in input_file: if (line.startswith('0x')): result.append(str(int(line, 0))) write_and_print_result(args, result) def get_cli_parser(): parser = argparse.ArgumentParser(description="Describe program here") parser.add_argument('input_file', type=str, help='The file to read the hexadecimal values from') return parser def cli_runner(): parser = get_cli_parser() args = vars(parser.parse_args()) if not os.path.exists(args['input_file']): print('Input file does not exist, please try again.') return read_and_convert_file(args) if __name__ == '__main__': cli_runner() As you can see, most of it is either error checking or command line arguments. The actual conversion is only one line of code.
What’s next
- If the input file is anything but *.txt , the output file might have a weird name. For example, a file called hex would simply become _1.txt , which isn’t very descriptive. To fix this, as pointed out by @Mathias Ettinger, you could use os.path.splitext . It is used as such:
filename, extension = os.path.splitext(args['input_file']) outfile = filename + "_1.txt" Python print hex file
Today we will share a quick python script for creating a hex dump of any file in Python. It can be pretty useful sometimes when we want to drill down into the low level data of a file — perhaps when we would like to analyse the file header of a piece of malware or understand the details of a specific file format. Of course there are Hex Editors and Unix tools to do so but sometimes a simple Python script can be nice due to it’s OS independent nature.
For our script, we will output not only the hex data but also the memory address and printable ascii characters as is the standard method. That is, memory offset on the left, hex data in the middle and printable ascii characters on the right. A non-printable byte will be replaced by a dot symbol in the ascii output.
The script is also given below.
#!/usr/bin/python
#
# HEX DUMP PYTHON SCRIPT - RUBY DEVICES 2017
#
# Usage:
# hexdump.py
#
import sys
import os.path
def check_file_provided():
# This method ensures a valid file was provided to the invoked script ##
if (len(sys.argv) < 2):
print ""
print "Error - No file was provided"
print ""
print "Correct Usage:"
print "python hexdump.py "
print ""
sys.exit(0)
if not os.path.isfile(sys.argv[1]):
print ""
print "Error - The file provided does not exist"
print ""
sys.exit(0)
def read_bytes(filename, chunksize=8192):
# This method returns the bytes of a provided file ##
try:
with open(filename, "rb") as f:
while True:
chunk = f.read(chunksize)
if chunk:
for b in chunk:
yield b
else:
break
except IOError:
print ""
print "Error - The file provided does not exist"
print ""
sys.exit(0)
def is_character_printable(s):
## This method returns true if a byte is a printable ascii character ##
return all((ord(c) < 127) and (ord(c) >= 32) for c in s)
def print_headers():
## This method prints the headers at the top of our hex dump ##
print ""
print "#### HEX DUMP PYTHON SCRIPT - RUBY DEVICES 2017 ####"
print ""
print "Offset 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F"
print ""
def validate_byte_as_printable(byte):
## Check if byte is a printable ascii character. If not replace with a '.' character ##
if is_character_printable(byte):
return byte
else:
return '.'
## main ##
check_file_provided()
memory_address = 0
ascii_string = ""
print_headers()
## Loop through the given file while printing the address, hex and ascii output ##
for byte in read_bytes(sys.argv[1]):
ascii_string = ascii_string + validate_byte_as_printable(byte)
if memory_address%16 == 0:
print(format(memory_address, '06X')),
print(byte.encode('hex')),
elif memory_address%16 == 15:
print(byte.encode('hex')),
print ascii_string
ascii_string = ""
else:
print(byte.encode('hex')),
memory_address = memory_address + 1
An example of invoking the script is given below. We run the dump on a Microsoft LNK file. It’s a good idea to pipe the output into «more» when running the script for a much more readable experience.
[user]~$ python hexdump.py GoogleChrome.lnk | more
#### HEX DUMP PYTHON SCRIPT - RUBY DEVICES 2017 ####
Offset 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
000000 4c 00 00 00 01 14 02 00 00 00 00 00 c0 00 00 00 L.
000010 00 00 00 46 df 40 00 00 20 00 00 00 20 55 e3 8e . F.@.. . U..
000020 f6 61 d1 01 28 2a d8 b3 a4 17 d3 01 78 e9 a2 77 .a..(*. x..w
000030 6a 12 d3 01 58 45 11 00 00 00 00 00 01 00 00 00 j. XE.
000040 00 00 00 00 00 00 00 00 00 00 00 00 0f 02 14 00 .
000050 1f 50 e0 4f d0 20 ea 3a 69 10 a2 d8 08 00 2b 30 .P.O. .:i. +0
000060 30 9d 19 00 2f 43 3a 5c 00 00 00 00 00 00 00 00 0. /C:\.
000070 00 00 00 00 00 00 00 00 00 00 00 88 00 31 00 00 . 1..
000080 00 00 00 18 4b 57 5c 11 00 50 52 4f 47 52 41 7e . KW\..PROGRA~
000090 31 00 00 70 00 08 00 04 00 ef be ee 3a a3 14 18 1..p.
0000A0 4b 57 5c 2a 00 00 00 3c 00 00 00 00 00 01 00 00 KW\*. 0000B0 00 00 00 00 00 00 00 46 00 00 00 00 00 50 00 72 . F. P.r
0000C0 00 6f 00 67 00 72 00 61 00 6d 00 20 00 46 00 69 .o.g.r.a.m. .F.i
0000D0 00 6c 00 65 00 73 00 00 00 40 00 73 00 68 00 65 .l.e.s. @.s.h.e
-- More -- That’s all for today! Have fun with it 🙂