Last Updated: 2021-05-16
After trying to read a CSV file and getting an encoding error I went down a rabbit hole about how to analyze and fix these issues.
Why? Because those bytes contain useful hints you can analyze. Often there is an optional BOM (Byte order mark), magic numbers at the start of a text stream that signal things to a program regarding the text.
$ hexdump -n10 Download.CSV 0000000 ef bb bf 22 44 61 74 65 22 2c
These bytes "ef bb bf" are the UTF-8 BOM.
$ file -I Download.CSV Download.CSV: text/plain; charset=utf-8
According to Wikipedia, UTF-8 is represented with the hex byte sequence:
Is this the same as what I saw? Yes it is, except for the 0000000 at the start in the
CSV. However, this
000000 comes from
hexdump - it's actually just offsets.
# It shows when reading just one character $ hexdump -n1 Download.CSV 0000000 ef
$ hexdump -n33 Download.CSV # hexdump shows 16 bytes per line # the left-hand column are offsets 0000000 ef bb bf 22 44 61 74 65 22 2c 22 54 69 6d 65 22 0000010 2c 22 54 69 6d 65 5a 6f 6e 65 22 2c 22 4e 61 6d 0000020 65
The bit with 000000s on the left is the starting offset of the file. Then there are 16 8bit values per line.
The problem, it turned out, was that the Ruby CSV file opener does not handle BOMs automatically. (In general, BOMs break compatibility with code that expects plain ASCII)
Here is how to write a
bom in Ruby:
There is also a new IO method for dealing with it:
IO#set_encoding_by_bom → encoding or nil
"Checks if IO starts with a BOM, and then consumes it and sets the external encoding. Returns the result encoding if found, or nil. If IO is not binmode or its encoding has been set already, an exception will be raised."
I ran into further issues due to it being a StringIO and not a true file:
contents = stringio.sub!("\xEF\xBB\xBF", ''); => Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
Indeed, this shows that with string operations involving two strings, the encodings of both ought to match
"\xEF\xBB\xBF".encoding => UTF8 stringio.read.encoding => Encoding:ASCII-8BIT> # no match!!
Here's a fix:
stringio.set_encoding("utf-8") contents = stringio.sub!("\xEF\xBB\xBF", ''); => works