Mojibake

Mojibake is garbled text like ä instead of ä, é instead of é or ’ instead of an apostrophe. The word is Japanese for “character transformation”. It happens when text is saved in one character encoding and read in another. Nothing is encrypted or damaged: the bytes are all still there, they are just read the wrong way. That is why mojibake can usually be repaired.

Characters are stored as bytes

A computer stores every character as one or more bytes. Which bytes stand for which character is decided by the encoding. Today the web, most databases and most programs use UTF-8. UTF-8 needs one byte for the 128 ASCII characters (A–Z, digits, punctuation) and two to four bytes for everything else:

  • A is one byte: 41
  • ä is two bytes: C3 A4
  • € is three bytes: E2 82 AC
  • 😀 is four bytes: F0 9F 98 80

Older encodings such as Windows-1252, ISO-8859-1 (Latin-1) or Mac OS Roman use exactly one byte per character and know only 256 characters. In Windows-1252 the byte E4 is ä, the byte C3 is à and the byte A4 is ¤. You can see the bytes of any text with the hex tool.

Why ä turns into ä

When a program reads UTF-8 bytes but believes they are Windows-1252, it turns every byte into a character of its own. The two bytes of ä become two characters:

  • ä → bytes C3 A4 → read as Windows-1252: à + ¤ = ä
  • é → C3 A9 → é
  • € → E2 82 AC → €
  • ’ → E2 80 99 → ’

The pattern gives the mistake away: German, French or Spanish text full of à followed by another symbol is almost always UTF-8 read as Windows-1252 or Latin-1. A stray  in front of °, © or a space is the same mistake.

Where it happens

  • CSV files in Excel. Excel opens a CSV without a byte order mark (BOM) as Windows-1252. The file itself is correct UTF-8.
  • Databases. A MySQL connection or column set to latin1 while the application sends UTF-8. The data looks fine in the application and garbled in every export.
  • Emails and web pages that send UTF-8 but declare another character set, or none.
  • Code that converts text that is already UTF-8 again, for example PHP’s utf8_encode() on a UTF-8 string.

If garbled text is saved as UTF-8 and read wrongly a second time, it gets worse: ä becomes ä and then ä. Each round adds another layer.

How to repair it

The repair runs the same mistake backwards. Turn each character back into its Windows-1252 byte, then read the bytes as UTF-8: Ã and ¤ become C3 A4, which is ä. Garbled twice means doing it twice. Good tools do it only where the bytes form valid UTF-8, so correct text in between stays as it is.

Repair is not possible when bytes were lost on the way. If a program replaced a character it could not convert with ? or the replacement character �, the original is gone. The same happens to emoji in MySQL’s old utf8 character set, which only stores three bytes per character: they turn into ????.

Fix garbled text or a CSV file in your browser →

How to prevent it

  • Use UTF-8 everywhere: files, databases, connections, web pages and emails. In MySQL use utf8mb4, not utf8.
  • Declare the encoding: <meta charset="utf-8"> in HTML, charset=utf-8 in the Content-Type header, SET NAMES utf8mb4 for database connections.
  • For Excel, save CSV files as UTF-8 with BOM, or import them via Data › From Text/CSV with “65001: Unicode (UTF-8)”.

Searches the list of pages on this site. What you type stays in your browser.