How Mojibake Works and How to Fix It: Character Encodings, UTF-8 vs Shift_JIS

You open an email and see a row of strange symbols, or you open a downloaded CSV in Excel and every Japanese character is broken. This "mojibake" (garbled text) is a trouble almost everyone hits. It looks mysterious, but the cause is simple: the rule used to save the text and the rule used to read it do not match. Once you understand the mechanism, you can fix it calmly and even prevent it. This article walks from encoding basics to concrete fixes.

What Mojibake Is

Mojibake is a phenomenon where the characters that should appear are shown as completely different symbols or unreadable strings. The appearance varies, but the root cause is the same.

Computers do not handle characters directly. Internally, every character is stored as a number. The character "a," for instance, is assigned a specific number and stored as that number; when reading, the number is turned back into a character to display on screen. A character encoding is the rule that defines which number corresponds to which character. If different rules are used for saving and reading, the same number is interpreted as a different character, and mojibake results.

Encoding Basics (UTF-8, Shift_JIS, EUC-JP)

Three encodings commonly appear in Japanese environments, each with a different origin and use.

UTF-8

The most widely used encoding in the world today. It is one efficient way to encode "Unicode," which can represent nearly every character on earth in a single system. Web pages, smartphones, programming, and most modern applications use UTF-8 as standard. A half-width letter takes one byte, while Japanese kanji and kana usually take three. When in doubt, choose UTF-8 is the modern baseline.

Shift_JIS

A Japanese encoding long used mainly on Windows. Some Windows features, Excel, older software, and legacy business systems still use Shift_JIS (or its extension CP932). It appears often alongside UTF-8 in Japanese data exchange, and the mismatch between Shift_JIS and UTF-8 is the single most frequent cause of mojibake.

EUC-JP

A Japanese encoding once used on UNIX systems and some websites. It is rarely chosen for new work today, but you may still meet it in old servers, data, or past emails.

The key point is that even the same Japanese text can be saved in multiple encodings. That is exactly why the saving side and reading side must agree.

Why Mojibake Happens

Boiled down, there is one cause: the encoding used to save does not match the encoding used to read. In practice it shows up like this:

• A UTF-8 file opened as Shift_JIS: Japanese turns into a mixed string of odd kanji and kana symbols.
• A Shift_JIS file opened as UTF-8: replacement marks such as the black diamond or empty boxes appear.
• An email whose declared encoding differs from its actual content: the sender and receiver software interpret it differently and it breaks.
• The font lacks the character: even with the right encoding, a font without the glyph shows a box (tofu). This differs slightly in cause but looks similar.

So garbled text is not necessarily corrupted. In many cases the data itself is intact, and re-specifying the correct way to read it restores the original. That is why mojibake can often be fixed.

The Classic Case: CSV Breaking in Excel

The most common mojibake question is about CSV files breaking when opened in Excel. There is a clear reason.

Modern web services and systems increasingly export CSV in the global standard UTF-8. Windows Excel, however, has long tried to interpret a CSV as Shift_JIS (CP932) when you open it by double-clicking. As a result, Excel assumes a UTF-8 CSV is Shift_JIS and the Japanese breaks. The data is not damaged; only the assumed encoding is wrong.

There are two main directions for fixing this:

1. Specify the encoding when opening in Excel: use "From Text/CSV" on the Data tab and select the original encoding at import time to display correctly.
2. Convert the file's encoding: convert the CSV to Shift_JIS (CP932) in advance so it opens correctly even by double-click, or convert to UTF-8 when handing it to a UTF-8 system. The Encoding Converter helps with this conversion.

Also, saving as "UTF-8 with BOM" (a marker at the start) sometimes lets Excel auto-detect UTF-8 and open it correctly. If a CSV is meant to be opened in Excel, providing it as BOM-marked UTF-8 or Shift_JIS is considerate.

How to Fix and Convert Garbled Text

To restore text that is already garbled, try the following depending on the situation.

1. Switch the reading encoding

Browsers and text editors let you change the encoding used for display. Try switching a broken page or file among UTF-8, Shift_JIS, and EUC-JP to find the setting that reads correctly. Since the file is intact, choosing the right interpretation restores it.

2. Repair a garbled string

For email bodies or pasted text where you cannot switch encoding per file, you may want to repair just the broken string. In that case, paste the garbled text into the Mojibake Fixer and try the likely encoding combinations to recover the original Japanese. The idea is to reverse a typical pattern such as "UTF-8 read as Shift_JIS."

3. Convert and save the file encoding

Aligning the encoding in advance to match the receiving system avoids trouble. To convert between UTF-8 and Shift_JIS, use the Encoding Converter and unify the save format.

4. Normalize half-width, full-width, and kana variations

This is not mojibake itself, but when exchanging data, half-width kana and full-width/half-width inconsistencies can also cause breakage. To unify hiragana and katakana notation, the Kana Converter helps reduce unexpected problems downstream.

Tips to Prevent Mojibake

Finally, here are practical points to avoid mojibake in the first place.

• Standardize on UTF-8 when unsure: for new files and systems, UTF-8 is the modern default unless there is a special reason. If everyone uses the same encoding, nothing breaks.
• Check the recipient's environment: when distributing a CSV meant for Excel, choose Shift_JIS or BOM-marked UTF-8 to match the receiving side's standard.
• Declare the encoding explicitly: on web pages and in email, declaring the encoding correctly makes the recipient's software less likely to misinterpret it.
• Convert with a dedicated tool: manual copying or retyping invites mistakes. Handle encoding conversion and repair with a dedicated tool for reliable results.

Mojibake can look like serious data corruption, but most of the time it is just a wrong reading specification. With the mechanics of encoding in mind, you can isolate the cause and calmly restore the text with the right tool.