Character Encoding and UTF-8 Basics: ASCII vs Shift_JIS and Mojibake

Have you ever opened a web page only to see broken text like "ã‚ã„ã†"? This is called mojibake (garbled text), and most of the time it is caused by a mismatch in character encoding. A character encoding is the foundation that converts the characters we use without thinking into numbers that computers can handle. This article walks through, step by step, what a character encoding is, the difference between ASCII, Shift_JIS, and UTF-8, how UTF-8's variable length works, and what garbled text really is.

What Is a Character Encoding

Internally, a computer processes everything as numbers made of 0s and 1s, and characters are no exception. To handle a character like "A" or "あ," you have to assign each character a number and prepare a mapping table that says "this number is this character." This correspondence itself is called a character encoding.

More precisely, a character encoding has two layers. One is the coded character set, which gives each character a number (for example, in Unicode "あ" is U+3042). The other is the encoding form, which decides how that number is actually laid out as a sequence of bytes. Even within the same Unicode, UTF-8 and UTF-16 arrange the actual bytes differently. When we casually say "character encoding," it is usually fine to think of it as referring to both of these together.

To see what bytes (numbers) a character is finally converted into, it helps to try it in the Character Encoding Converter. You can feel how even the same "あ" produces different numbers depending on the encoding you choose.

From ASCII to Shift_JIS, and Then Unicode

The history of character encodings is also a history of expanding the set of characters that can be handled.

ASCII is the most basic character encoding, defined in the United States in the 1960s. With 7 bits (128 values from 0 to 127) it represents uppercase and lowercase letters, digits, symbols, and control characters. That was enough for alphabetic languages, but it could not represent languages with a huge number of characters, such as Japanese.

So in Japan, Shift_JIS was created as an extension of ASCII. To represent hiragana, katakana, and kanji, it expresses characters that do not fit in one byte using two bytes. Likewise, each country created its own encoding (GB for Chinese, EUC-KR for Korean, and so on). But this "different in every country" situation caused a serious problem: opening a document made with another encoding produced garbled text.

Unicode was devised to solve this proliferation. With the grand vision of "giving a single number to every character in the world," it made it possible to handle characters uniformly across borders. And the leading way to express this Unicode as an actual sequence of bytes is UTF-8.

How UTF-8 Works: Variable Length, 1 to 4 Bytes

The biggest feature of UTF-8 is that it is "variable length," changing the number of bytes used per character. It represents a single character using 1 to 4 bytes.

• 1 byte: the same range as ASCII (letters, digits, basic symbols). The leading bit starts with 0.
• 2 bytes: extended Latin letters, Greek, Cyrillic, and so on.
• 3 bytes: characters of many languages, including hiragana, katakana, and kanji.
• 4 bytes: emoji and some rare characters.

Here lies the clever design of UTF-8. The ASCII range (0 to 127) is represented by exactly the same single byte in UTF-8. In other words, a file written only with letters and digits reads the same whether you read it as ASCII or as UTF-8. This backward compatibility with ASCII is one major reason UTF-8 spread explosively.

Another bit of cleverness is that you can tell "which byte of how many bytes this is" just by looking at the leading bits of each byte. The leading byte has a bit pattern that indicates how many bytes the character spans, while every continuation byte begins with "10." This makes it easy to find character boundaries even when you start reading from the middle of the data, giving it an error-resistant structure.

When you want to confirm concrete byte counts, such as "あ" being 3 bytes and "A" being 1 byte, convert characters to hexadecimal in the Hex Text Converter. You can see the variable-length mechanism with your own eyes, for example how "あ" becomes the three bytes E3 81 82.

What Is a BOM

A BOM (Byte Order Mark) is a special few bytes placed at the start of a file, serving as a marker that indicates which Unicode encoding the file uses and in what byte order it is written.

Originally the BOM was created for methods like UTF-16 and UTF-32, which line up numbers in 2-byte or 4-byte units, in order to distinguish their byte order (big-endian or little-endian). Since UTF-8 works byte by byte, it does not really need a byte-order distinction; even so, a 3-byte BOM of EF BB BF is sometimes placed at the start as a declaration that "this file is UTF-8."

UTF-8 BOM: EF BB BF

A BOM has the advantage of being able to make UTF-8 explicit, but caution is needed. In programs that do not support a BOM, these leading 3 bytes can be treated as extra characters or invisible symbols and cause trouble. For example, in program source code or configuration files, a UTF-8 file with a BOM can lead to unexpected errors, so "UTF-8 without BOM" is generally recommended in many situations.

Why UTF-8 Became the World Standard

Today more than 90% of web pages are written in UTF-8, making it the de facto world standard. The reasons are the accumulation of the features we have seen so far.

• It handles the world's characters with one method: Japanese, English, and emoji can all be mixed in a single file without switching encodings.
• It is compatible with ASCII: the letters-and-digits part is identical to traditional ASCII, so existing systems and files could migrate without breaking.
• It is space-efficient: for data that is mostly letters and digits, one byte is enough, so there is less waste than methods that use two or more bytes for everything.
• It has an error-resistant structure: because character boundaries can be identified from the leading bits of bytes, damage does not spread easily even if part of the data is corrupted.

Japanese environments once widely used Shift_JIS and EUC-JP, but thanks to the trend toward internationalization and these advantages, choosing UTF-8 for newly created systems and files is now the norm. When you want to align old data to UTF-8, you can unify it safely by converting in the Character Encoding Converter.

Why Garbled Text (Mojibake) Happens

In most cases the true cause of garbled text is that "the encoding used when writing and the encoding used when reading do not match." Characters are stored as numbers (byte sequences), but if you turn those numbers back into characters using a different mapping table, they appear as entirely different characters.

For example, if you open a Japanese file saved in Shift_JIS while assuming it is UTF-8, the same byte sequence is interpreted with a different table and turns into meaningless strings. This is typical mojibake. The same thing happens in reverse if you open a UTF-8 file as Shift_JIS.

To prevent garbled text, the following points matter.

• Unify files to UTF-8 and avoid mixing encodings.
• In HTML, declare UTF-8 with the meta charset and make it match the information the server sends.
• When handling old data, correctly identify the original encoding before converting.

If you have data whose encoding is unknown, you can make it easier to isolate the cause and repair it by checking the byte sequence in the Hex Text Converter or converting to the correct encoding in the Character Encoding Converter. Once you understand how character encodings work, you will be able to deal calmly with garbled text whenever you encounter it.

Please note that this article is intended to provide general information. The details of handling character encodings can differ by system and language environment, so make your decision after checking the latest official specifications and documentation.