Unicode and UTF-8 Explained: From Mojibake to Emoji

If you have ever received an email that said “Café” instead of “Café”, or opened a CSV where every accented name turned into a pair of odd symbols, you have met the core problem of text on computers. A computer does not store letters. It stores numbers, and the program reading them decides which letter each number means. When the writer and the reader use different tables, text breaks. Unicode and UTF-8 are the long, mostly successful effort to make the whole world use one table.

ASCII and the 128 slots

ASCII, standardised in the United States in 1963, used seven bits: 128 codes for English letters, digits, punctuation and control characters such as line feed. A is 65 (hex 41); line feed is 10 (0A). To this day, the first 128 positions of every major encoding match ASCII.

The trouble came from the eighth bit. A byte holds 256 values, and every country and vendor filled the upper 128 differently. Western Europe got ISO 8859-1 (Latin-1) with é, ñ and ü; Microsoft shipped the similar but not identical Windows-1252, which adds curly quotes and the euro sign in a range Latin-1 leaves for control codes; Russia, Greece, Israel and others each had their own sets. The byte 0xE9 was é in Paris and a Cyrillic letter in Moscow.

That is exactly where “Café” comes from. In UTF-8, é is two bytes, C3 A9. A program that assumes Windows-1252 reads C3 as à and A9 as ©. The bytes are intact — reading them again as UTF-8 recovers the text — but every accented letter has doubled into garbage.

East Asia: two bytes and still not enough

Chinese, Japanese and Korean needed thousands of characters, so their encodings used two bytes per character. Japan’s Shift_JIS, Korea’s EUC-KR and China’s GB2312 were all incompatible with each other.

These systems had sharp edges. In Shift_JIS the second byte of a character could be 0x5C — the ASCII backslash. Japanese words containing 表 (95 5C) or ソ (83 5C) confused C programs and shells, which treated that byte as an escape and swallowed the next character. And because Japanese and Korean standards put their own currency symbols at 0x5C, Windows file paths still render as C:¥Users in Japanese fonts and C:₩Users in Korean ones. It is the same backslash byte underneath.

Korea’s national standard (KS C 5601, now KS X 1001) included only 2,350 precomposed Hangul syllables out of the 11,172 that modern Korean orthography allows, so some real names and words simply could not be typed. Microsoft’s extension, code page 949, filled the gap, and for years a large share of pages labelled EUC-KR were actually CP949.

The Japanese word for garbled text, mojibake (文字化け, “character transformation”), entered English because Japanese users saw so much of it.

The replacement character trap

Korean internet users will recognise 占쏙옙 as the most famous garbled string. It has a precise origin. When a UTF-8 decoder meets bytes it cannot decode, it substitutes U+FFFD, the replacement character (�). Saved as UTF-8, that is EF BF BD. Two of them in a row — EF BF BD EF BF BD — read as CP949 pair up as EF BF, BD EF and BF BD, which happen to be 占, 쏙 and 옙. So 占쏙옙 is a double accident: text that was already destroyed once, then misread a second time. Unlike “Café”, nothing can be recovered from it.

Unicode: one number per character

In the late 1980s Joe Becker at Xerox, with Lee Collins and Mark Davis at Apple, began work on a single universal character set. The Unicode Consortium was incorporated in California on 3 January 1991, and Volume 1 of The Unicode Standard 1.0 appeared that October.

The founding assumption was that 16 bits — 65,536 code points — would be enough for every script in modern use. Windows NT and Java were designed around that assumption, storing text as 16-bit units. It did not hold: historic and rare CJK ideographs alone needed tens of thousands more slots. Unicode 2.0 in 1996 extended the code space to U+10FFFF, just over 1.1 million code points, and introduced surrogate pairs so that 16-bit systems could survive. A reserved block, U+D800–DFFF, is used in pairs to represent anything above U+FFFF. That is why “😀”.length is 2 in JavaScript and Java: they are counting 16-bit units, not characters.

Unicode itself only defines the mapping from character to number. How that number becomes bytes is the job of an encoding form: UTF-8, UTF-16 or UTF-32.

UTF-8: designed on a placemat

In September 1992, Ken Thompson and Rob Pike of Bell Labs were asked to comment on a proposed encoding, FSS-UTF, that X/Open was about to vote on. Pike objected that it lacked a property he considered essential — the ability to pick up a byte stream mid-run and find the next character boundary while losing less than one character. Over dinner at a New Jersey diner, Thompson worked out a better bit layout on a placemat. By Pike’s account, Thompson wrote the encoder and decoder that night, the proposal went to X/Open on 8 September, and by the end of that week their Plan 9 operating system was running entirely on what became UTF-8.

The scheme fits in a table:

Code point range	Byte pattern	Example
U+0000–007F	0xxxxxxx	A → 41
U+0080–07FF	110xxxxx 10xxxxxx	é → C3 A9
U+0800–FFFF	1110xxxx 10xxxxxx 10xxxxxx	€ → E2 82 AC
U+10000–10FFFF	11110xxx then three 10xxxxxx	😀 → F0 9F 98 80

The original design allowed up to six bytes to cover 31 bits. RFC 3629 (2003) trimmed it to four bytes to match Unicode’s ceiling of U+10FFFF and made it illegal to encode surrogate code points in UTF-8. You can see the pattern at work, character by character, in the UTF-8 byte viewer.

Why UTF-8 won

UTF-16 got into operating systems and languages first, yet UTF-8 took over files, protocols and the web. The reasons are in the design:

• Backward compatible with ASCII. Any ASCII file is already valid UTF-8, byte for byte. Nothing had to be converted.
• Bytes below 0x80 are always ASCII. Every byte of a multi-byte character is 0x80 or higher, so tools that search for /, newline or NUL never split a character. The Shift_JIS backslash problem cannot happen.
• Self-synchronising. Lead bytes start with 11 and continuation bytes with 10. Start reading anywhere, or lose a byte to corruption, and you are back in step at the next character.
• No byte order. UTF-16 must choose big- or little-endian and often needs a byte order mark to say which. UTF-8 is a byte sequence; there is nothing to swap.
• Byte order equals code point order. Sorting UTF-8 strings as raw bytes gives the same result as sorting by code point.

The cost is size for some scripts. Korean, Japanese and Chinese characters that took two bytes in legacy encodings take three in UTF-8. Compatibility won anyway. The WHATWG Encoding Standard, which browsers follow, now states that new protocols and formats “must use the UTF-8 encoding exclusively.”

Same text, different bytes: normalization

Unicode often allows more than one way to write the same visible text. é can be the single code point U+00E9, or e (U+0065) followed by COMBINING ACUTE ACCENT (U+0301). Korean syllables can be precomposed (한, U+D55C) or spelled out as three conjoining jamo.

Unicode Standard Annex #15 defines four normalization forms to reconcile them:

• NFC composes wherever possible. It is what keyboards and most web content produce.
• NFD decomposes everything.
• NFKC and NFKD also fold “compatibility” characters: ① becomes 1, fi becomes fi, full-width Ａ becomes A. Search indexes and username checks use these.

The most common place to hit this is filenames. Apple’s older HFS+ file system stored names in a decomposed form, and files created on a Mac still frequently arrive in zip archives or uploads with decomposed names. A search for “résumé” typed on a keyboard (NFC) will not match “résumé” from such a filename (NFD) unless someone normalizes first — in JavaScript, str.normalize(“NFC”).

There is no single “length”

Ask how long a string is and the answer depends on what you count. Take the family emoji 👨‍👩‍👧:

Unit	Count	Where it shows up
UTF-8 bytes	18	File size, database byte limits, network payloads
UTF-16 code units	8	JavaScript and Java .length
Code points	5	Python len(), Array.from
Grapheme clusters	1	What a person sees; cursor movement

The emoji is man + ZERO WIDTH JOINER + woman + ZWJ + girl, which is why it has five code points. What a reader perceives as one character is defined in Unicode Standard Annex #29 as an extended grapheme cluster, and browsers expose it through Intl.Segmenter. The word counter uses that to count characters the way people do. When a form, a text message and a social network disagree about your character count, they are usually counting different rows of this table.

Bugs that still ship

• Emoji rejected by MySQL. The old utf8 character set (utf8mb3) stores at most three bytes per character, so four-byte emoji fail or get truncated. Use utf8mb4.
• Truncating by bytes. Cut a UTF-8 string at an arbitrary byte and you can split a character; the leftover byte becomes �.
• Half a surrogate pair. substring() in JavaScript can cut an emoji in half, leaving \uD83D alone. The Unicode escape converter makes the orphan visible.
• Excel and UTF-8 CSV. Double-clicking a UTF-8 CSV often makes Excel guess the wrong encoding. Saving with a byte order mark (EF BB BF) at the start usually fixes it. The same BOM at the start of a JSON file or shell script causes parse errors.
• Double-escaped HTML. Seeing & on a page means text was escaped on the way into storage and again on output; the HTML entity encoder explains which characters need escaping where.
• Look-alike characters. A hyphen-minus (U+002D) and a minus sign (U+2212), or Latin a and Cyrillic а, are different to every program. The symbol picker shows the code point of what you choose.

Diagnosing garbled text

1. Look at the pattern. Sequences like é, ñ or ’ mean UTF-8 bytes were read as Latin-1 or Windows-1252. The bytes survived; reread the file as UTF-8.
2. If you see � or 占쏙옙, a decoder already replaced the bytes. The original text is gone and must be fetched again from the source.
3. If text looks right but will not match, suspect normalization (NFC versus NFD), a look-alike character or an invisible space such as U+00A0 or U+200B. Paste both strings into a byte viewer and compare the rows.