Two bytes for the world.

What a character already costs

Characters are stored with a length of 1–4 bytes. Which tier a script lands in was decided by Unicode's code-point ordering, not by how many people write in it — which is why Devanagari, Thai, Ethiopic and Chinese, between them the everyday scripts of billions, sit in UTF-8's expensive three-byte tier.

One bit, doing all the work

1 byte = 8 bits = 256 possible values. Call the first 128 the low byte, the last 128 the high byte. The high bit is the length signal: a high byte means “another byte follows,” a low byte means “the character ends here.” Decoding stops the instant a low byte appears — no length field, no lookahead.

The two positions have names, and they are just the high/low split above seen from the other side:

• Lead byte 0x80–0xFF — a high byte, so it always means “another follows.” It carries the character's slot: which of a page's 128 entries this is.
• Trail byte 0x00–0x7F — a low byte, so it terminates the character. It is the page index. Of its 128 possible values, 68 are usable, so tier 2 reaches 128 × 68 = 8,704 characters in 2 bytes where UTF-8 needs 3.

Note the order, because it inverts the usual arrangement: the second byte says which script you are in, and the first says which letter of it. UTF-8 puts the identifying information in the leading byte; UCE-8 puts it in the terminator. Devanagari न (U+0928) encodes as A8 49:

The range in that second line is not given — it is derived. The trail byte is the page index, and a page label is a code point with its last hex digit dropped, so putting it back gives the block:

You can see the two ends of that page directly: 80 49 decodes to U+0900 (slot 0) and FF 49 decodes to U+097F (slot 127).

Hold the lead byte at A8 and change only the trail byte, and the same slot number lands in a different script each time:

That is the whole addressing scheme: the trail byte chooses the page, the lead byte chooses the entry.

Which bytes can be a page index

Of the 128 possible trail values, 60 are excluded: the 33 control codes and space, plus 26 punctuation marks that carry structural meaning wherever text flows.

Only six punctuation marks survive as trail bytes, and none is a metacharacter in JSON, CSV, URLs, file paths, or the shell:

128 possible values minus 60 exclusions leaves 68 usable pages.

Where every page lives

Every page's address is a plain ASCII byte, so the whole page map fits in one table — row is the high nibble, column the low nibble, and the small character in each cell is the ASCII byte itself:

Read a cell as a two-byte character's second byte. Devanagari sits at row 4, column 9 — ASCII I — which is why नमस्ते is full of the letter I. The greyed cells are the 60 values ruled out in the previous section.

Spending 68 pages

Pages come in two kinds. 38 block pages are plain 128-code-point slices of Unicode, so the slot is arithmetic. 30 indexed pages are curated lists of 128 entries, so the slot is a lookup.

Chinese and Korean are indexed rather than block-mapped because Unicode orders Han characters by radical and Hangul syllables by jamo combination, and neither follows frequency. The first 1,920 Hangul syllables that 15 block pages would cover reach only 28% of real Korean text; 6 indexed pages reach 89%.

The flip side of frequency-ordering is a fallback: 768 of 11,172 Hangul syllables and 4,096 of 20,992 CJK ideographs reach 2 bytes. The rest — rare syllables, proper names, loanwords, most Traditional Chinese — cost 3.

Two layers, and only one of them is hard

It is worth separating the two things this page has been describing together, because they have very different prospects.

The savings measured further down belong to the tables. The properties worth arguing about — self-terminating, no structural bytes in the stream, exact fit to Unicode — belong to the core, and would survive a completely different set of tables. Anyone assessing UCE-8 should judge the two separately: a version with better tables is a data change, while a version with a different core is a different encoding.

Who it helps

These scripts all sit in UTF-8's three-byte range. “Page” is the page index — page 090 is U+0900–U+097F, and idx marks the three scripts that share one curated page instead of occupying an aligned block. “Trail byte” is the ASCII character each of that script’s characters ends on — Thai always ends on R, Devanagari on I. A few scripts span several pages and so several trail bytes.

Red trail bytes are block pages — an aligned 128-code-point slice of Unicode, so the slot is arithmetic. Blue are indexed pages — a curated list of 128 entries, so the slot is a lookup. Chinese uses both.

Page ^ is what indexing buys. Tamil fills 61 of its slots, Georgian 33 and Mongolian 34 — 61 + 33 + 34 = 128 — so three scripts share a single page exactly, with nothing left over. As block pages they would have taken four: one each for Tamil and Georgian, and two for Mongolian, whose characters straddle a 128-code-point boundary. Indexing turns those four pages into one and hands 384 slots back to the budget.

The Mongolian split is the one to keep straight: Mongolia writes in Cyrillic, which UCE-8 leaves exactly the same size. The 27% gain belongs to the traditional script, used mainly in Inner Mongolia. Counting all 10 million Mongolian speakers as beneficiaries of the traditional-script pages overstates the case, and it is the first thing a careful reader will check.

What it actually saves

The raw saving is deterministic: a character that cost 3 bytes costs 2, a flat 33% for that tier. Real text is a mix, so real documents land lower. And because text is usually compressed in transit and at rest, the compressed figure is the one that decides most real deployments.

So the defensible claim is narrower than “33% for everyone.” UCE-8 wins clearly on uncompressed text, and keeps a real but modest 5–9% once a compressor has run — consistently, across every script tested, at both gzip and brotli. That margin is worth having where text is stored or sent uncompressed, and worth little where it is not.

What it costs

UCE-8 is not a drop-in, and the case for it is only credible if the costs are stated alongside the savings.

Telling the two apart

The two coexist without an encoding label, BOM or configuration because UTF-8 gets first refusal: if the byte stream is valid UTF-8, it is treated as UTF-8; only when UTF-8 rejects it is UCE-8 considered.

Plain ASCII is byte-identical in both formats, so ASCII files need no distinction at all — either decoder produces exactly the same text.

That is the approach used by the converter and editor in this repository. On ordinary files, the distinction is decisive, so a single reader can handle a directory containing both UTF-8 and UCE-8 files without labels, configuration or user intervention.

So detection is dependable for ordinary text, and undecidable only where it does not matter — pure ASCII decodes identically either way. Ambiguity can still be constructed, but not usefully: of the 130,560 three-byte buffers that are valid both ways and disagree, every single one decodes on the UCE-8 side to a code point that is unassigned. C2 80 21 is \x80! in UTF-8 and U+8C400 in UCE-8, which is not a character at all — it sits in Plane 8, which Unicode has never allocated. The plane test below rejects the whole set. The real limitation is more mundane: sniffing needs the entire buffer up front, which a streaming reader does not have.

That 130,560 is not an estimate — it is the whole of the overlap, and it can be counted directly. Only one byte shape can be valid both ways: a two-byte UTF-8 character followed by an ASCII byte, which UCE-8 reads instead as two lead bytes closed by a terminator. Every other UTF-8 shape ends in a continuation byte 10xxxxxx, which is never a legal UCE-8 terminator, so three- and four-byte UTF-8 characters cannot collide at all.

The deeper limit is that sniffing has to be implemented. It helps exactly where the reader can be changed — which is where an explicit label would have served too. It does nothing for grep, a database column, or a terminal that was never told UCE-8 exists. Detection softens the transition; it does not remove the break.

Sharing a disk with GB 18030

In the markets the two-byte tier was built for, the incumbent is often GB 18030 — the Chinese national standard, mandatory for software sold in China — or its subset GBK. Unlabelled text is ambiguous between them, but the ambiguity is one-sided: forty Chinese characters offered to a GB decoder are structurally valid 45.7% of the time in UTF-8 and never in UCE-8, because eleven of the 32 Chinese pages are addressed by digits and by !, ( and ), which GB accepts as no ordinary trail byte. The overlap is unavoidable because both formats wanted the same bytes — ones that stay inert in C strings, shell quoting and data formats — and 65 of UCE-8’s 68 trail bytes are also GB continuation bytes. A GB-disjoint variant would have three trail bytes and hold 384 characters.

Reading the other way, the byte grammar is not enough on its own. A two-byte GB character followed by an ASCII letter is exactly UCE-8’s tier-3 shape, so text that strictly alternates between the two is accepted by a bare decode almost in full. Ordinary GB prose is not: consecutive GB characters put a high byte where UCE-8 requires a terminator, and the byte grammar rejects them before anything is decoded. What catches the alternating residue is the plane test: CA AE 61 — 十 then a — reads as U+9E061, in a plane Unicode has never allocated. With the full rule applied, 0 of 4,000 GB buffers were read as UCE-8. The plane test is therefore not optional, though on ordinary GB text it is the byte grammar that does nearly all of the work.

A reader updated for UCE-8 is not exposed to that risk, because it tests the buffer instead of assuming an encoding. Valid UTF-8 is read as UTF-8 outright; everything else passes through four stages, each removing part of what the one before it admits.

The rule this implies

Detection is only worth specifying if it is exact, and it can be. UCE-8 is for modern text: legacy single-byte encodings are out of scope, exactly as they are for JSON, which requires UTF-8, and for HTML5, which drops them from conformance. That leaves two formats, and a rule with one exception:

The margin that buys is larger than it looks. Unicode has 17 planes, and the last three — 14, 15 and 16 — are special-purpose and private use, leaving 14 planes for writing systems. Four of those are allocated, planes 0 to 3, and they already hold every widely used script. Ten remain, and the rule comes under pressure once allocation reaches plane 6 — two further planes away. Those two are 131,072 code points: 1.4 times everything Unicode has placed in the supplementary planes in the 25 years since they opened. At that rate they are roughly thirty-five years of allocation, and at the slower pace of recent releases, closer to fifty.

Planes 6–15 are not entirely spoken for either. They hold 655,360 code points, of which only the 130,560 ambiguous ones are genuinely unusable; the remaining 524,800 — eight more planes’ worth — no UTF-8 collision can reach. A decoder that tests the sequence itself, asking whether those same three bytes are also valid UTF-8, rather than refusing the planes wholesale, leaves that space free for Unicode to allocate. Counting both, the rule has 655,872 code points — ten planes — of runway before it needs revisiting at all.

Tested across 10,026 files in both encodings, that rule classified every one correctly, with binary rejected as neither. The distinction that matters in the wording is refuse rather than misread: an implementation need not decode Latin-1, but it must not present it as text either. A Latin-1 misread produces ordinary plane-0 characters — se然r ni然 for señor niño — so the plane test cannot see it, and a reader that wants to catch those during the transition needs a second, optional check.

One character, one spelling

Tier 3’s arithmetic runs over every code point from U+0000 up. It does not skip the ones tiers 1 and 2 already reach, so the arithmetic also produces a three-byte spelling for every ASCII character and every two-byte character. Walking the entire byte space counts them exactly:

A decoder that accepted those long forms would give those 8,832 characters two spellings each. That is UTF-8’s overlong-encoding bug, and it matters most for the shortest of them:

Enforcing it costs one comparison: a three-byte sequence is only valid if the code point it yields has no shorter form.

def decode_tier3(rest, tail):
    cp = rest * TAIL_COUNT + tail
    if len(encode(cp)) < 3:
        handle_error("overlong: this character has a shorter form")
    return cp

That leaves 128 one-byte, 8,704 two-byte and 1,105,280 three-byte sequences valid — 1,114,112 in total, exactly one per code point. The rule turns the encoding into a bijection: every character has one spelling and every valid sequence is one character. Stated plainly: a UCE-8 stream is canonical when no sequence is longer than it needs to be, and a decoder must reject the rest rather than accept two spellings of one character. The check refuses exactly the 8,832 alias sequences — 128 ASCII and 8,704 tier-2 — and keeps all 1,105,280 genuine three-byte ones, with every assigned code point still round-tripping. The reference implementation now enforces it, as do the command-line converter, the Python codec, the editor and the browser tools: 80 80 67 is refused rather than read as /.

The one-byte look-back

The third cost is the one that sounds fatal and isn’t, so it is worth working through. Devanagari’s page index is 0x49 — the letter I — so नमस्ते encodes to a stream full of the byte 49:

Search that plus the text ISO 8601 for the byte "I" and a naive scan returns 7 matches, only 1 of them real.

The fix is a single comparison. A low byte always terminates a character, so the byte at position i begins a character exactly when the byte before it is also low:

def is_char_start(data, i):
    """A low byte terminates a character, so position i begins one."""
    return i == 0 or not (data[i - 1] & 0x80)

def search(haystack, needle):
    hits, i = [], haystack.find(needle)
    while i != -1:
        if is_char_start(haystack, i):
            hits.append(i)
        i = haystack.find(needle, i + 1)
    return hits

That returns 1 match — the real one. And only the first byte of the match needs checking: if it starts a character and is low, it also terminates one, so the next byte starts a character too, and the property carries along the whole needle.

Why not make the terminator self-identifying?

Because that is most of the encoding’s advantage. Reserving a bit pattern to mark continuation bytes — UTF-8’s approach — costs half the two-byte tier:

UCE-8 buys 2.1× the capacity of the self-identifying variant by giving up context-free bytes. One byte of look-back is the cheaper side of that trade.

The encoder, in full

The whole encoder is three branches and a fallback. LEAD_BASE is 0x80 — setting the high bit marks a byte as “more follows.” TAIL_COUNT is 68, the number of usable trail values.

The four files above download directly from this site; the repository is canonical and stays current. uce.py has no dependencies beyond the Python standard library; run it directly to see the built-in test vectors, or run verify.py to reproduce every number on this page. uce8_convert.py imports uce.py from the same directory, so keep the two together. To convert a file without installing anything, the browser converter does the same job in a tab. uce.c is a second, independent implementation in C — the frozen tables laid out as flat arrays, a perfect hash for the indexed pages, and the same canonical-form check. It needs only a C compiler, and it encodes byte-identically to the Python across all 1,114,112 code points, which is what makes it useful: two implementations agreeing is a stronger statement about the specification than either one alone.

def encode(code_point):
    """Encode a Unicode code point into 1, 2 or 3 bytes."""

    if code_point < 0:
        handle_error("Negative code point")

    if code_point > MAX_CODE_POINT:
        handle_error("Beyond the Unicode range")

    # TIER 1 - plain ascii passes straight through
    if code_point < 0x80:
        return bytes([code_point])

    # TIER 2a - block pages: world scripts and the 8 Chinese blocks
    trail = PAGE_TO_TRAIL.get((code_point >> 7) << 3)
    if trail is not None:
        return bytes([LEAD_BASE | (code_point & 0x7F), trail])

    # TIER 2b - indexed Chinese
    slot = CHINESE_SLOT.get(code_point)
    if slot is not None:
        return bytes([LEAD_BASE | (slot & 0x7F), CHINESE_TRAILS[slot >> 7]])

    # TIER 2c - indexed Korean
    slot = KOREAN_SLOT.get(code_point)
    if slot is not None:
        return bytes([LEAD_BASE | (slot & 0x7F), KOREAN_TRAILS[slot >> 7]])

    # TIER 2d - indexed world pages
    slot = WORLD_SLOT.get(code_point)
    if slot is not None:
        return bytes([LEAD_BASE | (slot & 0x7F), WORLD_TRAILS[slot >> 7]])

    # TIER 3 - everything else
    index = code_point
    tail = index % TAIL_COUNT
    rest = index // TAIL_COUNT

    return bytes([
        LEAD_BASE | (rest >> 7),
        LEAD_BASE | (rest & 0x7F),
        TAIL_BYTES[tail],
    ])

Decoding reads the high bit of the second byte to pick the tier, then the trail byte to pick the page:

def decode(data):
    """Decode a 1, 2 or 3 byte sequence back into a Unicode code point."""

    if not data:
        handle_error("Empty sequence: nothing to decode")

    first = data[0]

    if first < 0x80:
        return first

    if len(data) < 2:
        handle_error("Truncated: 1 lead byte with no terminating byte")

    second = data[1]

    # TIER 2 - second byte clears the high bit, so it terminates
    if second < 0x80:
        page = TRAIL_TO_PAGE[second]

        if page is None:
            handle_error("Trail byte is not a page index")

        if page is WORLD:
            return WORLD_CODE_POINTS[(WORLD_PAGE[second] << 7) | (first & 0x7F)]

        if page is CHINESE:
            return CHINESE_CODE_POINTS[(CHINESE_PAGE[second] << 7) | (first & 0x7F)]

        if page is KOREAN:
            return KOREAN_CODE_POINTS[(KOREAN_PAGE[second] << 7) | (first & 0x7F)]

        return ((page >> 3) << 7) | (first & 0x7F)

    # TIER 3 - second byte keeps the high bit, a third follows
    if len(data) < 3:
        handle_error("Truncated: 2 lead bytes with no terminating byte")

    third = data[2]

    if third >= 0x80:
        handle_error("No terminating byte within 3 bytes")

    tail = TAIL_INDEX[third]

    if tail is None:
        handle_error("Third byte is not a legal terminating value")

    rest = ((first & 0x7F) << 7) | (second & 0x7F)

    code_point = rest * TAIL_COUNT + tail

    # CANONICAL FORM - a character may only be spelled the shortest way
    if len(encode(code_point)) < 3:
        handle_error("Overlong sequence: U+%04X has a shorter form" % code_point)

    return code_point

Encoding a whole string is a join, since every character is independent:

def uce8_encode(text):
    return b"".join(encode(ord(c)) for c in text)

Note what tier 3 does differently. Tiers 1 and 2 map a code point to a page and slot; tier 3 has no page table left, so it spreads the remaining code points across the 68 legal trail values arithmetically — index % TAIL_COUNT picks the terminator, index // TAIL_COUNT becomes the two lead bytes. That is why tier 3's capacity is exactly 128 × 128 × 68, and why the encoding never emits an illegal terminator even in its fallback path.