75 lines
5.3 KiB
HTML
75 lines
5.3 KiB
HTML
<!doctype html>
|
|
<html>
|
|
<head>
|
|
<title>UTF-63</title>
|
|
<meta charset="UTF-8">
|
|
<link rel="stylesheet" href="../../fonts/source-sans-pro.css">
|
|
<link rel="stylesheet" href="utf63.css">
|
|
</head>
|
|
<body>
|
|
<article>
|
|
<h1>UTF-63</h1>
|
|
<address>
|
|
<time>July 2026</time>
|
|
·
|
|
By <a rel="author" href="../..">Paul Brinkmeier</a>
|
|
</address>
|
|
<p>
|
|
UTF-8 is almost everywhere these days.
|
|
The web runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
|
|
It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols.
|
|
</p>
|
|
|
|
<p>
|
|
What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage?
|
|
Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up).
|
|
<!-- Unicode also defines UTF-16 and UTF-32 but they're even worse for this case, see e.g. the <a href="https://utf8everywhere.org/">UTF-8 Everywhere Manifesto</a>. -->
|
|
Even if we ignore the costly multibyte emoji encodings, using UTF-8, 1 out of every 8 bits of memory is wasted on the encoding, because 7-bit ASCII characters are encoded using a whole byte, i.e. we have an encoding overhead of roughly 15%.
|
|
</p>
|
|
<p>
|
|
For example, consider the UTF-8 encoding of the string “🙈🙉🙊 encoding” (see <a href="https://utoy.beany.club/text/🙈🙉🙊 encoding" title="also cURLable!">here</a> for codepoints):
|
|
</p>
|
|
<pre>11110<span class="red2">000</span>10<span class="red2">011111</span>10<span class="red2">011001</span>10<span class="red2">001000</span> <span class="red2">🙈</span>
|
|
11110<span class="orange2">000</span>10<span class="orange2">011111</span>10<span class="orange2">011001</span>10<span class="orange2">001001</span> <span class="orange2">🙉</span>
|
|
11110<span class="yellow2">000</span>10<span class="yellow2">011111</span>10<span class="yellow2">011001</span>10<span class="yellow2">001010</span> <span class="yellow2">🙊</span>
|
|
0<span class="aqua2">0100000</span>0<span class="blue2">1100101</span>0<span class="purple2">1101110</span>0<span class="red2">1100011</span> <span class="aqua2"> </span><span class="blue2">e</span><span class="purple2">n</span><span class="red2">c</span>
|
|
0<span class="orange2">1101111</span>0<span class="yellow2">1100100</span>0<span class="aqua2">1101001</span>0<span class="blue2">1101110</span> <span class="orange2">o</span><span class="yellow2">d</span><span class="aqua2">i</span><span class="blue2">n</span>
|
|
0<span class="purple2">1100111</span> <span class="purple2">g</span></pre>
|
|
<p>
|
|
Colored spans show which code points are encoded by which bits.
|
|
Neutral (white on black) bits are encoding overhead required by UTF-8.
|
|
For emoji, the most important medium of modern communication, this encoding is obviously horrible.
|
|
Every monkey requires a full 32 bits to encode its 21 bits.
|
|
And even for good old reliable ASCII, useless bits are sprinkled here and there.
|
|
There's gotta be a better way!
|
|
</p>
|
|
<h2>UTF-63</h2>
|
|
<p>
|
|
Enter: An entirely new way to encode Unicode!
|
|
</p>
|
|
<pre>1<span class="red2">000011111011001001000</span><span class="orange2">0000111110</span> <span class="red2">🙈</span><span class="orange2">🙉</span>
|
|
<span class="orange2">11001001001</span><span class="yellow2">000011111011001001010</span> <span class="yellow2">🙊</span>
|
|
0<span class="aqua2">0100000</span><span class="blue2">1100101</span><span class="purple2">1101110</span><span class="red2">1100011</span><span class="orange2">110</span> <span class="aqua2"> </span><span class="blue2">e</span><span class="purple2">n</span><span class="red2">c</span><span class="orange2">o</span>
|
|
<span class="orange2">1111</span><span class="yellow2">1100100</span><span class="aqua2">1101001</span><span class="blue2">1101110</span><span class="purple2">1100111</span> <span class="yellow2">d</span><span class="aqua2">i</span><span class="blue2">n</span><span class="purple2">g</span></pre>
|
|
<p>
|
|
For our entirely random example, this UTF-63 encoding uses only 16 bytes, a clear win over the whopping 21 bytes required by UTF-8 (that's 30% more!).
|
|
So how does it work?
|
|
</p>
|
|
<p>
|
|
Decoding is dead simple:
|
|
We consider each set of 8 bytes a Big Endian 64-bit code unit.
|
|
If the most significant bit is 0, the other 63 bits contain nine Basic Latin characters (i.e. ASCII characters) at 7 bits each.
|
|
If it's 1, they contain any three Unicode code points at 21 bits each (the maximum codepoint is U+10FFFF).
|
|
When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways).
|
|
</p>
|
|
<h2>Usage</h2>
|
|
<p>
|
|
Admittedly, this page actually still uses UTF-8, simply because most modern browsers don't support UTF-63 yet.
|
|
To my knowledge, most OSes, desktop applications and programming languages don't either.
|
|
It wouldn't surprise me if VLC supported UTF-63 subtitles or something though.
|
|
You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between.
|
|
</p>
|
|
</article>
|
|
</body>
|
|
</html>
|