UTF-63: First draft

2026-08-05 02:37:24 +02:00
parent a3ad633087
commit 1076663c8e
2 changed files with 157 additions and 0 deletions
@@ -0,0 +1,74 @@
+<!doctype html>
+<html>
+<head>
+  <title>UTF-63</title>
+  <meta charset="UTF-8">
+  <link rel="stylesheet" href="../../fonts/source-sans-pro.css">
+  <link rel="stylesheet" href="utf63.css">
+</head>
+<body>
+  <article>
+    <h1>UTF-63</h1>
+    <address>
+      <time>July 2026</time>
+      ·
+      By <a rel="author" href="../..">Paul Brinkmeier</a>
+    </address>
+    <p>
+      UTF-8 is almost everywhere these days.
+      The web runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
+      It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols.
+    </p>
+
+    <p>
+      What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage?
+      Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up).
+      <!-- Unicode also defines UTF-16 and UTF-32 but they're even worse for this case, see e.g. the <a href="https://utf8everywhere.org/">UTF-8 Everywhere Manifesto</a>. -->
+      Even if we ignore the costly multibyte emoji encodings, using UTF-8, 1 out of every 8 bits of memory is wasted on the encoding, because 7-bit ASCII characters are encoded using a whole byte, i.e. we have an encoding overhead of roughly 15%.
+    </p>
+    <p>
+      For example, consider the UTF-8 encoding of the string “🙈🙉🙊 encoding” (see <a href="https://utoy.beany.club/text/🙈🙉🙊 encoding" title="also cURLable!">here</a> for codepoints):
+    </p>
+    <pre>11110<span class="red2">000</span>10<span class="red2">011111</span>10<span class="red2">011001</span>10<span class="red2">001000</span>  <span class="red2">🙈</span>
+11110<span class="orange2">000</span>10<span class="orange2">011111</span>10<span class="orange2">011001</span>10<span class="orange2">001001</span>  <span class="orange2">🙉</span>
+11110<span class="yellow2">000</span>10<span class="yellow2">011111</span>10<span class="yellow2">011001</span>10<span class="yellow2">001010</span>  <span class="yellow2">🙊</span>
+0<span class="aqua2">0100000</span>0<span class="blue2">1100101</span>0<span class="purple2">1101110</span>0<span class="red2">1100011</span>  <span class="aqua2"> </span><span class="blue2">e</span><span class="purple2">n</span><span class="red2">c</span>
+0<span class="orange2">1101111</span>0<span class="yellow2">1100100</span>0<span class="aqua2">1101001</span>0<span class="blue2">1101110</span>  <span class="orange2">o</span><span class="yellow2">d</span><span class="aqua2">i</span><span class="blue2">n</span>
+0<span class="purple2">1100111</span>                          <span class="purple2">g</span></pre>
+    <p>
+      Colored spans show which code points are encoded by which bits.
+      Neutral (white on black) bits are encoding overhead required by UTF-8.
+      For emoji, the most important medium of modern communication, this encoding is obviously horrible.
+      Every monkey requires a full 32 bits to encode its 21 bits.
+      And even for good old reliable ASCII, useless bits are sprinkled here and there.
+      There's gotta be a better way!
+    </p>
+    <h2>UTF-63</h2>
+    <p>
+      Enter: An entirely new way to encode Unicode!
+    </p>
+    <pre>1<span class="red2">000011111011001001000</span><span class="orange2">0000111110</span>  <span class="red2">🙈</span><span class="orange2">🙉</span>
+<span class="orange2">11001001001</span><span class="yellow2">000011111011001001010</span>  <span class="yellow2">🙊</span>
+0<span class="aqua2">0100000</span><span class="blue2">1100101</span><span class="purple2">1101110</span><span class="red2">1100011</span><span class="orange2">110</span>  <span class="aqua2"> </span><span class="blue2">e</span><span class="purple2">n</span><span class="red2">c</span><span class="orange2">o</span>
+<span class="orange2">1111</span><span class="yellow2">1100100</span><span class="aqua2">1101001</span><span class="blue2">1101110</span><span class="purple2">1100111</span>  <span class="yellow2">d</span><span class="aqua2">i</span><span class="blue2">n</span><span class="purple2">g</span></pre>
+    <p>
+      For our entirely random example, this UTF-63 encoding uses only 16 bytes, a clear win over the whopping 21 bytes required by UTF-8 (that's 30% more!).
+      So how does it work?
+    <p>
+    </p>
+      Decoding is dead simple:
+      We consider each set of 8 bytes a Big Endian 64-bit code unit.
+      If the most significant bit is 0, the other 63 bits contain nine Basic Latin characters (i.e. ASCII characters) at 7 bits each.
+      If it's 1, they contain any three Unicode code points at 21 bits each (the maximum codepoint is U+10FFFF).
+      When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways).
+    </p>
+    <h2>Usage</h2>
+    <p>
+      Admittedly, this page actually still uses UTF-8, simply because most modern browsers don't support UTF-63 yet.
+      To my knowledge, most OSes, desktop applications and programming languages don't either.
+      It wouldn't surprise me if VLC supported UTF-63 subtitles or something though.
+      You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between.
+    </p>
+  </article>
+</body>
+</html>
@@ -0,0 +1,83 @@
+:root {
+  /* https://github.com/morhetz/gruvbox */
+  --red: #cc241d;
+  --red2: #fb4934;
+  --orange: #d65d0e;
+  --orange2: #fe8019;
+  --yellow: #d79921;
+  --yellow2: #fabd2f;
+  --aqua: #689d6a;
+  --aqua2: #8ec07c;
+  --blue: #458588;
+  --blue2: #83a598;
+  --purple: #b16286;
+  --purple2: #d3869b;
+  --bg: #928374;
+  --fg0_h: #fbf1c7;
+  --bg0_h: #1d2021;
+}
+
+html {
+  font-size: 20px;
+}
+
+body {
+  color: var(--bg0_h);
+  background-color: var(--fg0_h);
+  font-family: 'Source Sans Pro';
+}
+
+address {
+  font-size: .8em;
+  font-style: normal;
+}
+
+h1, h2, h3, h4, h5, h6 {
+  font-weight: normal;
+}
+
+p {
+  line-height: 1.5;
+}
+
+article {
+  margin: 0 auto;
+  width: 80ch;
+}
+
+pre {
+  padding: 1em;
+  background-color: var(--bg0_h);
+  color: var(--fg0_h);
+  line-height: 1.3;
+}
+
+.red2 {
+  color: var(--bg0_h);
+  background-color: var(--red2);
+}
+
+.orange2 {
+  color: var(--bg0_h);
+  background-color: var(--orange2);
+}
+
+.yellow2 {
+  color: var(--bg0_h);
+  background-color: var(--yellow2);
+}
+
+.aqua2 {
+  color: var(--bg0_h);
+  background-color: var(--aqua2);
+}
+
+.blue2 {
+  color: var(--bg0_h);
+  background-color: var(--blue2);
+}
+
+.purple2 {
+  color: var(--bg0_h);
+  background-color: var(--purple2);
+}