paul/pbrinkmeier.de
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

UTF-63: Snd version

Browse Source
This commit is contained in:
Paul Brinkmeier
2026-08-10 11:19:52 +02:00
parent 32e5f72dab
commit 1df3566997
1 changed files with 10 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
blog/utf63/index.html
+9 -7
View File
@@ -15,11 +15,10 @@
By <a rel="author" href="../..">Paul Brinkmeier</a> By <a rel="author" href="../..">Paul Brinkmeier</a>
</address> </address>
<p> <p>
UTF-8 is almost everywhere these days. <a href="https://www.unicode.org/versions/latest/core-spec/chapter-2/#G11165">UTF-8</a> is almost everywhere these days.
The web runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option. The Internet runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols. It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols.
</p> </p>
<p> <p>
What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage? What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage?
Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up). Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up).
@@ -43,9 +42,9 @@
And even for good old reliable ASCII, useless bits are sprinkled here and there. And even for good old reliable ASCII, useless bits are sprinkled here and there.
There's gotta be a better way! There's gotta be a better way!
</p> </p>
<h2>UTF-63</h2> <h2>A Better Way</h2>
<p> <p>
Enter: An entirely new way to encode Unicode! Enter: UTF-63, an entirely new way to encode Unicode!
</p> </p>
<pre>1<span class="red2">000011111011001001000</span><span class="orange2">0000111110</span> <span class="red2">🙈</span><span class="orange2">🙉</span> <pre>1<span class="red2">000011111011001001000</span><span class="orange2">0000111110</span> <span class="red2">🙈</span><span class="orange2">🙉</span>
<span class="orange2">11001001001</span><span class="yellow2">000011111011001001010</span> <span class="yellow2">🙊</span> <span class="orange2">11001001001</span><span class="yellow2">000011111011001001010</span> <span class="yellow2">🙊</span>
@@ -58,8 +57,8 @@
<p> <p>
Decoding is dead simple: Decoding is dead simple:
We consider each set of 8 bytes a Big Endian 64-bit code unit. We consider each set of 8 bytes a Big Endian 64-bit code unit.
If the most significant bit is 0, the other 63 bits contain nine Basic Latin characters (i.e. ASCII characters) at 7 bits each. If the most significant bit is 0, the other 63 bits contain nine Basic Latin code points (ASCII characters) at 7 bits each.
If it's 1, they contain any three Unicode code points at 21 bits each (the maximum codepoint is U+10FFFF). If it's 1, the other 63 bits contain any three Unicode code points at 21 bits each (the maximum code point is U+10FFFF).
When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways). When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways).
</p> </p>
<h2>Usage</h2> <h2>Usage</h2>
@@ -69,6 +68,9 @@
It wouldn't surprise me if VLC supported UTF-63 subtitles or something though. It wouldn't surprise me if VLC supported UTF-63 subtitles or something though.
You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between. You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between.
</p> </p>
<p>
If you enjoyed my Unicode ramblings you can read on over at <a href="https://www.channable.com/tech/so-long-surrogatesa">Channable</a>.
</p>
</article> </article>
</body> </body>
</html> </html>