From 1df3566997535ab563b5fa76724a8c6c55b96483 Mon Sep 17 00:00:00 2001
From: Paul Brinkmeier
- UTF-8 is almost everywhere these days.
- The web runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
+ UTF-8 is almost everywhere these days.
+ The Internet runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols.
What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage?
Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up).
@@ -27,7 +26,7 @@
Even if we ignore the costly multibyte emoji encodings, using UTF-8, 1 out of every 8 bits of memory is wasted on the encoding, because 7-bit ASCII characters are encoded using a whole byte, i.e. we have an encoding overhead of roughly 15%.
- For example, consider the UTF-8 encoding of the string βπππ encodingβ (see here for codepoints):
+ For example, consider the UTF-8 encoding of the string βπππ encodingβ (see here for code points):
11110000100111111001100110001000 π
11110000100111111001100110001001 π
@@ -43,9 +42,9 @@
And even for good old reliable ASCII, useless bits are sprinkled here and there.
There's gotta be a better way!
- Enter: An entirely new way to encode Unicode! + Enter: UTF-63, an entirely new way to encode Unicode!
10000111110110010010000000111110 ππ 11001001001000011111011001001010 π @@ -58,8 +57,8 @@Decoding is dead simple: We consider each set of 8 bytes a Big Endian 64-bit code unit. - If the most significant bit is 0, the other 63 bits contain nine Basic Latin characters (i.e. ASCII characters) at 7 bits each. - If it's 1, they contain any three Unicode code points at 21 bits each (the maximum codepoint is U+10FFFF). + If the most significant bit is 0, the other 63 bits contain nine Basic Latin code points (ASCII characters) at 7 bits each. + If it's 1, the other 63 bits contain any three Unicode code points at 21 bits each (the maximum code point is U+10FFFF). When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways).
Usage
@@ -69,6 +68,9 @@ It wouldn't surprise me if VLC supported UTF-63 subtitles or something though. You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between. ++ If you enjoyed my Unicode ramblings you can read on over at Channable. +