UTF-63
+ + + · + By Paul Brinkmeier + ++ UTF-8 is almost everywhere these days. + The web runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option. + It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols. +
+ ++ What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage? + Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up). + + Even if we ignore the costly multibyte emoji encodings, using UTF-8, 1 out of every 8 bits of memory is wasted on the encoding, because 7-bit ASCII characters are encoded using a whole byte, i.e. we have an encoding overhead of roughly 15%. +
++ For example, consider the UTF-8 encoding of the string “🙈🙉🙊 encoding” (see here for codepoints): +
+11110000100111111001100110001000 🙈 +11110000100111111001100110001001 🙉 +11110000100111111001100110001010 🙊 +00100000011001010110111001100011 enc +01101111011001000110100101101110 odin +01100111 g+
+ Colored spans show which code points are encoded by which bits. + Neutral (white on black) bits are encoding overhead required by UTF-8. + For emoji, the most important medium of modern communication, this encoding is obviously horrible. + Every monkey requires a full 32 bits to encode its 21 bits. + And even for good old reliable ASCII, useless bits are sprinkled here and there. + There's gotta be a better way! +
+UTF-63
++ Enter: An entirely new way to encode Unicode! +
+10000111110110010010000000111110 🙈🙉 +11001001001000011111011001001010 🙊 +00100000110010111011101100011110 enco +11111100100110100111011101100111 ding+
+ For our entirely random example, this UTF-63 encoding uses only 16 bytes, a clear win over the whopping 21 bytes required by UTF-8 (that's 30% more!). + So how does it work? +
+
+ Decoding is dead simple: + We consider each set of 8 bytes a Big Endian 64-bit code unit. + If the most significant bit is 0, the other 63 bits contain nine Basic Latin characters (i.e. ASCII characters) at 7 bits each. + If it's 1, they contain any three Unicode code points at 21 bits each (the maximum codepoint is U+10FFFF). + When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways). + +Usage
++ Admittedly, this page actually still uses UTF-8, simply because most modern browsers don't support UTF-63 yet. + To my knowledge, most OSes, desktop applications and programming languages don't either. + It wouldn't surprise me if VLC supported UTF-63 subtitles or something though. + You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between. +
+