UTF-63: Snd version
This commit is contained in:
+10
-8
@@ -15,11 +15,10 @@
|
|||||||
By <a rel="author" href="../..">Paul Brinkmeier</a>
|
By <a rel="author" href="../..">Paul Brinkmeier</a>
|
||||||
</address>
|
</address>
|
||||||
<p>
|
<p>
|
||||||
UTF-8 is almost everywhere these days.
|
<a href="https://www.unicode.org/versions/latest/core-spec/chapter-2/#G11165">UTF-8</a> is almost everywhere these days.
|
||||||
The web runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
|
The Internet runs on it, most common file formats use it, macOS, Linux and increasingly even Windows systems use it as the default option.
|
||||||
It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols.
|
It makes sense: It's quite compact for mostly-ASCII text and therefore most keywords and symbols of text-based file formats and protocols.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p>
|
<p>
|
||||||
What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage?
|
What if I told you that in this age of memory bandwidth-constrained workloads you are using a wasteful encoding for text storage?
|
||||||
Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up).
|
Most text these days is 99% ASCII with a few emoji sprinkled in (source: Made it up).
|
||||||
@@ -27,7 +26,7 @@
|
|||||||
Even if we ignore the costly multibyte emoji encodings, using UTF-8, 1 out of every 8 bits of memory is wasted on the encoding, because 7-bit ASCII characters are encoded using a whole byte, i.e. we have an encoding overhead of roughly 15%.
|
Even if we ignore the costly multibyte emoji encodings, using UTF-8, 1 out of every 8 bits of memory is wasted on the encoding, because 7-bit ASCII characters are encoded using a whole byte, i.e. we have an encoding overhead of roughly 15%.
|
||||||
</p>
|
</p>
|
||||||
<p>
|
<p>
|
||||||
For example, consider the UTF-8 encoding of the string “🙈🙉🙊 encoding” (see <a href="https://utoy.beany.club/text/🙈🙉🙊 encoding" title="also cURLable!">here</a> for codepoints):
|
For example, consider the UTF-8 encoding of the string “🙈🙉🙊 encoding” (see <a href="https://utoy.beany.club/text/🙈🙉🙊 encoding" title="also cURLable!">here</a> for code points):
|
||||||
</p>
|
</p>
|
||||||
<pre>11110<span class="red2">000</span>10<span class="red2">011111</span>10<span class="red2">011001</span>10<span class="red2">001000</span> <span class="red2">🙈</span>
|
<pre>11110<span class="red2">000</span>10<span class="red2">011111</span>10<span class="red2">011001</span>10<span class="red2">001000</span> <span class="red2">🙈</span>
|
||||||
11110<span class="orange2">000</span>10<span class="orange2">011111</span>10<span class="orange2">011001</span>10<span class="orange2">001001</span> <span class="orange2">🙉</span>
|
11110<span class="orange2">000</span>10<span class="orange2">011111</span>10<span class="orange2">011001</span>10<span class="orange2">001001</span> <span class="orange2">🙉</span>
|
||||||
@@ -43,9 +42,9 @@
|
|||||||
And even for good old reliable ASCII, useless bits are sprinkled here and there.
|
And even for good old reliable ASCII, useless bits are sprinkled here and there.
|
||||||
There's gotta be a better way!
|
There's gotta be a better way!
|
||||||
</p>
|
</p>
|
||||||
<h2>UTF-63</h2>
|
<h2>A Better Way</h2>
|
||||||
<p>
|
<p>
|
||||||
Enter: An entirely new way to encode Unicode!
|
Enter: UTF-63, an entirely new way to encode Unicode!
|
||||||
</p>
|
</p>
|
||||||
<pre>1<span class="red2">000011111011001001000</span><span class="orange2">0000111110</span> <span class="red2">🙈</span><span class="orange2">🙉</span>
|
<pre>1<span class="red2">000011111011001001000</span><span class="orange2">0000111110</span> <span class="red2">🙈</span><span class="orange2">🙉</span>
|
||||||
<span class="orange2">11001001001</span><span class="yellow2">000011111011001001010</span> <span class="yellow2">🙊</span>
|
<span class="orange2">11001001001</span><span class="yellow2">000011111011001001010</span> <span class="yellow2">🙊</span>
|
||||||
@@ -58,8 +57,8 @@
|
|||||||
<p>
|
<p>
|
||||||
Decoding is dead simple:
|
Decoding is dead simple:
|
||||||
We consider each set of 8 bytes a Big Endian 64-bit code unit.
|
We consider each set of 8 bytes a Big Endian 64-bit code unit.
|
||||||
If the most significant bit is 0, the other 63 bits contain nine Basic Latin characters (i.e. ASCII characters) at 7 bits each.
|
If the most significant bit is 0, the other 63 bits contain nine Basic Latin code points (ASCII characters) at 7 bits each.
|
||||||
If it's 1, they contain any three Unicode code points at 21 bits each (the maximum codepoint is U+10FFFF).
|
If it's 1, the other 63 bits contain any three Unicode code points at 21 bits each (the maximum code point is U+10FFFF).
|
||||||
When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways).
|
When any of the encoded code points is U+0000 NULL, the string ends there (yes, we're bringing null-terminated strings back. So. many. applications. don't allow NULL anyways).
|
||||||
</p>
|
</p>
|
||||||
<h2>Usage</h2>
|
<h2>Usage</h2>
|
||||||
@@ -69,6 +68,9 @@
|
|||||||
It wouldn't surprise me if VLC supported UTF-63 subtitles or something though.
|
It wouldn't surprise me if VLC supported UTF-63 subtitles or something though.
|
||||||
You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between.
|
You can do your part by spreading the word among friends, family, flatmates, coworkers and anybody in between.
|
||||||
</p>
|
</p>
|
||||||
|
<p>
|
||||||
|
If you enjoyed my Unicode ramblings you can read on over at <a href="https://www.channable.com/tech/so-long-surrogatesa">Channable</a>.
|
||||||
|
</p>
|
||||||
</article>
|
</article>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
Reference in New Issue
Block a user