This does relate to a question I've been thinking about for a while,
so even if actually offering diffs for that is still way above my pay
grade, I will offer these thoughts:
* Of ASCII's 128 characters, only 95 are actually printable (ASCII
sticks 2 thru 7 minus 0x7F DEL).[0]
* In principle, the console is capable of supporting 256 glyphs.
* With traditional Extended ASCII (EASCII) character sets, more than
95 characters were (still are) printable, but code assuming the use of
ISO 8859-1 is deprecated and no longer portable in this age of UTF-8,
and for EASCII sticks 8 thru F, there no longer is a direct
correspondence between code points and code units at all.
* Even if framebuffer console drivers could hypothetically be altered
to allow the use of more than 256 glyphs, I completely agree with Ingo
that that would be a fairly terrible idea for various reasons. While
the 256-glyphs limitation does stem from VGA console drivers
permitting no more than 256 text mode glyphs (or 512 with hacks), it
would be best to not totally break framebuffer and vga console
compatibility, but to stay within those limits.
* With "extremely minimalistic UTF-8 support", up to 161 "spots" might
be available.
* There are 1,112,064 legal Unicode character code points (0x11 *
0x10000 - 0x800, i.e. seventeen 65,536-character planes minus the
2,048 code points from U+D800 thru U+DFFF that are reserved for UTF-16
surrogates). Of those, 137,468 are private use, and 66 are
non-characters. If we also subtract the 95 printable ASCII
characters, that leaves 974,435 characters that might compete for
those 161 spots.
* There is an extremely strong argument for accommodating all
characters from ISO 8859-1 in any future minimalistic UTF-8 console
support. The non-breaking space and soft hyphen could use the same
glyphs as space and hyphen-minus, respectively. This means that to
maintain maximum backwards compatibility and UTF-8
forward-portability, 94 of those 161 spots would have to be taken,
leaving 67.
* There might also be a strong argument for accommodating all the
characters from ISO 8859-15 (so an additional 8) and Windows-1252,[1]
which despite no Unix pedigree is a common superset of ISO 8859-1,
with EASCII sticks A thru F being identical to ISO 8859-1. ISO
8859-15 differs from ISO 8859-1 in that it includes 8 characters in
sticks A/B that Windows-1252 encodes in sticks 8/9. However, with
UTF-8, code units and code points no longer match outside of sticks
0-7, so UTF-8 implementers of ISO 8859-1 and Windows-1252 backwards
compatibility get ISO 8859-15 support for free. Besides those 8,
Windows-1252 support would consume an additional 19 characters, so
we'd have to subtract 27 from those 27 remaining spots, leaving 40.
* 32 of those spots are from the C0 control codes from ASCII sticks
0/1. While Bemer et al. did originally propose alternatively
printable glyphs for those normally unprintable characters, their
glyphs were never commonly used. If "maximum printability" is a
criterion, Unicode does define so-called "Control Pictures" for them
(U+2400 thru U+241F).[2] It conceivably could be useful to have e.g.
a console-based hex editor render something printable for most code
units, however attempts at Control Picture inclusion would bump
against the technical limitation that the Control Pictures glyphs are
already barely legible in X11/xterm: So could actually useful Control
Pictures glyphs even be defined if one has just 8x16, 8x14, 8x10 or
8x8 pixels to play with, as may be the case on the console? It seems
doubtful. Perhaps those sparse spots and precious pixels are better
spent on something else, like Cyrillic for example.
* The once-common DOS code page 437 has 31 alternatively printable
glyphs for sticks 0/1. Of these, only the bullet point, section sign
and paragraph mark (pilcrow) can be found in the ISO 8859/Windows-1252
family. There is no compelling reason --like what's mentioned in
footnote [1]-- that could motivate the inclusion of its stick 0/1
glyphs, DOS having largely gone the way of the dodo. Also, full CP437
support would require many more glyphs, support for just this subset
of that old code page, never common in Unix-land, would seem wasteful.
That still leaves 40 spots that could potentially be used.
* The question is, which of the 974,435 candidates deserve one of
those 40 spots. With a look at a relevant map[3], Arabic, Cyrillic,
and Indic abugidas might have particularly strong claims. Arabic has
28 letters, but many contextual variants (though no case), Cyrillic,
or more specifically the Russian alphabet has 33 letters and it does
have case, so 40 spots might limit any support to UPPER CASE ONLY, or
should I say ЦРРЕЯ СА5Е ОИГУ. I do not feel I know enough about Indic
abugidas to say something intelligent.
* The question of what subset of Unicode to settle on for minimalistic
256-glyphs-only UTF-8 support might be bigger than OpenBSD. Other
Unix-like OSes might ask themselves the same question. Is this
something that ought to be standardised across Unix-land or something
OpenBSD would want to decide on its own?
* I mentioned "512 with hacks" above, but I do not know enough if it
could be viable, clean and VGA-compatible to blow past that 256
boundary. If yes, then an additional 256 spots might comfortably
allow for the inclusion of many more of the above.
* Either way, even if no code is created at this time, just having a
roadmap and knowing which glyphs ought to make the cut might be
desirable. It would also be possible to already make the font(s) once
that is known. Code that actually uses such a font to implement
minimalistic UTF-8 support (for the console) need not arrive at the
same time.
* On the other hand, if extending our minimum character set to cover
Windows-1252 and ISO 8859-15 and especially deciding upon the use of
the last 40 spots cannot be settled yet, then it might be fine to
leave that for later. The existing ISO 8859-1 fonts could actually be
useable by a minimalistic UTF-8 support implementation, if developed.
Again, once such an implementation has code points properly divorced
from code units, it could absolutely source its glyphs from those
fonts.
That would only leave the small issue of UTF-8 compliance by
everything else in base and ports...
I hope that was useful and worth the verbiage.
Thanks for your time,
Ian
(Ian Ropers)
Footnotes:
[0] Yes, they're properly called sticks. 8 sticks of 16 characters in
ASCII; 16 sticks of 16 characters in EASCII. See Bob Bemer's Inside
ASCII.
[1] Per enwp.org/CP1252, Windows-1252 text mislabelled as ISO-8859-1
is still very common (online), and "[m]ost modern web browsers and
e-mail clients treat the media type charset ISO-8859-1 as Windows-1252
to accommodate such mislabeling. This is now standard behavior in the
HTML5 specification, which requires that documents advertised as
ISO-8859-1 actually be parsed with the Windows-1252 encoding."
[2] https://en.wikipedia.org/wiki/Control_Pictures
[3] https://en.wikipedia.org/wiki/File:Writing_systems_worldwide.png
On 05/10/2021, Ingo Schwarze <schwarze@usta.de> wrote:
> Hi Slava,
>
> Slava Voronzoff wrote on Tue, Oct 05, 2021 at 03:01:26PM +0300:
>
>> I'm working right now on adding cyrillic to Spleen font. How can I later
>> add it to OpenBSD kernel and ports? Pull request to main font on github
>> (Hi, Frederic) or patch here?
>
> You cannot add it to the kernel because the kernel does not support
> UTF-8, but only US-ASCII, and US-ASCII contains no code points for
> cyrillic letters.
>
> Full UTF-8 support is definitely not wanted in the kernel. Adding
> extremely minimalistic UTF-8 support to the kernel is not completely
> out of the question, but some developers are likely to feel sceptic even
> about that. Consequently, trying to pursue a project of adding anything
> related to UTF-8 to the kernel is likely to end in frustration if the
> person trying that does not have a significant amount of experience with
> getting OpenBSD kernel patches committed.
>
> I'm sorry that i know absolutely nothing about fonts in ports, maybe
> someone else can answer that part of the question.
>
> Yours,
> Ingo
>
>
No comments:
Post a Comment