Re: UTF8 locale MFC for DragonflyBSD

[Dave Cuthbert <dacut@xxxxxxxxxxxxx>]

Joerg Sonnenberger wrote:
> On Sun, Mar 28, 2004 at 09:35:58PM -0500, David Cuthbert wrote:
> > 1. Representation of embedded NUL characters.  The UTF-8 spec says this 
> > is one byte == 00000000b; Java and a few others, though, have used a 
> > double-byte encoding (110/00000 + 10/000000) so that stuff like strlen() 
> > works "reasonably."
>
> This is a clear violation of the minimum requirement. Since you should
> always use explicitly sized strings instead of delimited strings for
> processing of strings with possible embedded NULLs, I don't think we want
> this. Actually this is one of the view checks we might want to do in the
> kernel if we want to go the UTF-8 road in the future. It would be pretty
> bad to be able to create undelete / unviewable files :)

Agreed.  Actually, I thought that this was a fairly elegant hack. 
Nonetheless, it is still a hack.

> I don't really want to force us to UCS2, just because MS did. It is pretty
> pointless if you think about Unicode as mean to encode every _written_
> script in the world. Therefore if we want to apply any length checks, the
> correct way is as specified by at least Unicode 3 e.g. UCS4.

Well, not just MS; a lot of folks (notably Sun/Java) were caught off 
guard when Unicode was extended beyond the base 64k characters.  I won't 
replicate the flame wars here, they're all on Google. :-)

My personal opinion: UCS-4 wastes a lot of space given that Unicode 3.1 
is a ~21-bit set and nobody is really using the >=U+10000 space in a 
practical manner (yet?).  But if you need to have a one-to-one mapping, 
you don't have much choice.

Unless you have a machine which uses 21-bit bytes, of course. ;-)