By: gallier2 (gallier2.delete@this.gmx.de),
Room: Moderated Discussions
rwessel (robertwessel.delete@this.yahoo.com) on December 12, 2019 9:17 am wrote:
> Jose (1.delete@this.2.com) on December 12, 2019 2:19 am wrote:
> > anonymou5 (no.delete@this.spam.com) on December 11, 2019 2:09 pm wrote:
> > > > > And SCAS can be used for strnlen...
> > > > >
> > > > > So, if Intel/AMD would just add the instructions that combine
> > > > > SCAS & MOVS (SCAMOVS) and SCAS & CMPS (SCACMPS)
> > > > > you have accelerators for strcpy and strcmp as well (which probably is used just as much as copy/clear).
> > > > >
> > > >
> > > > May be, 25 years ago.
> > > > Today ASCII text is not a dominant data type.
> > >
> > > NUL-terminated strings are still a thing though.
> > Yes, but strlen of an UTF-8 encoded string will not necessary return the number of characters in the string.
> >
>
> But if you're copying one, that *is* the size you need.
In Unicode character is not a valid concept anyway. It's codeunit, codepoint or grapheme and then one has also to define the encoding: UTF-8, UTF-16 or UTF-32 (let's forget about other exotic encodings like UTF-EBCDIC). A normal program needs generally to know the length in codeunits as it is the one necessary to know the size of the buffer, that's the strlen() in UTF-8.
UTF-8 is between 1 and 4 8-bit codeunits per codepoint
UTF-16 is 1 or 2 16-bit codeunits
UTF-32 is 1 32-bit codeunnit per codepoint
but this does still not give the information on a length in "character" as a grapheme can, and often does, consist of several codepoints.
à̴̵̶̷̸̡̢̧̨̛̖̗̘̙̜̝̞̟̠̣̤̥̦̩̪̫̬̭̮̯̯̰̱̲̳̹̺̻̼͇͈͉͍͎́̂̃̄̅̆̇̈̉̊̋̌̍̎̏̐̑̒̓̔̽̾̿̀́͂̓̈́͆͊͋͌̕̚͠͡ͅ
This above is 1 grapheme "character" (most renderers fail to display it properly as it is not something useful). It is composed of 83 codepoints. So it takes 165 bytes in UTF-8, 166 in UTF-16, 332 in UTF-32 but it is one character.
> Jose (1.delete@this.2.com) on December 12, 2019 2:19 am wrote:
> > anonymou5 (no.delete@this.spam.com) on December 11, 2019 2:09 pm wrote:
> > > > > And SCAS can be used for strnlen...
> > > > >
> > > > > So, if Intel/AMD would just add the instructions that combine
> > > > > SCAS & MOVS (SCAMOVS) and SCAS & CMPS (SCACMPS)
> > > > > you have accelerators for strcpy and strcmp as well (which probably is used just as much as copy/clear).
> > > > >
> > > >
> > > > May be, 25 years ago.
> > > > Today ASCII text is not a dominant data type.
> > >
> > > NUL-terminated strings are still a thing though.
> > Yes, but strlen of an UTF-8 encoded string will not necessary return the number of characters in the string.
> >
>
> But if you're copying one, that *is* the size you need.
In Unicode character is not a valid concept anyway. It's codeunit, codepoint or grapheme and then one has also to define the encoding: UTF-8, UTF-16 or UTF-32 (let's forget about other exotic encodings like UTF-EBCDIC). A normal program needs generally to know the length in codeunits as it is the one necessary to know the size of the buffer, that's the strlen() in UTF-8.
UTF-8 is between 1 and 4 8-bit codeunits per codepoint
UTF-16 is 1 or 2 16-bit codeunits
UTF-32 is 1 32-bit codeunnit per codepoint
but this does still not give the information on a length in "character" as a grapheme can, and often does, consist of several codepoints.
à̴̵̶̷̸̡̢̧̨̛̖̗̘̙̜̝̞̟̠̣̤̥̦̩̪̫̬̭̮̯̯̰̱̲̳̹̺̻̼͇͈͉͍͎́̂̃̄̅̆̇̈̉̊̋̌̍̎̏̐̑̒̓̔̽̾̿̀́͂̓̈́͆͊͋͌̕̚͠͡ͅ
This above is 1 grapheme "character" (most renderers fail to display it properly as it is not something useful). It is composed of 83 codepoints. So it takes 165 bytes in UTF-8, 166 in UTF-16, 332 in UTF-32 but it is one character.