mleku on Nostr: i have not looked deeply into it but the way Go does it is convert the variable ...
i have not looked deeply into it but the way Go does it is convert the variable length symbols into an array of 32 bit values (there is ~4bln points in the field) and you have to compute the byte length in UTF-8 encoding by the values of each element of the array...
i don't know about avoiding breaking up words, word boundaries are another thing again, as you know in some languages there isn't a notion of a "space" character but rather each symbol is an atomic unit, and as i'm fairly sure that they are maximum 4 bytes long encoded in most cases it's either you are splitting on a space or a symbol boundary, spaces are not considered as symbols except as far as the bytes they take up, which is 1, i think, just a hex 0x20
uniseg looks pretty much like a good choice. i should just point out that one of the Go Authors was one of the co-designers of the Unicode spec, Rob Pike, so Go's handling is especially good. It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this
most of it his handled already in javascript also, because internationalisation was very important to web systems from very early on, but server and desktop app languages have lived in a bubble of mostly english for a long time.
as regards to the truncation length decision, i think that visual distance should be what you are aiming for here, and for the most part, most distinctive language scripts have a fairly uniform relative width for the same symbols, so i think the right place to evaluate that question has to do with the typography side of the encodings, the byte length is probably not relevant at all
i don't know about avoiding breaking up words, word boundaries are another thing again, as you know in some languages there isn't a notion of a "space" character but rather each symbol is an atomic unit, and as i'm fairly sure that they are maximum 4 bytes long encoded in most cases it's either you are splitting on a space or a symbol boundary, spaces are not considered as symbols except as far as the bytes they take up, which is 1, i think, just a hex 0x20
uniseg looks pretty much like a good choice. i should just point out that one of the Go Authors was one of the co-designers of the Unicode spec, Rob Pike, so Go's handling is especially good. It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this
most of it his handled already in javascript also, because internationalisation was very important to web systems from very early on, but server and desktop app languages have lived in a bubble of mostly english for a long time.
as regards to the truncation length decision, i think that visual distance should be what you are aiming for here, and for the most part, most distinctive language scripts have a fairly uniform relative width for the same symbols, so i think the right place to evaluate that question has to do with the typography side of the encodings, the byte length is probably not relevant at all