Brian Kernighan adds Unicode support to Awk (github.com)

Article summary

Brian Kernighan has added Unicode support to Awk. The implementation uses UTF-32, which is a fixed-size encoding for Unicode code points. This choice is unexpected, as many systems choose to work entirely in UTF-8. The addition of Unicode support is a significant update to the Awk programming language.

Main themes

• Unicode support in Awk
• UTF-32 vs UTF-8
• Character encoding
• Regular expression matching
• Grapheme clusters
• Unicode standards compliance

What commenters say

• The choice of UTF-32 over UTF-8 may have performance implications, with some arguing that UTF-8 is faster for typical cases.
• UTF-32 allows for simple math operations on code points, but this may not be useful in practice due to Unicode's variable-length encoding.
• Some commenters argue that Unicode support requires implementing specific features, such as grapheme cluster matching, while others claim that this is not necessary for basic Unicode support.
• The use of UTF-32 may simplify certain operations, such as regular expression matching, but may also increase memory usage.
• There is disagreement over what constitutes a 'character' in Unicode, with some arguing that it refers to a grapheme cluster and others claiming it refers to a code point.
• The implementation of Unicode support in Awk may not be fully compliant with Unicode standards, and may not support all features, such as \X for grapheme cluster matching.