ICU stuff

[mr-martian]

ICU changes (closes #81)

• replace all instances of std::wstring with UString (= std::basic_string<UChar>)
• create InputFile wrapper to handle UTF-8 streams with nulls

efficiency, readability, and code style changes

• eliminate Ltstr and string_to_wostream
• simplify Makefile
• make transducer symbols int32_t rather than int
• make common symbols static attributes of Transducer
• extract some other string constants
• prefer std::vector to std::list
• prefer .clear() and .empty() to = "" and == ""
• prefer range-for loops
• remove old lsx code
• have regex_compiler iterate over the input string rather than modifying it
• lift a static computation out of a loop in Transducer::determinize()
• move constant initializers to class header

helper function and dependency changes

• move StringUtils here from apertium
• depend on external utfcpp rather than bundling it
• make XMLParseUtil functions more specific to their typical usecases
• add xml_walk_util.h for cleanly iterating over children of xmlNode*

[mr-martian]

Do we actually need that whole m4 script? Can we just ask pkg-config about icu-io directly?

[TinoDidriksen]

Do we actually need that whole m4 script? Can we just ask pkg-config about icu-io directly?

Agreed. Just do it like https://github.com/apertium/lexd/blob/master/configure.ac#L19-L20

And I see /home/daniel/lttoolbox/lttoolbox/nft.nrm in that diff.

[TinoDidriksen]

Also, I don't think a normalization tool belongs in lttoolbox - that's something we probably want to adjust separately, so a repo of its own would be nice.

[TinoDidriksen]

So far it looks like UnicodeString is suitable, but it will be interesting to see benchmarks. In CG-3 I use typedef std::basic_string<UChar> UString; for most strings, because it has a nicer interface and is movable.

[TinoDidriksen]

Future optimization: flat_set or sorted_vector

[TinoDidriksen]

LGTM

[TinoDidriksen]

Future work: None of our codebases should have .at().