Disable EQUS expansion for raw symbols (by lexing them as strings)

[Rangi42]

Fixes #1605

The goal here is to make #raw string symbols get treated like string literals/expressions instead of getting EQUS-expanded. The long-term goal is to implement that in a way compatible with removing EQUS expansion for all symbols (#905).

This approach works by just adding a new SYM_ID token type, and lexing raw identifiers which would otherwise have been EQUS-expanded as SYM_ID tokens instead. Then the parser has to treat this token type as a string in the right contexts. So now, in addition to the relocexpr_no_str distinction, it needs a scoped_anon_id_no_str distinction.

This PR needs further comparison with the alternative parser-based approach implemented in #1648.

[Rangi42]

What feels natural to me about this:

• Lack of repetition. The lexer does one check for whether a symbol is defined as a string or not, and returns different tokens depending on that.
• Fewer lines of code than the alternative.

The downsides:

• Reminiscent of the "lexer hack" in C, where an identifier is lexed differently depending on its category from previous context.
• The parser has a redundant error check for sym && sym->type == SYM_EQUS, even though the lexer whould never return a STR_SYMBOL if that condition were false.
  • This Discord comment has my reasoning why alternatives are worse, although I could be persuaded that assume(condition); do stuff; is better than if (condition) { do stuff; } else { error; }.
• Affects error messages, which now refer to "string symbol"s separately from "symbol"s.

[Rangi42]

Re: comparisons to the C lexer hack, Wikipedia claims:

Clang's lexer does not attempt to differentiate between type names and variable names: it simply reports the current token as an identifier. The parser then uses Clang's semantic analysis library to determine the nature of the identifier. This allows a simpler and more maintainable architecture than The Lexer Hack. This is also the approach used in most other modern languages, which do not distinguish different classes of identifiers in the lexical grammar, but instead defer them to the parsing or semantic analysis phase, when sufficient information is available.

This "modern" approach sounds similar to the one used in the other PR, #1648.

[ISSOtm]

The spirit of this PR looks good to me, I think I only have bikeshedding-level changes to request. Thanks!

[ISSOtm]

I think this should be more explicit that “raw symbol” means “prefixed with a hash”.

I'm also realising that “raw identifier” is a better name, since it might not be referring to a symbol yet? (And identifiers are names for symbols. I'd be fine with calling those “raw names”, if that sounds better than “identifier”.)

[ISSOtm]

In the vein of my previous comment:

Suggested change

	Expansion is also disabled for raw string constant symbols (string constant symbols prefixed by a hash
	.Sq # ) .
	Also, expansion is
	.Sy not
	performed if the string symbol's name is preceded by a hash
	.Sq # .

[ISSOtm]

Hmm... this sounds confusing. “Why is it expecting a symbol or a symbol?”

[Rangi42]

Yeah, this is one reason why I prefer #1648 as well.