[C++ PATCH] simplify deferred parsing lexer

[C++ PATCH] simplify deferred parsing lexer

https://gcc.gnu.org/ml/gcc-patches/2019-10/msg01962.html

We use an eof_token global variable as a sentinel on a deferred parse
(such as in-class function definitions, or default args). This
complicates retrieving the next token in certain places.

As such deferred parses always nest properly and completely before
resuming the outer lexer, we can simply morph the token after the
deferred buffer into a CPP_EOF token and restore it afterwards. I
finally got around to implementing it with this patch.

One complication is that we have to change the discriminator for when
the token's value is a tree. We can't look at the token's type because
it might have been overwritten. I add a bool flag to the token
(there's several spare bits), and use that. This does simplify the
discriminator because we just check a single bit, rather than a set of
token types.

	* parser.h (struct cp_token): Drop {ENUM,BOOL}_BITFIELD C-ism.
	Add tree_check_p flag, use as nested union discriminator.
	(struct cp_lexer): Add saved_type & saved_keyword fields.
	* parser.c (eof_token): Delete.
	(cp_lexer_new_main): Always init last_token to last token of
	buffer.
	(cp_lexer_new_from_tokens): Overlay EOF token at end of range.
	(cp_lexer_destroy): Restore token under the EOF.
	(cp_lexer_previous_token_position): No check for eof_token here.
	(cp_lexer_get_preprocessor_token): Clear tree_check_p.
	(cp_lexer_peek_nth_token): Check CPP_EOF not eof_token.
	(cp_lexer_consume_token): Assert not CPP_EOF, no check for
	eof_token.
	(cp_lexer_purge_token): Likewise.
	(cp_lexer_purge_tokens_after): No check for EOF token.
	(cp_parser_nested_name_specifier, cp_parser_decltype)
	(cp_parser_template_id): Set tree_check_p.

From-SVN: r277514