Tokens

Tokens represent a part of a String from a formal language input. They are used for easier handling later on in the Parser. As we do not need to worry about selecting just the right part of our string in that task.

__init__(id, ident='', emotion=None)

Creates a Token object.

Parameters:

Name	Type	Description	Default
id	Id	The ID of the token	required
ident	str	The text of the identifier, if the token is an identifier token.	''
emotion	Emotion | None	The emotion contained within the token, if the token is supposed to showcase an emotion.	None

get_keyword()

Creates a new Token object corresponding to the keyword, if possible.

Returns:

Name	Type	Description
None	None	If the current token is not an identifier
Token	Token	The new Token object with added attributes, if the Token is an identifier. The Token.emotion parameter is set, if the identifier corresponds to an Emotion The Token.ident parameter is cleared, if the identifier is a Keyword.

ident_to_keyword()

Converts an identifier into an emotion or a keyword, if possible.

Beware

This mutates the attributes of the current Token, if the Token is a keyword.

Raises:

Type	Description
Exception	If the current Token is not an identifier.

Bases: Enum

The current ID list for the lexer. Types the token can be. The IDs can be split into various types.

Keywords:

Name	Type	Description
BEGIN	Id	The Token that should be at the beggining of a text.
END	Id	The Token that should be at the end of a text.
EMOTION	Id	A Token that should have Token.emotion set to a corresponding Emotion.
MOVE	Id	This Token signals a Move-Command.

Symbols:

Name	Type	Description
LITERAL	Id	This Token type is fetched from reading ".
SEMICOLON	Id	This Token type is fetched from reading ;.
LCHEVRON	Id	This Token type is fetched from reading <.
RCHEVRON	Id	This Token type is fetched from reading >.
LSQBRACKET	Id	This Token type is fetched from reading [.
RSQBRACKET	Id	This Token type is fetched from reading ].

Other:

Name	Type	Description
IDENT	Id	This Token contains a general text area that could not be resolved into a Keyword or an Emotion. Token.ident should contain the text for this Token.
EOF	Id	This Token type should be returned if the end of the input string is reached.