Struct Lexer

pub struct Lexer<'lex, 'src> { /* private fields */ }

Implementations

impl<'lex, 'src> Lexer<'lex, 'src>

pub fn new( text: &'src str, base_offset: usize, em: &'lex mut ErrorManager, ) -> Self

Builds a new Lexer

Arguments

• text: Source code to tokenize
• em: An ErrorManager, where all the errors will be sent

pub fn into_token_stream(self) -> TokenStream<'lex, 'src>

Turns this Lexer into a TokenStream

pub fn next_token(&mut self) -> Option<Token>

Parses the next token

Returns None if the EOF was reached

Trait Implementations

impl<'lex, 'src> IntoIterator for Lexer<'lex, 'src>

fn into_iter(self) -> Self::IntoIter

Converts this Lexer into an iterator of tokens

This is the same as calling Self::into_token_stream

type Item = Token

The type of the elements being iterated over.

type IntoIter = TokenStream<'lex, 'src>

Which kind of iterator are we turning this into?