htcc-0.0.0.1: The full scratch implementation of tiny C compiler (x86_64)
Copyright(c) roki 2019
LicenseMIT
Maintainerfalgon53@yahoo.co.jp
Stabilityexperimental
PortabilityPOSIX
Safe HaskellNone
LanguageHaskell2010

Htcc.Tokenizer

Description

The tokenizer

Synopsis

Documentation

module Htcc.Tokenizer.Token

tokenize :: (Integral i, Read i, Show i) => Text -> Either (ASTError i) [TokenLC i] Source #

Tokenize the Text. If an invalid chraracter matches as C language, the part and the character are returned. Otherwise, [TokenIdx i] is returned.