kmy.jint.parser
Class AbstractTokenizer

java.lang.Object
  |
  +--kmy.jint.parser.AbstractTokenizer

Direct Known Subclasses:

ReaderTokenizer

public abstract class AbstractTokenizer

extends java.lang.Object

implements TokenConst, ErrorCodes

This class represents a stream of Java/Jint tokens. Method nextToken() can be used to fill in next token's tType and possibly other fields. Method close() frees resources used by this stream.

Field Summary

double	dVal Float/double token value.
int	filePos Integer that represents token position inside the stream.
Identifier	ident Identifier token value.
long	lVal Integer/long token value.
NameTable	nameTable Field that stores a NameTable that was passed to constructor.
java.lang.String	sVal String token value.
java.lang.String	sVal2 Additional String token value.
int	tType Token type.

Fields inherited from interface kmy.jint.constants.TokenConst

TOK_OPT_BUFFERONLY, TOK_OPT_DECLARE, TOK_OPT_GLOBAL, TOK_OPT_IGNORE, TOK_OPT_MULTILINE, TOK_OPT_OFFLINE, TOK_OPT_RETAINALL, TOK_OPT_SINGLELINE, TOK_OPT_STANDALONE, TOKEN_ABSTRACT, TOKEN_AE, TOKEN_AND, TOKEN_ANDE, TOKEN_BOOLEAN, TOKEN_BREAK, TOKEN_BYTE, TOKEN_BYVALUE, TOKEN_CASE, TOKEN_CATCH, TOKEN_CHAR, TOKEN_CLASS, TOKEN_CONST, TOKEN_CONTINUE, TOKEN_DEFAULT, TOKEN_DO, TOKEN_DOUBLE, TOKEN_EE, TOKEN_ELSE, TOKEN_EOF, TOKEN_EXTENDS, TOKEN_FALSE, TOKEN_FINAL, TOKEN_FINALLY, TOKEN_FLOAT, TOKEN_FOR, TOKEN_GE, TOKEN_GG, TOKEN_GGE, TOKEN_GGG, TOKEN_GGGE, TOKEN_GOTO, TOKEN_IF, TOKEN_IMPLEMENTS, TOKEN_IMPORT, TOKEN_INSTANCEOF, TOKEN_INT, TOKEN_INTERFACE, TOKEN_LE, TOKEN_LL, TOKEN_LLE, TOKEN_LONG, TOKEN_ME, TOKEN_MM, TOKEN_NATIVE, TOKEN_NE, TOKEN_NEW, TOKEN_NONE, TOKEN_NULL, TOKEN_OR, TOKEN_ORE, TOKEN_PACKAGE, TOKEN_PCE, TOKEN_PE, TOKEN_PP, TOKEN_PRIVATE, TOKEN_PROTECTED, TOKEN_PUBLIC, TOKEN_RETURN, TOKEN_SE, TOKEN_SHORT, TOKEN_SS, TOKEN_STATIC, TOKEN_SUPER, TOKEN_SWITCH, TOKEN_SYNCHRONIZED, TOKEN_TE, TOKEN_THIS, TOKEN_THREADSAFE, TOKEN_THROW, TOKEN_THROWS, TOKEN_TRANSIENT, TOKEN_TRUE, TOKEN_TRY, TOKEN_V_BOUND, TOKEN_V_CHAR, TOKEN_V_DOUBLE, TOKEN_V_FLOAT, TOKEN_V_FORM, TOKEN_V_INT, TOKEN_V_LONG, TOKEN_V_PATT, TOKEN_V_REGEX, TOKEN_V_STRING, TOKEN_V_SUBST, TOKEN_V_TRANS, TOKEN_VOID, TOKEN_VOLATILE, TOKEN_WHERE, TOKEN_WHILE, TOKEN_WORD, TOKEN_XE, UNARY_MINUS

Fields inherited from interface kmy.jint.constants.ErrorCodes

ERR_A_ARREMPTY, ERR_A_ARRNOTEMPTY, ERR_A_BADDCL, ERR_A_BADMETHODNAME, ERR_A_BADOP, ERR_A_BADSWITCHTYPE, ERR_A_BOOLCAST, ERR_A_CANTACCLABEL, ERR_A_CANTCAST, ERR_A_CANTMAKEARR, ERR_A_CANTRESOLVE, ERR_A_CONSTERROR, ERR_A_EXPLCONTEXT, ERR_A_FIELDNOTIDENT, ERR_A_FIELDNOTSTAT, ERR_A_IMPLICITDCL, ERR_A_INSTOFPRIM1, ERR_A_INSTOFPRIM2, ERR_A_INTERNAL, ERR_A_LABELREDEF, ERR_A_MATCHARG1, ERR_A_MATCHARG2, ERR_A_MUSTBEBOOL, ERR_A_NEVER, ERR_A_NEWARRARGS, ERR_A_NOBESTMATCH, ERR_A_NOCONSTRUCTOR, ERR_A_NOFIELD, ERR_A_NOMETHOD, ERR_A_NONSTATICACC, ERR_A_NOOUTERCLASS, ERR_A_NOTCAUGHT, ERR_A_NOTINDEXABLE, ERR_A_NOTINTARGS, ERR_A_NOTINTEGER, ERR_A_NOTLVALUE, ERR_A_NOTNUMBER, ERR_A_NOTSIMPLEID, ERR_A_NOTTHROWABLE, ERR_A_NOTTHROWN, ERR_A_NOVAR, ERR_A_OBJTOPRIMCAST, ERR_A_PARAMUNRES, ERR_A_PRIMSYNC, ERR_A_PRIMTOOBJCAST, ERR_A_REGEXTE, ERR_A_STATICTHIS, ERR_A_TOBOOLCAST, ERR_A_TOVOIDCAST, ERR_A_TYPEMISMATCH, ERR_A_TYPENOTFOUND, ERR_A_VARREDECL, ERR_A_VOIDCAST, ERR_C_BADOP, ERR_C_CANTASSIGN, ERR_C_CASEMISMATCH, ERR_C_CVTNULLTOPRIM, ERR_C_DUPCASE, ERR_C_GENERIC, ERR_C_INTERFCONSTR, ERR_C_INTERNAL, ERR_C_MATCHARG1, ERR_C_NESTING, ERR_C_NEVER, ERR_C_NOCNSTRWOPRM, ERR_C_NOENCLOSING, ERR_C_NONSTATINIT, ERR_C_NOTBOOL, ERR_C_PICKEDNONSTR, ERR_C_PRIMELEM, ERR_C_RETURNREQ, ERR_C_THISASSIGN, ERR_C_TWODEFAULT, ERR_L_BACKSLASHEOF, ERR_L_CHAREND, ERR_L_COMMENTEND, ERR_L_EMPTYCHAR, ERR_L_EOFINCHAR, ERR_L_EOFINREGEX, ERR_L_EOFINSTRING, ERR_L_ILLEGALCHAR, ERR_L_IOERROR, ERR_L_NEVER, ERR_L_NUMFORMAT, ERR_L_REGEXEND, ERR_L_STRINGEND, ERR_P_ARRNOTEMPTY, ERR_P_CASEEXPECTED, ERR_P_CATCHEXPECTED, ERR_P_CLASSKEYWORD, ERR_P_DUPEXTENDS, ERR_P_DUPIMPLEMENTS, ERR_P_IDENTEXPECTED, ERR_P_NOTIMPL, ERR_P_NOTSTANDALONE, ERR_P_OPEXPECTED, ERR_P_REGEXEXPECTED, ERR_P_TOKENEXPECTED, ERR_P_TOKENINSERTED, ERR_P_TYPEEXPECTED, ERR_P_UNEXPECTED, ERR_P_VALUEEXPECTED, ERR_P_VAREXPECTED, ERR_T_ARRNOTEMPTY, ERR_T_CANTRESOLVE, ERR_T_IDEXPECTED, ERR_T_INTERNAL, ERR_T_MULTIINHERIT, ERR_T_NOTLIST, ERR_T_NOTSIMPLEID, ERR_T_ONFINISH

Fields inherited from interface kmy.jint.constants.MiniErrorCodes

ERR_F_BADEND, ERR_F_BADFORMATCHAR, ERR_F_GENERIC, ERR_F_INCOMPLETE, ERR_F_NOCURLBRACKET, ERR_F_NOTWITHJAVA, ERR_F_NOVARNAME, ERR_MASK_CLASSTABLE, ERR_MASK_COMPILER, ERR_MASK_ERROR, ERR_MASK_FATAL, ERR_MASK_FORMAT, ERR_MASK_GENERATOR, ERR_MASK_INFO, ERR_MASK_INTERNAL, ERR_MASK_LEXER, ERR_MASK_MODULE, ERR_MASK_PARSER, ERR_MASK_REGEX, ERR_MASK_SEVERITY, ERR_MASK_TYPER, ERR_MASK_VERBOSE, ERR_MASK_WARNING, ERR_R_BADBACKREF, ERR_R_BADSTART, ERR_R_BQBAD, ERR_R_BQUNFINISHED, ERR_R_CTLINVALID, ERR_R_CTLUNFINISHED, ERR_R_EXTRABRACKET, ERR_R_GENERIC, ERR_R_HEXBADNUMBER, ERR_R_HEXUNFINISHED, ERR_R_INTERNAL, ERR_R_NOBRACKET, ERR_R_NOCURLBRACKET, ERR_R_NOSQBRACKET, ERR_R_NOVAREXPR, ERR_R_NOVARNAME, ERR_R_STRAYBSLASH

Constructor Summary

protected

AbstractTokenizer(NameTable nameTable) Create and initialize an AbstractTokenizer instance.

Method Summary

abstract void	close() Frees resources held by this stream.
static int	getColumn(int fpos) Extracts column number in a stream given the stream position.
static int	getLine(int fpos) Extracts line number in a stream given the stream position.
static int	makeFilePos(int line, int col) Creates representation of stream position based on line and column values.
abstract void	nextToken() Reads new token from the stream.

Methods inherited from class java.lang.Object

, clone, equals, finalize, getClass, hashCode, notify, notifyAll, registerNatives, toString, wait, wait, wait

Field Detail

tType

public int tType

Token type. Values are defined in TokenConst interface as TOKEN_*. Also valid values are single-character operator signs, such as +, -, etc. - they ase represented by character numerical values. Negative values represent keywords and identifiers

lVal

public long lVal

Integer/long token value. TOKEN_V_LONG, TOKEN_V_INT, TOKEN_V_CHAR return their numerical value in this field.

dVal

public double dVal

Float/double token value. TOKEN_V_FLOAT, TOKEN_V_DOUBLE return their numerical value in this field.

sVal

public java.lang.String sVal

String token value. TOKEN_STRING, TOKEN_REGEX, TOKEN_TRANS, TOKEN_PATT, TOKEN_FORM, TOKEN_SUBST, and TOKEN_BOUND set this field.

sVal2

public java.lang.String sVal2

Additional String token value. TOKEN_TRANS and TOKEN_SUBST set this field.

ident

public Identifier ident

Identifier token value. If tType is negative, this field contains corresponding Identifier object.

filePos

public int filePos

Integer that represents token position inside the stream. It is not byte count! Use getLine, getColumn, and makeFilePos to work with stream positions.

nameTable

public NameTable nameTable

Field that stores a NameTable that was passed to constructor. NameTable holds all Identifiers that this stream returns and they are inserted in the table as needed.

Constructor Detail

AbstractTokenizer

protected AbstractTokenizer(NameTable nameTable)

Create and initialize an AbstractTokenizer instance.

Method Detail

nextToken

public abstract void nextToken()

Reads new token from the stream. Stores token type into tType field and possibly initializes other fields (depending on tType value).

close

public abstract void close()

Frees resources held by this stream. Method nextToken() must not be called after call to close.

makeFilePos

public static int makeFilePos(int line,
                              int col)

Creates representation of stream position based on line and column values. If column value is more then a 1000 it might not get represented properly.

getLine

public static int getLine(int fpos)

Extracts line number in a stream given the stream position.

getColumn

public static int getColumn(int fpos)

Extracts column number in a stream given the stream position.