org.opensolaris.opengrok.analysis
Class JFlexTokenizer

java.lang.Object
  extended by org.apache.lucene.analysis.TokenStream
      extended by org.apache.lucene.analysis.Tokenizer
          extended by org.opensolaris.opengrok.analysis.JFlexTokenizer
Direct Known Subclasses:
CSymbolTokenizer, CxxSymbolTokenizer, FortranSymbolTokenizer, JavaSymbolTokenizer, LispSymbolTokenizer, PlainFullTokenizer, PlainSymbolTokenizer, ShSymbolTokenizer, TclSymbolTokenizer, TroffFullTokenizer

public abstract class JFlexTokenizer
extends org.apache.lucene.analysis.Tokenizer

this class was created because of lucene 2.4.1 update which introduced char[] in Tokens instead of String Created on August 24, 2009


Field Summary
protected static org.apache.lucene.analysis.Token reuseToken
           
 
Fields inherited from class org.apache.lucene.analysis.Tokenizer
input
 
Constructor Summary
JFlexTokenizer()
           
 
Method Summary
 org.apache.lucene.analysis.Token next(org.apache.lucene.analysis.Token preusableToken)
          This is a convenience method for having correctly generated classes who reuse Tokens and save gc for lucene summarizer you MUST consume the returned token to properly get the null value !
abstract  org.apache.lucene.analysis.Token yylex()
           
 
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, reset
 
Methods inherited from class org.apache.lucene.analysis.TokenStream
next, reset
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

reuseToken

protected static final org.apache.lucene.analysis.Token reuseToken
Constructor Detail

JFlexTokenizer

public JFlexTokenizer()
Method Detail

yylex

public abstract org.apache.lucene.analysis.Token yylex()
                                                throws java.io.IOException
Throws:
java.io.IOException

next

public final org.apache.lucene.analysis.Token next(org.apache.lucene.analysis.Token preusableToken)
                                            throws java.io.IOException
This is a convenience method for having correctly generated classes who reuse Tokens and save gc for lucene summarizer you MUST consume the returned token to properly get the null value !

Overrides:
next in class org.apache.lucene.analysis.TokenStream
Parameters:
preusableToken -
Returns:
null if no more tokens, otherwise a pointer to the modified token
Throws:
java.io.IOException