You are looking at documentation for an older release.
Not what you want? See the
current release documentation.
Lowercase Tokenizeredit
A tokenizer of type lowercase that performs the function of
Letter Tokenizer and
Lower Case Token Filter together. It divides text at non-letters and converts
them to lower case. While it is functionally equivalent to the
combination of
Letter Tokenizer and
Lower Case Token Filter, there is a performance advantage to doing the two
tasks at once, hence this (redundant) implementation.