Contains information about a tokenization process.
uint DocumentIndexOffset
Document Index Offset. Used to identify sequences of tokens in a document.
Tokenizes a set of human readable texts.
async Task Tokenize(object Value, TokenizationProcess Process)
Tokenizes an object.
Interface for full-text-search tokenizers