starfuzz/tokenize

A module providing tokenization capabilities. Tokenization splits raw text into smaller components such as characters, words, or n-grams before similarity metrics or vector transformations.

Types

TokenizeError

</>

Represents tokenization errors.

pub type TokenizeError {
  InvalidN(Int)
}

Constructors

```
InvalidN(Int)
```
Returned when the n-gram size n is invalid (e.g. less than or equal to 0).

Values

character_ngrams

</>

pub fn character_ngrams(
  text: String,
  n: Int,
) -> Result(List(String), TokenizeError)

Generates character n-grams of size n from the input text.

characters

</>

pub fn characters(text: String) -> List(String)

Splits the string into its constituent Unicode grapheme characters.

word_ngrams

</>

pub fn word_ngrams(
  text: String,
  n: Int,
) -> Result(List(List(String)), TokenizeError)

Generates word n-grams of size n from the input text.

words

</>

pub fn words(text: String) -> List(String)

Splits the string into words, using whitespace characters as boundaries.