starfuzz/normalize

A module providing text normalization capabilities. Normalization is typically the first step before calculating text similarities, ensuring formatting differences (casing, spaces, punctuation) do not distort scores.

Types

Normalizer

</>

An immutable normalizer configuration.

pub type Normalizer {
  Normalizer(
    lowercase: Bool,
    trim: Bool,
    collapse_whitespace: Bool,
    remove_punctuation: Bool,
    replace_punctuation_with_space: Bool,
  )
}

Constructors

Normalizer(
  lowercase: Bool,
  trim: Bool,
  collapse_whitespace: Bool,
  remove_punctuation: Bool,
  replace_punctuation_with_space: Bool,
)

Values

apply

</>

pub fn apply(normalizer: Normalizer, input: String) -> String

Applies the normalizer pipeline to the input string.

basic

</>

pub fn basic(input: String) -> String

A preset performing lowercase mapping, trimming, punctuation stripping, and collapsing spaces.

new

</>

pub fn new() -> Normalizer

Creates a new default normalizer with all options disabled.

search_text

</>

pub fn search_text(input: String) -> String

A preset performing lowercase mapping, trimming, replacing punctuation with spaces, and collapsing spaces.

with_collapsed_whitespace

</>

pub fn with_collapsed_whitespace(
  normalizer: Normalizer,
) -> Normalizer

Configures the normalizer to collapse multiple consecutive whitespaces into a single space.

with_lowercase

</>

pub fn with_lowercase(normalizer: Normalizer) -> Normalizer

Configures the normalizer to convert all text to lowercase.

with_punctuation_as_space

</>

pub fn with_punctuation_as_space(
  normalizer: Normalizer,
) -> Normalizer

Configures the normalizer to replace punctuation marks with spaces.

with_trim

</>

pub fn with_trim(normalizer: Normalizer) -> Normalizer

Configures the normalizer to trim leading and trailing whitespaces.

without_punctuation

</>

pub fn without_punctuation(normalizer: Normalizer) -> Normalizer

Configures the normalizer to strip all punctuation marks from the text.