Skip to main content

split_tokens

Function split_tokens 

Source
pub fn split_tokens(text: &str) -> impl Iterator<Item = Part<'_>>
Expand description

Splits text into plain text and {tokens}.