CommunityToolkit / CommunityToolkit/dotnet
Support tokenize span using more than 1 separator
- Dominant language
- C#
- Stars
- 3.8k
- Forks
- 400
- PR merge metrics
- No merged PRs in 30d
Description
### Overview
Sometimes a [ReadOnly]Span needs to be tokenized using more than one separator.
### API breakdown
```csharp
namespace CommunityToolkit.HighPerformance;
public static class SpanExtensions
{
public static SpanTokenizer2 Tokenize(this Span span, T separator0, T separator1);
public static SpanTokenizer3 Tokenize(this Span span, T separator0, T separator1, T separator2);
public static SpanTokenizerAny Tokenize(this Span span, ReadOnlySpan separators)
}
```
```csharp
namespace CommunityToolkit.HighPerformance;
public static class ReadOnlySpanExtensions
{
public static ReadOnlySpanTokenizer2 Tokenize(this ReadOnlySpan span, T separator0, T separator1);
public static ReadOnlySpanTokenizer3 Tokenize(this ReadOnlySpan span, T separator0, T separator1, T separator2);
public static ReadOnlySpanTokenizerAny Tokenize(this ReadOnlySpan span, ReadOnlySpan separators)
}
```
### Usage example
```csharp
ReadOnlySpan content = "John; 1960, USA";
foreach (var token in content.Tokenize(';', ',')
{
Console.WriteLine(token.ToString());
}
```
### Breaking change?
No
### Alternatives
Not that I'm aware of.
### Additional context
The new tokenizer structs can use `IndexOfAny` instead of `IndexOf`.
### Help us help you
Yes, I'd like to be assigned to work on this item
Contributor guide
Assessment
This issue has not been assessed yet.