CommunityToolkit / CommunityToolkit/dotnet

Support tokenize span using more than 1 separator

Open
#807 4 comments 0 reactions 0 assignees View on GitHub
feature request :mailbox_with_mail:
Dominant language
C#
Stars
3.8k
Forks
400
PR merge metrics
No merged PRs in 30d

Description

### Overview

Sometimes a [ReadOnly]Span needs to be tokenized using more than one separator.

### API breakdown

```csharp
namespace CommunityToolkit.HighPerformance;

public static class SpanExtensions
{
public static SpanTokenizer2 Tokenize(this Span span, T separator0, T separator1);
public static SpanTokenizer3 Tokenize(this Span span, T separator0, T separator1, T separator2);
public static SpanTokenizerAny Tokenize(this Span span, ReadOnlySpan separators)
}
```

```csharp
namespace CommunityToolkit.HighPerformance;

public static class ReadOnlySpanExtensions
{
public static ReadOnlySpanTokenizer2 Tokenize(this ReadOnlySpan span, T separator0, T separator1);
public static ReadOnlySpanTokenizer3 Tokenize(this ReadOnlySpan span, T separator0, T separator1, T separator2);
public static ReadOnlySpanTokenizerAny Tokenize(this ReadOnlySpan span, ReadOnlySpan separators)
}
```

### Usage example

```csharp
ReadOnlySpan content = "John; 1960, USA";

foreach (var token in content.Tokenize(';', ',')
{
Console.WriteLine(token.ToString());
}
```

### Breaking change?

No

### Alternatives

Not that I'm aware of.

### Additional context

The new tokenizer structs can use `IndexOfAny` instead of `IndexOf`.

### Help us help you

Yes, I'd like to be assigned to work on this item

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.