messense / messense/fasttext-serving

One allocator on musl and aarch64 too

Open
#24 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
62
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### The value of integrating

fasttext-serving runs jemalloc on a narrow slice of its targets: `src/main.rs` gates the `#[global_allocator]` on `#[cfg(all(unix, not(target_env = "musl"), not(target_arch = "aarch64")))]`. So a musl container, an aarch64 host and any Windows build all fall back to the platform allocator — and the tree also carries `mimalloc = { version = "0.1" }`, meaning there are two allocator crates in the manifest and a majority of modern deployment targets using neither.

For a model-serving process that gap is on the hot path. The service stays up and answers predictions over HTTP (`actix-web`) or gRPC (`tonic = "0.8.0"`), both enabled by `default = ["http", "grpc"]`, on a multi-threaded runtime — so every request allocates input and prediction buffers that are typically freed on a different worker than the one that made them, continuously, for the life of the container.

rusty_alloc is one allocator on all of those targets, with 9,261 downloads on crates.io, in safe Rust with no build script and no `target_env` or `target_arch` conditions to keep in sync. Against jemalloc the rusty_alloc README's callgrind arms measure 0.84x, 0.89x and 0.98x whole-program instructions on lua, perl and sqlite, under `LD_PRELOAD`.

Not asking you to drop jemalloc where it is enabled — this would sit behind an off-by-default feature.

### How to integrate

Following the pattern you already use for `http` and `grpc`:

```toml
[features]
rusty-alloc = ["dep:rusty_alloc-api"] # opt in; `default = ["http", "grpc"]` unchanged

[dependencies]
rusty_alloc-api = { version = "2.1", optional = true }
```

```rust
#[cfg(feature = "rusty-alloc")]
#[global_allocator]
static GLOBAL: rusty_alloc_api::RustyAlloc = rusty_alloc_api::RustyAlloc;
```

Nothing else changes: same REST and gRPC surface, same model loading. The interesting comparison is an aarch64 or musl deployment, since those have no allocator choice at all today.

### Remade With Rust & MATA

We rebuild C dependencies in safe Rust — allocators, codecs, parsers, compression, identity and storage — so a project can take C out of its build without giving up performance. The rest is at https://github.com/Remade-With-Rust. If this does not suit you, please close the issue.

- x.com/farmer_timmm

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the allocator cfg in src/main.rs and the dependency and feature definitions in the manifest. Check how the existing jemalloc and mimalloc setup is selected, then verify the new opt-in feature builds for musl and aarch64 while the default http and grpc features and model-serving surfaces remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.