snowflakedb/ArcticInference

View on GitHub

ArcticInference: vLLM plugin for high-throughput, low-latency inference

Stars: 485
Forks: 68
Open beginner issues: 0
Indexed issues: 48
Avg merge: 1d 23h
Merged PRs (30d): 4

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 8, 2026
Latest indexed: Sep 20, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

48 open issues indexed

How to save `SuffixDecodingCache` for future use? Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

snowflakedb/ArcticInference#261 · 1 comment · Opened Apr 27, 2026
[NPU Adaptation] Contribution of NPU Support Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

snowflakedb/ArcticInference#259 · Opened Apr 9, 2026
vllm0.18.0 Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

snowflakedb/ArcticInference#256 · 1 reaction · Opened Mar 23, 2026
是否支持SuffixDecoding + EAGLE-3 Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

snowflakedb/ArcticInference#255 · Opened Mar 16, 2026
[Question] Is the engine not supported on Blackwell architechture? Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

snowflakedb/ArcticInference#254 · Opened Mar 13, 2026
[Question]How to run arctic_inference on vllm 0.15.0? Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

snowflakedb/ArcticInference#253 · Opened Mar 10, 2026
Can not run suffix in vllm Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

snowflakedb/ArcticInference#252 · Opened Mar 4, 2026
RPC timeout error after 5 min hang with speculative config arctic-suffix=True Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#250 · 1 comment · Opened Feb 23, 2026
Install error on Ubuntu 20.04 due to C++20 <span> Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

snowflakedb/ArcticInference#248 · Opened Feb 10, 2026
Structured output error in parallel - v0.1.2 Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

snowflakedb/ArcticInference#246 · Opened Feb 6, 2026
Any plan to support suffix decoding for async scheduling? Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#243 · 3 comments · Opened Jan 28, 2026
[Q] Compatibility / Update to vLLM 0.13.0 and 0.14.0 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#240 · 1 comment · 2 reactions · Opened Jan 20, 2026
SwiftKV graph capture w/ Ulysses Open

bug SwiftKV

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#231 · Opened Dec 30, 2025
[feat] could provide a switch to enable the dynamic output of draft tokens during suffix decoding in vllm Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

snowflakedb/ArcticInference#230 · 3 comments · Opened Dec 16, 2025
Fast Serialization Open

Difficulty 1/5 Under an hour Newbie friendliness 15/100

snowflakedb/ArcticInference#227 · Opened Dec 3, 2025
What is the corresponding relationship between the vllm version and the ArcticInference version? Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

snowflakedb/ArcticInference#225 · 1 comment · Opened Nov 25, 2025
Structured output error in parallel Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#223 · 1 comment · Opened Nov 10, 2025
Reproduction question. Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

snowflakedb/ArcticInference#220 · Opened Nov 3, 2025
What is the difference between shift parallelism and vllm's decode context parallelism? Open

snowflakedb/ArcticInference#219 · 1 reaction · 1 assignee · Opened Oct 31, 2025
Swiftkv model degraded accuracy Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#217 · Opened Oct 29, 2025
Add `suffix_decoding_max_cached_tokens` to trigger eviction based on token count Open

Suffix Decoding

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#215 · Opened Oct 22, 2025
Add a batch-asynchronous version of suffix cache update Open

Suffix Decoding

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#205 · 11 comments · 1 reaction · Opened Oct 10, 2025
Add a batch-parallel version of suffix speculation Open

Suffix Decoding

Difficulty 5/5 Over a week Newbie friendliness 35/100

snowflakedb/ArcticInference#204 · 2 comments · Opened Oct 10, 2025
Starting Llama-3.1-SwiftKV-8B-Instruct arctic method or an error is reported Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

snowflakedb/ArcticInference#202 · 5 comments · Opened Oct 9, 2025
arctic 0.0.10 transformer 4.51.1 error indicates that an upgrade is required Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#197 · 4 comments · Opened Sep 29, 2025
Error using Speculator with vllm=0.10.0 for gpt-oss Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#190 · 6 comments · Opened Sep 22, 2025
suffix deocding cause performance degradation under high concurrency Open

Suffix Decoding

Difficulty 4/5 3-5 days Newbie friendliness 42/100

snowflakedb/ArcticInference#185 · 4 comments · Opened Sep 17, 2025
Question: lower-than-expected throughput and latency on benchmarks Open

Ulysses

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#184 · 10 comments · Opened Sep 17, 2025
Suffix Decoding compatibility with guided decoding Open

Suffix Decoding

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#183 · 1 comment · Opened Sep 16, 2025
Ulysses sequence parallelism crashes when --max-num-seqs is not divisible by sp_size Open

bug Ulysses

snowflakedb/ArcticInference#175 · 2 comments · 1 assignee · Opened Sep 12, 2025
Installation Issues Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#174 · 1 comment · Opened Sep 11, 2025
Modal container guide? Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

snowflakedb/ArcticInference#160 · 1 comment · Opened Aug 11, 2025
[USAGE] Failure to Load and run a vllm server for Model Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

snowflakedb/ArcticInference#159 · 1 comment · Opened Aug 11, 2025
Why do we see a performance boost for short sequences ? Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

snowflakedb/ArcticInference#155 · Opened Aug 4, 2025
kv cache manager bug Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

snowflakedb/ArcticInference#152 · 1 comment · Opened Jul 31, 2025
the benchmark between HTTP and gRPC Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

snowflakedb/ArcticInference#147 · Opened Jul 24, 2025
Release pre-compiled wheels Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

snowflakedb/ArcticInference#141 · 1 comment · Opened Jul 19, 2025
Support for TP>1 Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

snowflakedb/ArcticInference#136 · 9 comments · Opened Jul 10, 2025
No module named 'arctic_inference.common.suffix_cache._C' Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

snowflakedb/ArcticInference#135 · 1 comment · Opened Jul 9, 2025
Support for Reranker inference Open

snowflakedb/ArcticInference#130 · 1 assignee · Opened Jul 8, 2025
Installation issues Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#113 · 8 comments · 2 reactions · Opened Jun 18, 2025
Support for newer vLLM versions Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

snowflakedb/ArcticInference#111 · 7 comments · Opened Jun 18, 2025
Need a page in documentation for Dynasor Open

documentation

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

snowflakedb/ArcticInference#102 · Opened Jun 10, 2025
Instructions to apply to QWQ/ Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

snowflakedb/ArcticInference#96 · 1 comment · Opened Jun 6, 2025
Question about SGLang benchmark results Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

snowflakedb/ArcticInference#91 · 3 comments · Opened Jun 4, 2025
Add GitHub Actions CI for basic CPU tests Open

snowflakedb/ArcticInference#83 · 1 assignee · Opened May 30, 2025
Error checking for models compatible with shift-parallelism Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

snowflakedb/ArcticInference#81 · 1 comment · Opened May 30, 2025
Investigate JIT-compilation for embeddings gRPC Open

enhancement

snowflakedb/ArcticInference#80 · 1 comment · 1 assignee · Opened May 30, 2025

snowflakedb/ArcticInference

Get new issues in your inbox