rust-lang / rust-lang/rfcs

Need calling convention for stdcall methods (different from functions!)

Open
#1,342 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

T-dev-tools T-lang
Dominant language
Markdown
Stars
6.6k
Forks
1.7k
Avg merge
16h 14m
Merged PRs (30d)
1

Description

The crux of the matter is that stdcall methods on Windows using msvc handle struct returns differently than stdcall functions. This is particularly noticeable in COM, where although most of COM doesn't use struct returns, a few of them do, causing incorrect behavior when you attempt to call them from Rust (or even C for the matter).

Thus I propose that we add a new calling convention specifically for stdcall methods. When faced with this new calling convention Rust should do what Clang does:

(2:07:35 PM) rnk: right, this is sret vs. this
(2:07:47 PM) rnk: clang handles this by swapping the order of the arguments
(2:07:54 PM) rnk: and moving the sret attribute to the second parameter

Relevant discussion in #LLVM https://gist.github.com/retep998/604861ea06aa984ee6c7
Relevant discussion on discourse https://internals.rust-lang.org/t/need-custom-calling-convention-for-com/2389

gist demonstrating the calling convention difference: https://gist.github.com/retep998/9503145841a61551d3c6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with issue #1342 and the linked LLVM discussion, discourse thread, and gists; compare the proposed stdcall-method behavior with existing Rust calling-convention handling. Define the RFC's precise scope and acceptance criteria for COM and struct-return cases, since the payload names no Rust files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, rust
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.