Proposal: amd64 SystemTap Extension to allow patching in function calls in place?

Basically my thought is to make it possible to leave hooks for profilers (e.g. Tracy) in place and then use a specialized linker to patch in the tracy client library and turn the probes into calls into that library. While you could use the current interface to do profiling, the overhead from INT3 might invalidate any data you collect (although I haven't tested this out).

The current SystemTap probe can't be replaced with a function call because it only leaves a single byte of space in the instruction stream and function call instructions are larger than that. Unless you want to do something clever with recognizing and rewriting the instructions after it you'll need more space.

My proposal boils down to "add more NOPs". Originally I was thinking of using a five byte NOP (since amd64 has those), but I realized that you could just add more NOPs to make the change backwards compatible with anything that checks that the instruction actually is a NOP before overwriting it. 5 bytes because that is large enough to fit a E8 call procedure opcode + 32bit offset or an E9 jump opcode + 32bit offset.

This extension could add something to the ELF notes, but it could also rely on analyzing the code and finding multiple NOPs.

Anyway, mostly I wanted to get this idea out there and hear if anyone thinks this could be useful or if this is just a terrible idea.

Start by reviewing the current SystemTap probe interface and how the project emits or records probe instructions. Compare the proposed five-byte call or jump space with the existing single-byte probe behavior and investigate whether ELF notes or NOP analysis is the intended mechanism. Done would require a decided design, implementation scope, and validation for backward compatibility.