Protecting from stack overflow in recursive functions.

Suppose you have some recursive function:

fn recursee(n: i64) -> Result<i64, ()> {
    if n > 0 {
        Ok(1+ try!(recursee(n-1)))
    } else {
        Ok(0)
    }
}

fn main() {    
    println!("{:?}", recursee(3));
    println!("{:?}", recursee(100));
    println!("{:?}", recursee(999999999));
    println!("{:?}", recursee(5000));
}

It works for small values (that you need). But attacker can supply special unbalanced bad data that cause stack overflow. Getting rid of recursion requires big redesign and regretting about committing the "Let's use clever recursion here" idea. What if we can just add a direct protection against stack overlow?

#![feature(asm)]

fn sp() -> usize {
    let sp : usize;
    unsafe {
        asm!("":"={esp}"(sp):::"volatile");
    }
    sp
}


static mut near_stack_top : usize = 0;

fn get_stack_gauge() -> f64 {

    let stack_size = if cfg!(target_env="musl") { 0x18000 } else { 0x800000 };

    let (s, st, ss) = (sp() as f64, unsafe { near_stack_top } as f64, stack_size as f64);

    (st - s) / ss
}


fn recursee(n: i64) -> Result<i64, ()> {
    if get_stack_gauge() > 0.95 {
        // stack overflow is nearing
        return Err(());
    }

    if n > 0 {
        Ok(1+ try!(recursee(n-1)))
    } else {
        Ok(0)
    }
}

fn main() {
    unsafe { near_stack_top = sp(); }

    println!("{:?}", recursee(3));
    println!("{:?}", recursee(100));
    println!("{:?}", recursee(999999999));
    println!("{:?}", recursee(5000));
}

Ok(3)
Ok(100)
Err(())
Ok(5000)

Maybe functions for being aware of how much stack is still available should be a bit closer to language, so there can be

fn get_stack_gauge() -> f64 {
    (::std::stack:remaining() as f64) / (::std::stack::size() as f64)
}

?

Alternative ideas:

• Special macro like try_recurse!(recursee(n)) that tries to check if there enough stack for calling and throws othewise;

No repository files or tests are named. Start by reviewing the recursive Rust example and the proposed stack gauge and try_recurse! alternatives; done would require a defined language-level mechanism that prevents recursive stack overflow without requiring callers to redesign the recursion.