Calling fopen for writing ends up calling __isatty which sets errno - is it intended?

Hi everyone,

Running the following bit of code in a WASM module built for Microsoft Flight Simulator (which uses wasi_libc - thanks a lot for your hard work!):

	printf("errno 1 - %d - %s\n", errno, strerror(errno));
	FILE* hFile = fopen("\\work\\testerrno.txt", "w");
	printf("errno 2 - %d - %s\n", errno, strerror(errno));
	fclose(hFile);
	printf("errno 3 - %d - %s\n", errno, strerror(errno));

I noticed that errno is set to 59 (E_NOTTY) after the call to fopen. One important observation is that it does NOT happen when opening a file for reading.

Now, running a similar bit of code (only the path changed) in a Windows Console Application, errno remains at zero all along.

Browsing through the wasi_libc source code, I noticed that fopen calls __fdopen which itself calls __isatty for files opened with "write" rights:

	/* Activate line buffered mode for terminals */
	f->lbf = EOF;
#ifdef __wasilibc_unmodified_upstream // WASI has no syscall
	if (!(f->flags & F_NOWR) && !__syscall(SYS_ioctl, fd, TIOCGWINSZ, &wsz))
#else
	if (!(f->flags & F_NOWR) && __isatty(fd))
#endif
		f->lbf = '\n';

The issue is that __isatty always sets errno to E_NOTTY if the file descriptor is not that of a TTY, which is the case when opening a regular file.

The only thing I am wondering is whether this behavior is intended or not. If that's the case then we can turn errno to zero in our runtime when fopen returns E_NOTTY (which doesn't really mean much in our case anyway). If not, maybe errno should be saved & restored manually in this bit of code?

Best regards,

Eric / Asobo

Start with fopen, __fdopen, and __isatty in wasi-libc, comparing the write and read paths shown in the report. Check the existing errno behavior for regular files and terminal detection; done means the intended errno semantics are documented or covered by a regression test.