open on nonexistent file returns wrong error

[open](https://pubs.opengroup.org/onlinepubs/9799919799/functions/open.html) should return `ENOENT` if

> O_CREAT is not set and a component of *path* does not name an existing file, or O_CREAT is set and a component of the *path* prefix of *path* does not name an existing file, or *path* points to an empty string.

It should return `ENOENT` or `ENOTDIR` if

> O_CREAT is set, and the *path* argument contains at least one non- character and ends with one or more trailing characters. If *path* without the trailing characters would name an existing file, an `ENOENT` error shall not occur.

Currently, `open("/a")` returns `EINVAL`, if `/a` does not exist.