On the state of the file system APIs

Nobody has claimed this yet.

One of the earliest issues opened in this repo was an issue around improved StorageFile APIs - https://github.com/microsoft/WindowsAppSDK/issues/8. I believe that the ultimate conclusion was let's do nothing, and let developers use existing .Net or Win32 APIs, given that the AppContainer model (and thus access permission issues) seems to have been dropped.

This was for the most part not a bad solution, but it still needs some work. In this very length issue I would like to highlight some of the problems. This is based on the experience of trying to migrate quite a large app from using the Windows.Storage.IStorageItem types to using file paths (or a thin custom class wrapping a file path), along with .Net and Win32 APIs.

The main thing I would like to highlight is that if we have dropped StorageFile and friends, then all APIs that accept StorageFile etc need alternatives that accept paths. This may need to be provided at system level or could be polyfilled by WindowsAppSDK, possibly using private/undocumented APIs - see the section 'Issues with APIs which only accept IStorageItem'.

I've also linked to some Gists so that people can see what code is needed to work around some of these issues (note the Gists do not always include all supporting code).

### A recap - what is wrong with StorageFile and related APIs?
The biggest problem with StorageFile and related APIs is that getting a StorageFile is slow and memory-intensive. To list a folder containing 2500 files takes 1-2ms using .Net or Win32 APIs (e.g. Directory.GetFiles), but takes 800ms using StorageFolder.GetFiles() (around 500 times slower). Furthermore, each StorageFile takes 50-100kB of memory (probably around 1000 times more than just the file path), and more if one uses the Properties on the StorageFile, so reading 10000 files just to get the file path (e.g. a large folder of photos) could consume 1GB of RAM.

The API surface itself is not bad, except StorageFile.GetBasicPropertiesAsync() is very inefficient for retrieving the modified date or file size. Due to the fact that this includes ItemDate, which may require image decoding to retrieve, this can be extremely slow (up to 100ms if the codec DLL needs loading), compared to using a .Net or Win32 API to retrieve the modified date, which takes a fraction of a millisecond.

Finally, none of the methods on StorageFile such as opening a stream support long file paths, which seems very backwards. Also, they do not support symbolic links (although this is more of an advanced scenario).

In conclusion, StorageFile is not suitable for use in serious apps.

### Could the problems with StorageFile be overcome?
I think a lot of the performance problems with StorageFile come from the fact that is based on the shell, and creating an IShellItem is slow, so I don't know if the performance could be improved. Of course one could ditch the shell, but then it might be difficult to support virtual files (more on that later). So perhaps StorageFile etc is best relegated to history.

### Issues with APIs which only accept or provide IStorageItem
Given that the implicit conclusion of https://github.com/microsoft/WindowsAppSDK/issues/8 was to drop StorageFile/IStorageItem, all APIs that accept them need an alternative involving file paths. This is especially important given that the most obvious place (from a user experience perspective) to pay the cost of getting the StorageFile instances is when the user first selects the files/folders. To go from a list of file paths to a list of StorageFile instances later on is very expensive (note that calling StorageFile.GetFileFromPathAsync() for a list of paths is probably another 10 times slower than StorageFolder.GetFiles()).

#### Here is a list of all the APIs that I could not find an alterative for
- `StorageFile.GetThumbnailAsync` - I tried to use IShellItemImageFactory instead but it was slower. It was still necessary to use that to support long paths, so here is the code if anyone wants - https://gist.github.com/benstevens48/ce22a1d991e7e8a5aa256fc3b44a10a4 (based on Win32 sample for how to save a HBITMAP). I notice that the Photos app gets its own private API [IThumbnailStreamCache](https://learn.microsoft.com/en-us/windows/win32/api/thumbnailstreamcache/nn-thumbnailstreamcache-ithumbnailstreamcache). Maybe we could get access to something similar (I was able to get this API to work but had no idea how cacheId was used - I just used a value of 1).
- `MediaClip.CreateFromFileAsync` - not such a big deal because probably won't create thousands of these, but why not support creating from a path?
- `RandomAccessStreamReference.CreateFromFile` - note there is `RandomAccessStreamReference.CreateFromUri` but it doesn't accept general file paths for some reason - it should do (and make sure these can be long paths)! Also, note that creating a custom `IRandomAccessStreamReference` may be possible but `DataPackage` APIs require `RandomAccessStreamReference` rather than `IRandomAccessStreamReference` for some reason.
- `UserProfilePersonalizationSettings` methods for setting wallpaper etc - not a massive issue.
- `FolderLauncherOptions.ItemsToSelect` - this was one where I actually had to remove functionality from my app. Previously one could select 1000s of files and the launch File Explorer with them selected. Now I have limited it to 1 due to the cost of creating the IStorageItems.
- `MediaSource.CreateFromUri` doesn't support long paths. Edit: actually, creating from URI also seems to keep an open file handle so one can't modify video metadata such as "System.Title" until the media source is disposed. A workaround is to use `MediaSource.CreateFromStreamReference` and implement `IRandomAccessStreamReference`. In order to do this I used the function `CreateRandomAccessStreamOnFile` - please document that it can actually output `IRandomAccessStreamWithContentType`, which makes implementing `IRandomAccessStreamReference` much easier. See this Gist - https://gist.github.com/benstevens48/e67a552519b9c617f5acec9888289a3f.

#### `DataPackage` and `DataPackageView`
It is especially problematic to create storage files from a list of paths on starting drag, since the operation really needs to complete in a few ms (and it turns out that delay adding storage items isn't really possible since apps like file explorer seem to request the storage item data on hover).

Here is a workaround for `DataPackage` and `DataPackageView` that allows setting file paths instead of storage items. The classic HDROP format is the best way to copy/drag and drop file paths. It turns out one can use interop on `DataPackage` and `DataPackageView` for this. `DataPackage` supports the COM API [IDataObjectProvider](https://learn.microsoft.com/en-us/windows/win32/api/shobjidl_core/nn-shobjidl_core-idataobjectprovider), which allows setting the file paths in an HDROP format. It turns out that `DataPackage` and `DataPackageView` both implement the COM interface IDataObject, which allows retrieving the paths, although this is not documented - **please document it**. See this Gist - https://gist.github.com/benstevens48/4b27d7e8cf2415812b07442843af8267.

However, note that due to wither a bug or by design, `DataPackageView.GetStorageItemsAsync` will only retrieve the first 16 storage items if they are set via file path (HDROP). **I hope this can be fixed, or that the use of `DataPackageView.GetStorageItemsAsync` can be discouraged**, with an API for getting/setting file paths provided. I think it's reasonable to expect apps to work with the HDROP format which has been the standard going back decades.

#### FileActivatedEventArgs.Files
The fact that this uses StorageFiles, means that if one selects 1000s of files in a File Explorer folder, right-clicks and presses open, the app takes a few seconds to open, when presumable if just the file paths were passed it would be quite instant.

#### Neighboring files
The `FileActivatedEventArgs.NeighboringFilesQuery` has been problematic since its inception. On newly created folders, it frequently only returns a partial list of files. I've not been able to determine a reason, but it's happened on each PC I've had since 2015 and I've had multiple user reports, and I've reported it as a bug in the past, and it affects built-in apps like Photos, but, incredibly, it's never been fixed. The speed is also slower than using .Net/Win32 APIs to list a folder, but it's not too bad so the main issue is this bug. Now that I have a desktop app without file access permission issues, I can load the whole folder based on the provided `FileActivatedEventArgs.Files[0]` (except for virtual files - see later section). However, one nice thing about `FileActivatedEventArgs.NeighboringFilesQuery` is that it matches File Explorer's sort order and some of my users rely on this. Miraculously, it turns out that there is a horrible hack, that I'm now using in my app, that allows you to get hold of the currently active File Explorer folder view and read the contents of it as a workaround. See this Gist, based on some code found on the Old New Thing blog - https://gist.github.com/benstevens48/4ad53578e5090ef4d870c3f4ec0f1ce9 (note not all supporting files are included). Note that you need to run it before activating your window. Again, this is kind of a horrible hack, but it does seem more reliable and quicker than `FileActivatedEventArgs.NeighboringFilesQuery`.

#### Getting the canonical path of a file/folder
One nice thing about StorageFile/Folder is that the path appears to be canonicalized (always the 'long' path and always cased as it is on disk). It turns out there is no way to get this using .Net (a surprise and oversight in my opinion). Here is a Gist showing how to get it with Win32 APIs - https://gist.github.com/benstevens48/89f890513143e76bce1ca536042ead49. I like to use file path to identify files so have been using this method on potentially unnormalized paths (note that names from file system enumeration like FindNextFileW are already normalized).

### Virtual files/folders
One of the main advantages of StorageFile over using file paths is that StorageFile supports virtual files without a path. However, this is very rare. In my case, then only time this occurs is if the user connects their phone to their PC with a USB cable and enables file transfer, then double-clicks on a photo, which launches my app with a NeighboringFilesQuery consisting on virtual files. Already there are pain points here. Although the NeighboringFilesQuery is virtual, the file clicked on is copied by Explorer to a local internet cache folder, with the suffix [1] appended to it's name, and then than is given as FileActivatedEventArgs.Files[0] to the app. So trying to match the file in the NeighboringFilesQuery doesn't work. What I have to do here, is firstly detect the scenario by assuming that a file in the internet cache folder with a NeighboringFilesQuery present is a virtual file, and get its name by stripping the [n] suffix. Then to get the index in the query to start adding files from, I use a variant of the hack described above to read the file explorer folder view. Note that in the case of virtual files I decided not to create my own custom wrapper around shell items, but just to use StorageFile, since it's less work. Also, note that random access (rather than sequential) to these files is slow, so copying to a local file or a memory stream first is must faster when reading these files, and also only one read operation (on any file returned by the NeighboringFilesQuery from a phone using media transfer protocol) is allowed at a time, which is very awkward.

Given how awkward dealing with these virtual files is, I think it would be better if they were handled in a different way at the level of the operating system. We already have to partial file technology used to support cloud services like OneDrive. I propose that all virtual files could be handled in this way so that the app can just deal with file paths without any special handling. The only thing is it would be useful to know that file operations on these files are likely to be slow, so maybe that could be indicated by a special path prefix for such virtual files.

I would like to emphasise how much easier it is for all files and folders to be represented by canonical file paths, which can be used to compare files quickly, added to sets/maps etc and serialized easily.

### Conclusion
I hope some of the issues mentioned above can be addressed, either in WindowsAppSDK or at the OS level. However, if at the OS level then changes may need to be serviced to lower versions in order to be useful. I am happy for StorageFile to become obsolete (especially now that I've spent a couple of months migrating away from it!), but I still see new APIs being added to the OS that use it! If a decision has been made not to improve the performance of StorageFile, then all new APIs (and ideally existing ones) need to have file path alternatives to StorageFile. Also note these should support paths longer than MAX_PATH.

Start by reviewing the linked issue 8 and the listed StorageFile, DataPackage, media, activation, and neighboring-file APIs, along with the referenced Gists. The issue needs to be narrowed to a specific API or documentation change before work can begin; done would mean an agreed path-based alternative, fix, or documented limitation.

Written by the indexing model from the issue text.