Better Asset System: Externals Part 1 - Locator

## Introduction

The Engine needs to interoperate with other systems, such as:

* the file system,
* a http(s) server,
* stdio,
* …,
* _a logical abstraction over another system_

It does so by getting input from and giving output to those systems in constructs provided by these systems, such as:

* a file, directory,
* request, response,
* …

They are often referred to as Resources, but …

We may call them __Externals__.

### What is an External?

The Engine interacts with Externals for the means of input and output.

### How is an External interacted with?

Every system works differently, has different kind of constructs and therefore different interface(s).
We may call an abstraction over an interface of a system __Protocol__.

### Where do Externals exist?

Externals exist in their system. They may be hierarchically structured within their system.
Relevant Externals are mostly local - grouped together, such as in the application directory in the file system, or a superior endpoint of an http(s) server.
Related or alike Externals may also be local, such as Assets, Scripts, …; Models, Scenes, …

## Requirements

The representation must allow definition of:

* different Protocols,
* hierarchies,
* relative Locations,
* in-External Locations

It must also allow trivial transformations to established representations, such as file paths, URLs, …

## Proposal

###### _This is the first of hopefully four indepently applicable proposals. They aim to split Externals and Assets, while trying to improve the Terminology, Structure and/or API for and between them. They should be applicable for both Engine Runtime and Suite._
###### _Some terminology and functionality is borrowed from the URL standard._

### How is an External located?

An External is located by a Locator and its components `scheme` and `path`.

### What is a Locator?

A Locator describes a Location and how to access it.

### Locator Components

#### _Scheme_

Identifies the Protocol.

#### _Path_

Location of the External. Every Locator detail other than `scheme` and `fragment` belongs to `path`.

Representations of `path` are _scheme-specific_, except the character `/` has reserved meaning.

For the file scheme this would be just: _path_

For the http(s) scheme this would be: _domain/ip_:_port_ + _path_

…

##### Further distinction within `path` must be done by other in-Engine libraries if necessary.

#### __How is part of an External located?__

A part of an External is located by the Location of an External and a `fragment`.

#### _Fragment_

Locates part of an External. We may call parts of an External Fragments from now on.

Not every External has Fragments.

If no `fragment` was defined the Locator locates an External, a Fragment otherwise.

##### Note: A file is not a Fragment of a directory. Both file and directory are Externals. Distinction may not be clear.

### What is an Asset?

An Asset is a deserialized External or Fragment.

## Representation

A Locator is represented by

`scheme` :// `path` # `fragment`

#### __Examples__

```
file:///path/to/app/path/to/file.ext#fragment
└─┬┘ └────────────┬─────────────┘ └──┬───┘
│ │ │
scheme path fragment
```

```
http://192.178.1:8080/path/to/file.ext#fragment
└─┬┘ └──────────────┬──────────────┘ └──┬───┘
│ │ │
scheme path fragment
```

```
path
│
│
asset:///#assetid
└─┬─┘ └──┬──┘
│ │
scheme fragment
```

### Relative Locations

A Locator can represent a relative Location. This is _scheme-specific_.

A Locator does not know whether it is absolute or relative. This is _scheme-specific_.
Wrongfully Joining an absolute Locator to a relative Locator without `fragment` cannot be caught as an error by Locator.

Locations can only be split into multiple Locators at instances of `://` or `/`, `scheme` must always be suffixed by `://` and `fragment` always be prefixed by `#`.

Locators defining a `scheme` can be relative to another Locator with the same `scheme` - not only absolute Locators can define a `scheme`.

#### __Examples__

```
file:///path/to/app
```

```
file://path/to/file.ext
```

```
http://path/to/file.ext
```

```
file://
```

```
path
```

```
file.ext
```

```
#fragment
```

#### __Joining Relative Locators__

Relative Locators must have either the same `scheme` or one or both have none. Locators with a `fragment` each cannot be joined. Minor Normalization might be applied after Joining.

### Normalization

Locator may feature minor Normalization by replacing contiguous sequences of `/`s in `path` with just one `/`. If `/` is the last character of `path`, truncate it by one character.

Further Normalization is _scheme-specific_.

To limit representation variaties of Locator and therefore handling cases for consumers Normalization is desired, but may also be delegated completely.

### URLs Problem

For unknowing developers representations such as:

`file://path/to/file.ext`, or `path/to/file.ext`

might describe a reasonable Location. But adhering to the URL specification means that `"path"` is the host and `"/to/file.ext"` is the `path`.

Correct would be:

`file:///path/to/file.ext`, and `/path/to/file.ext`

So whats different? `path` must be prefixed with `/` to indicate the end of host.

Most implementations know the file `scheme` specifies no host and will treat both representations equally. For the http(s) `scheme` both representations cannot be treated equally.

#### __So why is this not a problem within Locator?__

Locator delegates distinctions within `path`. A relative Locator's `path` must therefore not be prefixed with `/`, but can be.
Absolute `path`s must still adhere to their specifications and `scheme`s where there is an expected part before the first `/` for it's `path` like http(s) are not freed from this problem. But relative `path`s can freely start with or without `/`.

Therefore, joining a Locator with relative Locators from either `path/to/file.ext` or `/path/to/file.ext` result in equal joined Locators by minor Normalization or consumers are responsible to treat one `/` or a contiguous sequence of `/`s equally to uphold this guarantee.

Since game developers will mostly always use relative Locators this problem is likely hidden from them.

### Location Local

In-Engine systems, such as the External or Asset system, may define a Location Local. All relative Locations are then assumed to be relative to the Location Local within the system.

### Instructor

Locator may feature URLs Query syntax `?` or something similar to support for example custom asset pipelines.
Since its applications are generic, usage depends on consumers.

I tend towards waiting for other asset related systems to be prototyped or decided upon before deciding for a standard here. E.g.:
* Whether build params can be set and trigger a corresponding build by a cache distributing built assets, or whether they must be reimported manually. First option may include transmission to a http(s) server at some point and must be representable as URL.
* How about describing different params for only one Fragment? Fragments must then be discovered first to be handled individually. Is this desired?
* …

No files, tests, or entry points are named. Start by reviewing the Locator requirements and proposal, then inspect the existing asset-related systems in the Engine and Suite. A completed effort would have an agreed, independently applicable Locator design covering protocols, hierarchies, relative locations, fragments, joining, and normalization.

Written by the indexing model from the issue text.