Offloading of objects during registration is a difficult to debug trap for inexperienced users

^{Originally posted by **fg91** January 18, 2024}
Let us consider an example where 1) the type engine transports an object by offloading it to blob storage (in this case as a pickle file) and 2) where this object is instantiated not in a task but when calling a task in a workflow:

```py
from flytekit import task, workflow

class Config:
"""Something that is transported by the Flyte type engine via pickle."""
def __init__(self, a) -> None:
self.a = a

@task
def print_config(config: Config) -> None:
print(config)

@workflow
def wf():
print_config(config=Config(a=5))
```

This workflow fails with:

```console
File ".../flytekit/core/base_task.py", line 626, in dispatch_execute
raise type(exc)(msg) from exc

Message:

Failed to convert inputs of task 'wf.print_config':
[Errno 2] No such file or directory: '/var/folders/vz/l684gsw57pndcbm_909n9jp40000gn/T/flyte-e45vhbku/raw/91b6f84917aa22176a7ba5a4f4e71fd5/74a1306209f894ebf0fc720fde719b3b'

SYSTEM ERROR! Contact platform administrators.
```

The reason is that during registration, `pyflyte` does not realize that the object needs to be uploaded to blob storage. The user would have to proactively configure the raw data prefix.

I would argue that this example is very difficult to understand and debug for users that don't have a clear understanding of Flyte's data model and too "simple" to let users fall into this trap.

As a user I would want flytekit to 1) realize that during registration, files need to be offloaded to blob storage and 2) the backend to specify a default raw data prefix during registration unless I configured it explicitly in my flyte config file.

---

How could this be fixed?

In the `FileAccessProvider`, we need to prevent that [`put_raw_data`](https://github.com/flyteorg/flytekit/blob/6279b817d5df76e651c866c6c5ba7e12a65e12b2/flytekit/core/data_persistence.py#L285) stores offloaded objects locally during registration.

* Is there a way to determine from "the flyte context that we are in registration mode"? Then we could at least catch this error and tell users to configure the raw data prefix.
* Could `pyflyte` request the `raw_data_prefix` from `flyteadmin` if the user didn't set it explicitly and "make the file access provider aware of it"?