Modeling Files and File Dependencies

How are things stored today?

Courses in Studio Storage

• store static assets (images, PDFs, etc.) in MongoDB/GridFS
• files are in a folder structure, though only a flat view is available in the UI
• files are not versioned by the system, and do not follow any draft/publish flow

Libraries v2 Storage

• store assets via django-storages
• files are associated with specific components, and are local to those components

Current shortcomings

1. Course storage of assets becomes a disorganized mess, and it's hard to find files and where they're used.
2. Changes to course static assets are reflected immediately and will break XBlock content that references them.
3. The v2 library approach of storing things locally within the component makes it cumbersome to share assets across multiple Components.

Other considerations

1. Some of the latest mockups start to really blur the line between Files and Components, e.g. wanting to upload a Video and then organize it into folders alongside.
2. Many files have relative links to each other that would not be captured by our system, e.g. code files that do relative imports, href links, etc.
3. We have support in the data model to make a Component have many associated files.

Proposal: Folders as a type of Component with explicit dependencies

1. Files are stored locally to a given Component (e.g. image upload in a Problem) by default.
2. Top level folders are a Component type under the covers (however we represent them to the user).
3. We capture explicit dependencies between ComponentVersions in the data model, including the whole "null value for version means latest published version" convention.

So in this case, a Folder is its own namespace–you could use it for something like "all the PDFs in this course". It can have subdirectories in it, but these aren't Components.

Implications

• This could be made generic for any particular asset you're using from another Component, which could mean that your Problem uses stuff from another Problem. I think this is okay, though it might get messy in practice.
• The way we currently store ComponentVersion to RawContent associations means that making many small edits to a large folder will create a lot of rows.
• We might eventually need a way to model extended paths, since MySQL constrains us on the indexable length of file paths now. (Extra columns maybe?)
• File-local references (e.g. an HTML file including a JS file) would work within a given Folder Component and any of its sub-directories, but would not work across Folder Components.

Migration Path

We'd put everything in a current course into one top level Folder Component. I'm not sure how we'd differentiate this in the UX though–we definitely need a better name than Folder Component to differentiate between creating different ones of these vs. nested sub-directories.

We might be able to do this in a way that works in our favor by having them explicitly move the stuff that they want out of the legacy space, and they can leave/ignore the stuff they don't care about.

No implementation files or tests are named. Start by reviewing the current course and Libraries v2 storage models, then map the proposed Component, ComponentVersion, file dependency, folder, and migration concepts. Done requires an agreed data model and migration path that addresses sharing, versioning, relative references, and path constraints.