deconflate "don't import anything when inspecting" feature and "don't stat a lot of files" optimization of plugins

	@glyph reported
Trac ID	trac#3774
Type	enhancement
Created	2009-04-14 05:12:55Z

Right now we use dropin.cache for two things:

1. avoiding doing a lot of stat() calls by having one well-known location for the plugin metadata, which we can load very quickly if it exists and matches the directory's mtime, and
2. keeping metadata about plugins around so we don't have to import any python modules to get an enumeration of all the available plugins.

The first optimization is cool, but ultimately not very useful because we're not persisting a lot of metadata anyway and we end up loading plugins. The second could be very useful if we could rely on it, but we can't, because the cache might not be hot, and we might load code at any time. So we re-implement the don't-import-me optimization in every system that uses plugins, because it's actually important to be sure.

radix has convinced me that smashing these together has caused a bunch of problems.

I propose that we break this optimization into two parts, which are run at very different times. We write a tool which loads and inspects modules, and generates a file, which we check into version control and provide with our releases, that contains all the information about the plugins provided by a particular package. That would include persistent metadata about the plugin, as described in #3773.

In fact, in order for this to be really useful, we really need to have cached metadata, otherwise there's no way to determine whether it's worthwhile to load a plugin or not.

This file is generated from the code to avoid any redundant statement of metadata, but it is an explicit and canonical representation for the metadata about that plugin. We can then have an API like getPlugins can then be guaranteed to never import anything, and have a discrete, separate "load" step, so that only the desired plugins get loaded.

A straw man implementation:

We would put a new script in bin/, "twisted-plugins-update". Let's say you add a new tap maker to twisted/plugins/twisted_mail.py or something. You then run "twisted-plugins-update" and it writes twisted/plugins/twisted_mail.dropin. Then you check in. When the plugin system goes around to load plugins, it only looks at .dropin files. It builds the cache out of dropin files.

Without getting into the extra persistent metadata I am proposing in #3773, i.e. storing only what we're storing now, twisted/plugins/twisted_mail.dropin would look something like this:

name: twisted.plugins.twisted_mail.TwistedMail
interfaces: twisted.plugin.IPlugin, twisted.internet.interfaces.IServiceMaker

although I'd like this to start being the place where names and descriptions are kept as well.

The format is unimportant, since this is serving the exact same purpose that dropin.cache does now, but without running code at runtime; I propose it be text-based simply to minimize deployment issues and confusion though.

For the stat() optimization, we could still write dropin.cache pickles, which would load fast and mash all the data into a single file, but if they were out-of-date we would never need to execute any code to update them, and if they can't be updated due to permissions issues, there's no serious problem.

This would also make packagers jobs' easier, as for example eggs could skip the post-install step but still get a reasonable level of performance (no module importing), and we could really confidently skip cache persistence if permissions were unfavorable, knowing that we weren't altering the semantics of the program at all.

Having a tool to update this metadata for us will prevent us from having to maintain a redundant pile of metadata separate from the code; the code is authoritative, and if we ever screw up the metadata we just need to run the update tool and commit.

trac-id__3774 3774
type__enhancement enhancement
reporter__glyph glyph
priority__normal normal
milestone__ 
branch__ 
branch_author__ 
status__new new
resolution__None None
component__core core
keywords__ 
time__1239685975000000 1239685975000000
changetime__1239895539000000 1239895539000000
version__None None
owner__ 

Start by tracing the current dropin.cache handling and the getPlugins API, then review the proposed bin/twisted-plugins-update entry point and twisted/plugins files. Done means the metadata-generation and stat-cache concerns are separated, with plugin enumeration able to use checked-in metadata without importing modules at runtime.