asynchronous, non-blocking file I/O

	@glyph reported
Trac ID	trac#3983
Type	enhancement
Created	2009-08-27 01:37:55Z

Twisted should have an official API for (at least notionally) non-blocking filesystem input and output.

This may not be possible without cheating a little bit, because asynchronous file I/O support is not really there in most operating systems. However, we could:

1. do the file I/O in a subprocess
2. do the file I/O in a thread (one thread per mount point? a threadpool?)
3. just block, with timers, but present an API which would be amenable to non-blocking later. (Possibly, this strategy is already partially satisfied by twisted.protocols.basic.FileSender.)
4. some combination of these, with a back-end that could be selected somehow

Something that FileSender does not satisfy is some equivalent to 'tail -f'; allowing a Twisted program to easily and non-blockingly monitor the log or file output of another program as if it were coming from a socket. This is somewhat tricky to do in the general case, since it's hard to detect when the log or file is done being written. Still there are heuristics (wait until this process has died, wait until this other file has disappeared) which could provide a useful substitute for certainty of connectionLost.

(I will be surprised if this is not a duplicate of something, but searches yield nothing.)

trac-id__3983 3983
type__enhancement enhancement
reporter__glyph glyph
priority__low low
milestone__ 
branch__ 
branch_author__ 
status__new new
resolution__None None
component__core core
keywords__ 
time__1251337075000000 1251337075000000
changetime__1363352220000000 1363352220000000
version__None None
owner__ 
cc__detly cc__yac cc__jamesbroadhead cc__oberstet

Start by reading twisted.protocols.basic.FileSender and compare it with the proposed subprocess, thread, timer, or combined approaches. The work would need a decided official non-blocking filesystem API, including behavior for tail-like monitoring and completion detection.