perf: two-layer ibl hashtable with inner fixed-size

_From [derek.br...@gmail.com](https://code.google.com/u/117968039472581148324/) on February 21, 2009 10:52:11_

We should try a two-layer scheme for rets: a 256-entry table updated on
every call, with collision chaining at the target.

See Ole's paper: http://engweb.vmware.com/~agesen/wbia2006.pdf summary of scheme:

```
2 level return lookup hashtable
1st level fixed size, direct-mapped 256 entries
no cmp for empty or for collision (cmp at target)
2nd level full table
@ every call prime the first level w/ a store
mov after_call_frag_prefx => table_slot
@ return
spill eax, ecx
ret addr -> ecx
movzx cl -> eax
jmp *table_base + eax
@ after_call_frag_prefx
lea compare ecx to after_call_addr
=> miss: spill flags, full ret ibl
hit: continue (common case no eflags and small dcache footprint)
```

discussion notes:
- thread-shared full ret ibl table
- thread-private 256-entry table: but then w/ shared fragments need
spill + extra instrs on call store?
stick whole table in TEB
or use own segment (PR 208009)
or try shared table: on uniprocessor may work fine
- no call inlining
- mark after-call as FRAG_X, propagate to trace if 1st bb there
- insert collision prefix if fragment starts w/ FRAG_X
- need to manage hardcoded code cache addr versus cache deletion
hack to combine w/ linking:
- put in unreachable jmp to after-call addr, after jmp to callee
- when unlinked, put table-empty addr there
- when linked, put code cache addr there
- selfprot: how allow write to table?
if through segment and ds has limit below it then could protect

_Original issue: http://code.google.com/p/dynamorio/issues/detail?id=31_