Analysis Info Provenance

Start with the developer guide section linked in the issue and compare its auto/user terminology with the provider examples in the issue. The payload names no implementation files, tests, or concrete completion criteria; this issue is primarily an open design discussion about replacing the existing model.

Within Binary Ninja we currently have a concept of user and auto analysis information, the difference in the two is actually more complicated than what is said in the developer guide here the only commonality between differing analysis info types (eg. data variable and section) and their auto/user API's is that they (auto/user) are what the core can key off of to adjust its behavior, as an example:

Behavior of auto sections:

>>> bv.add_auto_section("test", 0x0, 0x1000)

>>> # Save the database and reopen
... bv.get_section_by_name("test")

Ah yes, as expected, we no longer have the auto section!

Behavior of auto symbols:

>>> my_sym = Symbol(SymbolType.DataSymbol,0x1e9dbe188, "test")
... bv.define_auto_symbol(my_sym)

>>> # Save the database and reopen
... bv.get_symbols_by_name("test")
[<FunctionSymbol: "test" @ 0x20>, <DataSymbol: "test" @ 0x1e9dbe188>]

What??? I thought that auto symbols were thrown away?? At one point they might have been, but as the product is developed the meaning of auto & user changed. But why?

To understand we must first list off the providers of this analysis information:

• Views (MACH-O, ELF, PE, Shared Cache)
• Debug Info (DWARF, PDB, IDB)
• General Analysis (Pointer sweep, Linear sweep, RTTI, Objective-C)
• UI (Shared Cache Triage View, Scripting console, Sidekick)
• User (You!)

Not only do each of these categories vary in the application time but also in their persistence:

• Views: Expected to be re-applied on every open
• Debug Info: Expected to be applied once
• General Analysis: Expected to be overwritten by subsequent analysis
• UI: Both (see: scripting console: bv.define_auto_*)
• User: Expected to be applied once

Now lets discuss what providers should be consulted first:

1. User (Obviously)
2. UI, Debug Info
   • Example: Shared cache loads image, and there is some debug info applied on top of it, the debug info would be prioritized higher than the UI load image action.
3. Views
4. General Analysis

So we have 4 general priorities, obviously there is a bunch of overlap which we wont discuss further as its not important. And to add on even more, providers such as UI and User might want their actions to be reversible (through undo actions)!

Laying this out it becomes clear that auto vs user is not about persistence, but instead about priority, what gets to override what.

Clearly we cannot cover all use cases with just two "buckets" of information, we need to provide a simple way to extend the core analysis model with new buckets.

Enter analysis provenance, replacing the old auto & user model with providers.

An Analysis Provider has:

• A name (not used to determine priority)
• An associated ID

That is it, the provider does not hold any actual data. That is managed by the view, exactly how the auto and user information is handled currently.

When you create an analysis object such as a data variable you will now be required (at least at the FFI boundary) to pass the provider ID that the analysis object is associated with.

The provider ID is special in that it also doubles as the priority, higher the number, higher the priority. This provides the ability to do cheap comparisons between analysis objects to determine which one to use. The provider ID is not stable by design as the ID should be derived from the providers relationship with other providers. Take for example:

You create a new plugin that parses out all of the exception information of a specific binary, you know that the auto analysis will create pointers in that information so you want to make sure that your plugin is prioritized over that, you will create your provider and say "I want to be prioritized over the auto analysis provider".

By deriving the ID like this we can define clear relationships between different providers while also allowing providers in the future to insert themselves before and after other ones without knowing ahead of time all of the possible providers at a certain ID. This comes with the requirement that we register the providers all at the same time, so that we can avoid ID collisions and have correct ordering, this is the single most controversial requirement.

One issue with this model is there are certain analysis objects which do not need to modeled like this, instead they want to overlay information from the user onto analysis, such as indirect branches.

This is an incomplete issue, the general purpose of making this is to provide a place to discuss openly about our intention to replace user/auto with something more flexible.