3.4: Finally Freed

To find out why our mascot for this release has a pitchfork and more on nerdy naming, read below the fold. For the summary of Braize’s (3.4) major new features (including one surprise feature that appeared mid-roadmap), here’s the highlights:

Inherited Types (C++ Support)
Automatic Outlining
Sparse Switch Recovery
Import from BNDB
Enterprise Improvements
Debugger Improvements

Experimental Features

Components UI
MH_FILESET

Many More

You’ll notice the theme of this release has been major improvements in decompilation, we’re really excited with the quality of improvements for the first three major features described above and they’re joined by several other important improvements as well.

So why is the mascot for this release Binjy with a pitchfork? And what is Braize anyway? In case you missed our last announcement, we’ve begun code-naming releases after Sci-Fi/Fantasy planets and in this case Braize is one of the lesser known planets in Brandon Sanderson’s Cosmere, but despite its reputation as a prison/hellscape planet in that universe, as huge nerds, we couldn’t resist including it early in our naming scheme. In this case, super-powered heroes were released from torment there, just like 3.4 has been released into the world with some super powers of its own! It’s also fitting that “C” is Coruscant, with tomorrow being May the 4th! Feel free to send us your suggestions for “E” and beyond.

Major Changes

While major releases get all the press, even our point releases include a huge amount of changes. The first three can produce significantly better decompilation results across a variety of binaries and platforms.

Inherited Types

It will take more than one new feature to solve C++ reverse engineering, but one of the biggest obstacles has been resolved with this latest feature: the core type system now supports inherited types. So what does that mean? For starters, the UIs and APIs around creating structures have changed. Now when you use s to create a structure in linear view or the type sidebar, you’ll see many new options:

The updated type system now lets you assign Base Structures, whose members will be inherited by your structure. You can avoid having to create duplicate members for every class in a hierarchy, and cross references will be propagated up the inheritance chain. Base Structures can also be located at an offset, supporting structures with multiple inherited bases.

For virtual function table-like classes, there is now the ability to propagate data variable references to the structure members. Looking at the cross references to a structure member will then include any annotated data variables using that structure. Simply put, if you mark up virtual tables, you can follow their functions from where they are used to what values they could have.

What really matters is what this allows you to do. Consider the following simple example:

You can create these types in the above UI, or you can use the notation shown there as we’ve extended the clang type parser to include the __inherited keyword.

Notice that the virtual tables support inheritance as well!

The beauty of this is that, when the types are applied, you get cross-references across the entire binary for not only the base class, but all the children as well. Check out the cross references for A’s virtual table versus B’s virtual table in the following sample file:

All this combines to make a very powerful type system for object oriented languages like C++. For more details and an even more complicated example showing multiple inheritance, check out the C++ Types help document.

One thing to keep in mind is that compilers don’t always follow the layout you expect! While debug info formats like PDBs contain the true data, your types may not always copy/paste directly from source code, so creating and applying the appropriate types may require some manual effort (for now!).

Automatic Outlining

Yes, we’ve talked about it before, but not only is the feature now enabled by default, it has had some important upgrades! To recap, this feature identifies instances where either a function call like strcpy might have been inlined and turns it back into an outlined function, or even just identifies situations where memory setting operations might have been unrolled (think initializing a stack array or structure to zero) and replaces them with a more convenient to a memset operation.

You can tell the feature is enabled by looking in your analyzed binary view for a .synthetic_builtins section:

This fake section is created and appended to your existing memory layout, as a place to store function prototypes for these built in functions. Then we re-write analysis to point to those built-in methods, replacing anywhere they are inlined. This makes it easy to select one of the function prototypes to identify all the places in the binary where it was applied via cross references:

In this new release, outlining can identify stack and global memory being used for similar patterns. It avoids some false positives by following type alignment constraints, adds supports for UTF-16 wide strings, and will also outline memcpy and memset for some well-known native architecture instructions such as rep stos* and rep movs*.

This new feature can dramatically improve decompilation. As you create variables, the analysis will create memset calls based on your types, automatically converting sequences of assignments to more concise outlined calls. It may happen in some unexpected cases, but properly defining your types should improve results.

Sparse Switch Recovery

Next on the theme of decompiler improvements, we’ve enhanced our switch statement recovery to make it much more robust in the face non-contiguous values! In practice, that means you’ll see nested groups of if-statements recovered as switch statements:

How often does this happen? We weren’t sure ourselves so we took a few hundred random binaries from the the Decompiler Explorer (Dogbolt), and compared the switch statement recovery before and after:

Total Switch Statements 3.3
861

Total Switch Statements 3.4
1023

Improvement
18.82%

Binaries Scanned
184

Binaries Improved
53

Turns out, what we thought was a small ability to better handle sparse switch statements resulted in almost a 20% improvement in recovering switch statements!

Import from BNDB

Have you ever wanted to copy data from a previous BNDB into a new one? That process just got a whole lot easier with the new Import from BNDB dialog.

The Import from BNDB feature imports types from a previous BNDB into your currently open file. In addition, it applies types for matching symbols in functions and variables. Import BNDB will not port symbols from a BNDB with symbols to one without – the names must already match. To first match similar functions without symbols and port symbols over, consider using BinDiff, whose binexport plugin works with Binary Ninja.

Enterprise Improvements

For this release, we’ve focused on providing additional support for deploying the Enterprise server in different environments. We’ve also done a round of updates on the user experience of managing an Enterprise server. And, finally, we’ve addressed some annoying performance issues and some reported bugs.

Up until now, deploying an Enterprise server has been easy, simple, and fast…but, also somewhat inflexible. We are still working to address this inflexibility in future releases, but beta support for cloud deployments on Azure and deployments with Podman (which improves our Red Hat compatibility) have landed in this release. We’ve also changed the way we handle SSL, and now operate more cleanly behind edge routers, web application firewalls, and reverse proxies.

One of the major changes we’ve made server-side is moving Enterprise user data into an object store. For customers that are deploying the Enterprise server on local hardware, we provide one and migrate your data automatically. But, for customers that have a cloud deployment and want to store their data with their cloud provider, we can now support external S3-compatible object stores.

In the client, we’ve improved network and UI performance for files with a large amount of snapshots and projects with a large number of files. We’ve also addressed a number of edge-cases that could lead to errors or crashes.

Debugger Improvements

Thanks to Xusheng’s hard work, the debugger has really flourished since the last stable release, and we’ve heard from many users that they’ve been leveraging it with good results. The full list of debugger improvements is below, but the headliner is the ability to debug dynamic libraries. You’ll want to use the debug adapter settings menu option item to specify the binary launcher (for example, rundll32), distinct from the “input file” which would be the DLL or shared library itself:

Experimental Features

Experimental features are those that are disabled by default but still available for users to try out with some limitations. Two features that didn’t quite make the cut-off for reliability and performance are still available for early testing in this release.

Components UI

The new Components UI gives you a new way to organize your analysis. You can use it to your symbols into folders based on names, compiler units, or anything your heart desires! If you like scripting, there is a convenient API to enable you to write your own automatic organization. While performance for small and medium-sized binaries is good, for those working with large binaries that have large numbers of functions or global variables (on the order of >100k), the performance of the Components UI isn’t quite up to snuff yet. That said, we encourage everyone to give it a try! You can enable it with the ui.components.enabled setting.

MH_FILESET

MH_FILESET is a new Mach-O feature in recent macOS and iOS binaries where kernel extensions and libraries are collected into a single large file. The new support for these files can extract these modules without needing an external tool. This experimental feature is enabled by opening a Mach-O file using “Open with Options” and enabling the loader.macho.processFileset setting:

Note that this feature is disabled by default due to some crashes that have not yet been resolved. If robust MH_FILESET support is important for your workflow, make sure to switch to the dev branch as we expect improvements to the feature on that branch after the 3.4 release.

Other Updates

Community Contributions

Special thanks to op2786, WhatTheFuzz, Joe Rozner, and mkrasnitski, for your contributions to our open source components during this release cycle!

UI Updates

Feature: Header Import Dialog now assigns types for matching Import/Export symbols.

Feature: Plugin dependencies can now be installed out of the box on MacOs (matching Linux/Windows)

Feature: Show enumeration value on hover

Feature: Shift+double-click now opens in a new pane

Feature: Added context menu for Dock icon on macOS

Feature: Added new ‘Dark Ninja’ theme

Feature: Add current_*_ssa to the python console

Improvement: Upgraded to Qt 6.4.3

Improvement: Reduce UI lag due to symbols view

Improvement: Command line arguments are now preserved on restart

Improvement: Windows task manager description

Improvement: Install/Reinstall/Uninstall state in Plugin Manager

Improvement: Plugin Manager UI consistency fixes

Improvement: Counts are shown next to tag types in Tags View

Improvement: Plugin Manager plugin dependencies are now reinstalled on update.

Improvement: Type View now has horizontal scrolling when necessary

Fix: Make Triage View update as analysis updates (exports, base address, etc.)

Fix: Disambiguate Restart from Restart and Reopen

Fix: help() in the python console now works on Windows

Fix: Issue where type would be duplicated when renamed

Fix: Issue where Symbol list would miss double-clicks

Fix: Focus of import file picker dialog

Fix: Issue where python console could throw exceptions

Fix: Synchronized navigation for LLIL/LLIL-SSA to MLIL-SSA

Fix: ‘Item Type’ field in Create Array Dialog could be wrong

Fix: Width calculation of plugin manager to avoid truncation of descriptions

Fix: Condition where ESC key wouldn’t be processed in “Filter Views”

Fix: Crash resulting from malformed stack variable

Fix: Issue where Linear View wouldn’t update when a type was updated

Fix: Crash caused by Linear View sticky header

Fix: Crash in Triage View when platform is null

Fix: Bug that caused Enumeration member values to change sign when passed through the API

Removed: Legacy BNDB Merge Tool

Binary View Improvements

Improvement: ELF View now creates an array for .init_array

Improvement: Parsing of Mach-O exported symbols

Improvement: Create names for functions in the __mod_init_func section

Fix: Issue where ELF View could create sections outside of the address space of the file

Fix: Don’t create entry point at Binary View start

Fix: Issue where TLS, SEH, and CFG related function pointers weren’t properly handled when rebasing

Analysis

Feature: Initial support for >8 byte constant propagation

Feature: Recover untyped symbols in PDB

Feature: Add Type Libraries for MFC ordinals

Improvement: Don’t show parameter name if its the same as the argument name

Fix: Issue where structure member cross references could become stale

Fix: Inlining of tail calls within tail calls

Fix: Display of enumeration constants for small width enumerations

Fix: Enumeration display where ~value == 0

Fix: Handling of namespaces in the type parser

Fix: Showing the wrong size token in Pseudo-C

Fix: PDB parser continuing to run after closing the file

Fix: char signedness

Fix: Issue where no-return status wasn’t accounted for when overriding call type

Fix: Occasional crash when using objective-c workflow on Apple silicon

Fix: Crash parsing malformed __convention()

Fix: Stack parameter resolution on x86 tailcalls

Fix: Dead store elimination for partial writes

Fix: Crash related to inlined code

Fix: Properly propagate parameters through thunks

Fix: Issue where small types used as returns were not properly handled

API

Feature: Add APIs for dumping Type Libraries to JSON

Feature: IsConstantData API now available on C++ RegisterValue API

Feature: implemented __bool__ for BinaryView, Segment, and Section classes

Feature: Added MetadataChoiceDialog

Feature: New API for handling Constant Data Expressions

Feature: Set the __version__ property of the python API

Feature: New API for getting/setting application font

Feature: Data Variables can now be added to Components

Feature: New visit methods on IL instructions.

Feature: Many new APIs for handling derived structures

New Example: minidump implemented in Rust

Improvement: TypeLibrary::Finalize() not returns true on success

Improvement: Make splitPane return the created pane

Improvement: Expose GetSymbolsByRawName in the C++ API

Improvement: Implemented Rust version of interaction::get_form_input

Improvement: BinaryReader.seek has new optional wence parameter

Improvement: Document that you can’t create databases in the context of analysis callbacks

Improvement: Add additional documentation on how to access types

Improvement: Update the instruction_iterator.py example to work headlessly

Improvement: install_api.py script has been refactored with new options

Improvement: Deprecation warnings now LogWarn instead of LogError

Fix: Prevent plugins from calling GetMinimumCoreABIVersion while loading to avoid… shenanigans

Fix: HLIL SSA memory and variable version bug

Fix: set_comment_at in the scripting console

Fix: Issue where segment could be truncated

Fix: Crash when creating zero width enumerations

Fix: Crash in get_instruction_info when returning the wrong type

Fix: Suppress warning about partial string in get_string_at API

Fix: Bug in bv.get_symbol_at when Namespace was not None and symbol was an auto symbol

Fix: Rust headless Binary View not loading correctly

Fix: Many deficiencies in the C++ Binary View API including UAF and Type object leaks

Fix: InstructionTextToken default constructor not initializing members

Fix: ScriptingProviders.update_locals raising exceptions causing console entry to fail

Fix: Bug in C++ demangler not clearing output buffer.

Fix: Issue where you couldn’t read a Data Variable’s array members inside a struct

Fix: In C++ the Confidence::operator== was broken

Fix: Many many type hint improvements

Deprecation: __len__ of Segment objects in the python API

Deprecation: Function.function_type is deprecated in favor of Function.type

Deprecation: Function.explicitly_defined_type is deprecated in favor of Function.has_explicitly_defined_type

Architectures

Improvement: rep mov/stos x86/x64 instructions are now lifted as intrinsics
**Improvement: **Default carry/zero flag behavior around shifts/rotates

Improvement: x86/x64 floating point comparisons and loads

Fix: R_AARCH64_RELATIVE Relocation

Fix: Thumb2 case labels for switch cases based on address.

Fix: Aarch64 lifting of UMULH/SMULH instructions

Debugger

Feature: Added a safe mode that prohibits launching any file

Improvement: Updated LLDB to 16.0.0

Improvement: Added a button to open the debug adapter settings dialog from the debugger side bar

Improvement: Show a progress bar when adding a binary view into the debugger binary view

Improvement: Do not save the debugger binary view into the database

Improvement: Confirm the launch operation when a file is launched for the first time

Improvement: Open the register’s value in a new tab when a register value is middle-clicked

Improvement: Register dereference now recognizes wide chars, utf16/utf32 strings, functions, and data variables

Improvement: Show a preview for the address when hovering over an register

Improvement: Support editing register value by pressing the “E” key

Improvement: Support adding a breakpoint from the breakpoints widget

Fix: Breakpoints do not work after the module has been rebased

Fix: LLDB adapter crashes when connecting to a remote process using gdb-remote on a real remote host

Fix: LLDB adapter reports the launch as failure if the target launches and then exits without hitting a breakpoint

Fix: Fix a memory leak and properly free the debugger binary view after the target exits

Fix: Clicking the three-dots button in the debug adapter settings dialog now opens the folder the file is in

Fix: Disable resizing the debug adapter settings dialog

Fix: Fix a crash when opening a database that has been saved during debugging

Fix: Fix a UAF in attach process dialog

Fix: Pressing ESC does not close the attach process dialog

Fix: Do not add an entry breakpoint if the module has no entry function

Fix: Properly set the default platform and architecture for the debugger binary view

Fix: Keybindings do not work in register/breakpoint/threads widget

Fix: Properly calculate the width of the hint column in the register widget

Fix: DebuggerController.register_event_callback does not work

Fix: DbgEng adapter cannot debug a file whose name is a valid hex integer

Fix: Font settings do not hot-update when they are changed

Fix: Demo version of BN crashes when there are at least two tabs open

Enterprise

Feature: Enterprise server storage now uses an S3-compatible object store, rather than relying on having a local filesystem available (which can be replaced with an external object store, if required)

Feature: Added a get_bundle command to the management interface as another way to obtain a copy of the offline install bundle

Feature: Added a –latest-only flag to the client_updates download command in the management interface, which will download Enterprise client updates for only the latest stable and dev channel builds

Feature: Added preliminary support for Podman deployments

Improvement: Re-wrote the handling of progress functions, which makes cancelling certain actions far more responsive

Improvement: Enabled links in file metadata

Improvement: The “Focus Chat” action now focuses inside the text input box

Improvement: The sync button display logic was re-written, causing it to visually change far less frequently

Improvement: The Remote Browser is now properly themeable

Improvement: “Remember Me” option is now on by default

Improvement: Performance improvements for servers with a large number of project files

Improvement: Added caching for projects in the client

Improvement: Added support for alternate Docker Swarm stack names

Improvement: Added support for turning off server TLS (connections with Enterprise are still required to be encrypted, but this makes hosting behind an existing proxy far easier)

Improvement: The management interface’s install command now states which version has been installed or updated to

Improvement: Offline updates downloaded from the management interface now use our much smaller “bundle” format by default (use –legacy-format if you need the old one for clients older than version 3.0)

Improvement: Added support for changing internal service hostnames (required for supporting ACI deployments)

Improvement: A more helpful message is shown while the backend container is still starting up

Improvement: Client downloads are now gated behind user login

Improvement: Users are now able to log into the server using SSO via the browser (previously, only the client was able to use SSO)

Fix: Files can now be uploaded directly into a project folder, instead of always being uploaded to the root of the project

Fix: Prevent duplicate license checkouts due to client missing response from server

Fix: Clear saved credentials when server login fails

Fix: The collaboration API examples are included with the client again

Fix: Server key changes are now handled more gracefully in the client

Fix: The refresh action now correctly updates project file lists

Fix: Renamed the C++ Enterprise API module (matches the Python API)

Fix: Fixed a crash in the Remote Browser when selecting a deleted project

Fix: Fixed a crash when dropping an item onto a deleted project folder

Fix: Fixed a crash when quickly navigating back-and-forth between projects

Fix: Added an error message for when downloading an original file fails

Fix: Added missing param name in the Project.upload_new_file API

Fix: Filename extensions are no longer stripped from uploaded BNDBs

Fix: Temporary chat and user position disconnects are now warnings, not errors

Fix: Large merge conflicts will now no longer hang the client

Fix: The logic for merging tags now has less superfluous conflicts

Fix: Newly uploaded files no longer show as “Unknown File” in the New Tab widget

Fix: The Remote Browser’s folder sorting now works as expected

Fix: Address parsing in chat no longer fails to properly identify a linked address in certain circumstances

Fix: Added missing Docker Swarm support for the backup, restore, and change_password management interface commands

Fix: A more appropriate error message is returned when trying to delete the last remaining snapshot in an uploaded project file

Fix: Set a more appropriate default range for license reservation stats

Fix: Updated Enterprise server dependencies to address potential denial of service opportunities

Fix: Client now properly uses the local trust store for certificates on macOS

Documentation

Feature: Offline search now works in user documentation!

Improvement: Update docs for parse_expression

Improvement: Do not import packaging in deprecation.py

Improvement: Document that progress functions must return True to continue loading

Improvement: Using latest mkdocs and sphinx

Improvement: Better documentation on python dependency installation

Improvement: New cookbook examples

Improvement: Document silent install process on windows

Fix: Fonts in offline documentation were still using online fonts

Miscellaneous

…and, of course, all of the usual “many miscellaneous crashes and fixes” not explicitly listed here. Check them out in our full milestone list: https://github.com/Vector35/binaryninja-api/milestone/18?closed=1

Article Link: Binary Ninja – 3.4: Finally Freed

1 post – 1 participant

Read full topic

About The Author