openclaw/apk-reverse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Dart AOT depth (reference + 3 scripts); fix encoding rule and add P26

Browse Source 
New reference:
- dart-aot.md: pinning the Dart version, building a matching decompiler, the object-pool
  model and reference indexes, register/boolean conventions, the three assembly signatures
  that identify most business logic, the locating workflow, patch strategy, and the traps
  specific to this layer (including ambiguous identifiers that resolve to protocol fields).

New scripts (parameterized, no machine-specific paths):
- dart_pprefs.py: object-pool offset -> code-site index, decoded arithmetically. Avoids the
  full-capstone pass that costs minutes and 1-2 GB and reads as a hang.
- dart_pool_strings.py: framed literal recovery with file offsets. Defaults to one-byte
  because automatic two-byte (UTF-16) detection is unreliable; --find searches both
  encodings and is the reliable way to locate a known literal.
- dart_disasm.py: annotated windowed disassembly (pool + boolean annotations) and a B/BL
  caller index that needs no disassembler.

Fixes and integration:
- framework-runtimes.md: point the Dart depth at dart-aot.md, and make the string-encoding
  rule precise (ASCII is one-byte/UTF-8; CJK is UTF-16LE) instead of leaving it as 'try both'.
- pitfalls.md: P26 -- self-built tooling that fails in a way that reads as a target finding
  (silent arithmetic, helper shadowing a stdlib module, measuring against an incomplete
  reference, and a knob that does not move the metric).
- SKILL.md / README.md: index the new reference and scripts.

Validated by running all three scripts against a real Dart AOT sample: the pool-ref index
agrees with an independently built index to 99.2%, and the literal extractor's one-byte
output was verified string-by-string against a decompiler's pool listing.
This commit is contained in:
inliver233
2026-09-19 22:19:55 +08:00
parent 46b0858efb
commit cf636c4263
8 changed files with 970 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+7
View File
@@ -41,6 +41,9 @@ references/ loaded on demand, one topic each
boundary with single-variable tests, choosing a native host
framework-runtimes.md Flutter / React Native / Unity: which layer owns the UI, and how to
find logic when there are no symbols (string encoding traps)
dart-aot.md Dart AOT in depth: version pinning and building a matching decompiler,
the object pool and reference indexes, register/boolean conventions,
the three signatures that identify business logic, locating, patching
native-and-so.md .so hosts, DT_NEEDED vs JNI_OnLoad, relocation limits,
relocation-free bootstrapping, replacing Java methods natively,
and which ABI/library is *actually loaded and executing*
@@ -67,6 +70,10 @@ scripts/ parameterized, path-agnostic
dex_strpatch.py byte-level string patch with a string_ids ordering guard
dex_classdiff.py prove a dex edit was surgical
dex_strings.py strings/URLs/SDK markers without a decompiler
dart_pool_strings.py recover literals from a Dart AOT snapshot (framed entries, the
one-byte vs UTF-16 split, file offsets, run-length noise filter)
dart_pprefs.py build/query the object-pool -> code-site index for a Dart snapshot
dart_disasm.py annotated windowed disassembly of Dart AOT code + B/BL caller index
find_refs.py count callers of a method before patching it
repack.py rebuild APK, strip only signatures, sign, verify
devsh.py quoting-safe ADB shell helper
SKILL.md
+4
View File
@@ -87,6 +87,7 @@ Load only what the current step needs.
| `references/recon.md` | Starting any new sample; identifying packer, SDKs, code location, ABI |
| `references/packers.md` | The app is packed/hardened, or an edit makes it die before your code runs. Also load before discarding any route as "blocked by the shell" |
| `references/framework-runtimes.md` | The UI is not native (Flutter / React Native / Unity / Cordova), or Java-layer hooks fire zero times while the UI clearly works |
| `references/dart-aot.md` | The logic lives in a Dart AOT snapshot (`libapp.so`): pinning the Dart version, building a matching decompiler, the object pool and reference indexing, register/boolean conventions, locating and patching Dart code |
| `references/native-and-so.md` | Patching in a `.so`, needing code to run before the app's own code, hand-built native payloads that crash inside the linker, or **deciding which library/ABI is actually loaded and executing** |
| `references/ad-removal.md` | Task involves ads, trackers, sponsored cards, splash/interstitial/reward |
| `references/membership-and-limits.md` | Task involves VIP, subscription, paid content, unlock, "fully cracked" |
@@ -113,6 +114,9 @@ All scripts are parameterized and path-agnostic; pass paths explicitly. Run `--h
| `scripts/dex_strpatch.py` | Byte-level string constant patch with **string_ids ordering guard** |
| `scripts/dex_classdiff.py` | Compare two dex class tables (set + access flags) to prove a patch was surgical |
| `scripts/dex_strings.py` | Dump/extract strings and endpoints from dex without a decompiler |
| `scripts/dart_pprefs.py` | Build/query the object-pool offset -> code-site index for a Dart AOT snapshot (arithmetic decode; seconds, not minutes) |
| `scripts/dart_pool_strings.py` | Recover string literals from a Dart AOT snapshot: framed entries, the one-byte vs UTF-16 split, file offsets, and a run-length noise filter |
| `scripts/dart_disasm.py` | Annotated windowed disassembly of Dart AOT code (pool + boolean annotations) plus a B/BL caller index |
| `scripts/find_refs.py` | Count and list callers of a method/field (blast-radius check) |
| `scripts/repack.py` | Rebuild APK with replaced dex, strip only signatures, sign |
| `scripts/dexpatch/` | dexlib2 method-level surgical rewriter (the preferred patch tool) + build notes |
references/dart-aot.md
+254
View File
@@ -0,0 +1,254 @@
# Dart AOT (Flutter) — reaching the layer that actually owns the logic
Load this when: the target is a Flutter app (`libflutter.so` + `libapp.so`) and the behaviour you
must change is decided **inside** the Dart snapshot, not in the dex. The Java side of a Flutter app
is a thin plugin shell; a dex-only plan will stall (`framework-runtimes.md` §the layer trap).
Everything below assumes you already confirmed the runtime is Flutter and that Java hooks fire
**zero times** while the UI plainly works.
---
## 1. Pin the Dart version before choosing any tool
The snapshot format is version-specific. A decompiler built for another Dart version may run and
produce plausible-but-wrong output, which is worse than failing.
```python
import re
blob = open('libflutter.so', 'rb').read() # the engine, not libapp.so
m = re.search(rb'(\d+\.\d+\.\d+ \(stable\)[^\x00]{0,60})', blob)
print(m.group(1).decode() if m else 'not found') # e.g. b'3.9.0 (stable) (Mon ...) on "android_arm64"'
```
If the string is absent, fall back to the snapshot hash the loader reports at runtime, or to the
engine build id in `libflutter.so`'s version string.
## 2. Get a decompiler that matches that version
**blutter** is the practical choice for Android `libapp.so` (arm64). It reconstructs the object pool
and per-class declarations. It is a *source tree you build*, not a binary you download — and its
prebuilt outputs only cover the Dart versions someone already built.
The version question is the first obstacle, and it has a clean answer:
- blutter's git HEAD typically supports newer Dart than its shipped `dartsdk/` directory.
- When the target's version is missing, blutter fetches it itself: it sparse-clones
`dart-lang/sdk` at that version tag (only `runtime`, `tools`, `third_party/double-conversion`),
generates a source list, and builds a `dartvm` static library for the target ABI.
- That build needs a real toolchain: `cmake` (>=3.20 is fine, including 4.x), `ninja`, `git`, and a
C++ compiler with C++20 `<format>` support (VS 2022 on Windows, gcc>=13 / clang>=16 elsewhere).
A `vcvars64`-style environment must be active before the build, and `CMakeLists` in the tree pins
`cmake_minimum_required(3.20)`.
- Budget **tens of minutes** for the first build of a new version and run it as a background job;
the compiler log is enormous (per-file include trees), so read only its tail.
Outputs of interest:
| Output | What it gives you | What it does *not* give |
|---|---|---|
| `pp.txt` | every object-pool entry with its `pp+0x…` offset: strings, types, closures, fields, stubs | who references what |
| `objs.txt` | reconstructed object shapes with type annotations | code |
| `asm/` | per-class declarations | **instructions — there are none here** |
| `ida_script/`, `blutter_frida.js` | symbol/annotation helpers | a finished analysis |
> `asm/` having one file per (obfuscated) class invites the assumption that it holds disassembly.
> It does not. Do not plan around it.
**`product` builds carry no debug info.** Expect `no-code_comments`, no function names, and
obfuscated identifiers. That is normal; the strings still survive (see §7).
## 3. The object pool is the whole game
Dart AOT uses **compressed pointers**: heap references are 32-bit offsets from a base held in a
dedicated register. Constants, strings, types, closures and field metadata live in one contiguous
**object pool**, reached through that register.
Two offset spaces exist and they are **not the same number** — mixing them wastes hours:
- **PP offsets** (`pp+0x…`) — what `pp.txt` and every pool-load in disassembly use.
- **file offsets** — where a byte actually sits in `libapp.so` on disk.
Keep the mapping explicit in your notes. When a tool reports "this string is at X", state which
space X is in.
## 4. Build your own reference index (blutter will not give you this)
`pp.txt` tells you what is in the pool. To *find the code that uses it* you need pool-offset →
referencing-instruction-address. Generate it once, then query it constantly:
```
python dart_pprefs.py libapp.so pp_refs.json
python dart_pprefs.py --lookup pp_refs.json 0x1d1a8 0xc9d0
```
**Do not build this index with a full-capstone pass.** Decoding every instruction of a multi-MB
`.text` with `detail=True` and querying `regs_access()` per instruction costs minutes of CPU and
1–2 GB of peak memory, and on a 16 GB host it presents as a hang with no progress output. Only three
encodings can read the pool, so decode them from the raw 32-bit words instead — the shipped script
does exactly that and finishes in seconds. The same argument applies to building a call graph
(§9): decode `B`/`BL` arithmetically.
## 5. What the registers mean
Stable across Dart 3.x arm64 AOT. Verify once on a known function, then rely on it.
| Register | Role |
|---|---|
| `x27` | **object pool base (PP)** — every constant/string/type load goes through it |
| `x26` | current thread (`[x26,#0x38]` stack limit, `[x26,#0x68]` isolate group) |
| `x28` | heap base — how compressed pointers are decompressed |
| `x22` | **null/base for booleans** (see §6) |
| `x15` | Dart's own stack pointer (not the system SP) |
| `x21` | class dispatch table (virtual calls load a target from it) |
## 6. Booleans are not 0/1
`true` and `false` are objects immediately adjacent to null, so they appear as small fixed offsets
from `x22`:
```asm
add x0, x22, #0x20 ; construct TRUE
add x0, x22, #0x30 ; construct FALSE
tbnz w0, #4, <label> ; test bit 4 of the value
tbz w0, #4, <label>
```
The bit-4 test is the standard "is this the false object" check, and it is the cheapest place to
force a decision: replacing the conditional branch with an unconditional one, or replacing a
constructed constant, flips the outcome without changing structure.
## 7. String encoding — get this exactly right
Pool string entries are serialized as:
```
[ tag byte ][ payload ]
tag = 0x80 | (len << 1) | two_byte_bit
```
- **one-byte strings** (ASCII / Latin-1): `payload` is the raw bytes.
- **two-byte strings**: `payload` is **UTF-16LE**.
Consequences that decide whether your search works:
- An **ASCII** literal (`/api/foo`, `isVip`, a URL) is found by a **UTF-8 byte search**. It is a
one-byte string — searching UTF-16 for it will miss.
- A **CJK / non-Latin** literal is a two-byte string: a UTF-8 search returns **zero**, a UTF-16LE
search hits. This is the case that makes people wrongly conclude "the strings were stripped".
- Therefore: **search ASCII as UTF-8 and non-ASCII as UTF-16LE**, rather than forcing one encoding
on everything. (This is the concrete form of `pitfalls.md` P25.)
- A raw byte scan for the tag pattern finds many false candidates. Accept a candidate only when
entries **chain** — the previous entry's `tag+payload` must end exactly where the next candidate
begins. Without that constraint you get order-of-magnitude more garbage than strings; a
per-byte scan of a multi-MB snapshot once produced ~880k "strings" of which essentially all were
misaligned noise, and a "no such feature present" conclusion was drawn from it.
```
python dart_pool_strings.py libapp.so strings.tsv --min 3
```
## 8. Reading AOT code: three signatures that carry most of the weight
Once you can disassemble a window with pool annotations, most business logic resolves into these
shapes.
**A — reading a map / JSON object (`map[key]`)**
```asm
ldur x3, [x29, #-8] ; the Map
ldur x0, [x3, #-1] ; compressed class id sits at offset -1
ubfx x0, x0, #0xc, #0x14 ; decode class id
mov x1, x3
... x2 = <pool string> ; <- the key, annotated by your pool map
add x30, x0, #0x342
ldr x30, [x21, x30, lsl #3]
blr x30 ; this is map[key]
```
Remember the quartet: `ldur [..,#-1]` + `ubfx #0xc,#0x14` + `add x30,x0,...` + `blr`.
**B — writing a map (`map[key] = value`)**
```asm
... x16 = <pool string> ; the key
stur w16, [x0, #0xf] ; key stored into the freshly built map
... ; then the value at the next slot
```
`stur` writes, `blr` reads. Confusing the two means patching a serializer that never touches the UI.
**C — function entry**
```asm
stp x29, x30, [x15, #-0x10]!
mov x29, x15
sub x15, x15, #0x20
ldr x16, [x26, #0x38]
cmp x15, x16
b.ls <stack-overflow handler>
```
Use this to find function boundaries when you need to delimit one.
## 9. The locating workflow
1. **Anchor on a string.** Search the pool for the shortest distinctive token — a field name, an
endpoint path, a label. Prefer identifiers over sentences (a sentence may be assembled from
fragments).
2. **Find its referencing instructions** via your index (§4).
3. **Disassemble a window** around each reference with pool annotations:
```
python dart_disasm.py libapp.so --pp pp.txt --refs pp_refs.json 0x26e390
python dart_disasm.py libapp.so --pp pp.txt --refs pp_refs.json --range 0x64f310 0x64f3e0
```
4. **Classify the site**: read (A) or write (B)? For writes, check whether the value is a constant
from the pool — that is a "send this flag" site, not a "decide locally" site.
5. **Walk outward.** Find the enclosing function entry (§8C), then find its callers:
```
python dart_disasm.py libapp.so --index callers.json --build-index
python dart_disasm.py libapp.so --index callers.json 0x26e230
```
6. **Cross-validate by clustering.** Related strings usually sit **adjacent in the pool** and are
referenced from **adjacent code**. If two field names are a few bytes apart in the pool and their
reference sites are a few instructions apart, you are looking at one coherent piece of logic —
strong evidence you are in the right place.
## 10. Patching this layer
- **Prefer changing the data source over the decision point.** Locate the chain
"read field from map → type conversion → store to stack slot" and replace the middle with a load
of a pool constant. One edit then feeds every consumer, instead of chasing each comparison. This
has produced clean results where per-condition patches did not.
- **Boolean sites** are the cheapest: `tbnz`/`tbz` → `nop` (always fall through) or → an
unconditional branch; `add xD, x22, #0x20` ↔ `#0x30` to flip a constructed constant.
- **A pool string may be shared.** Before editing one, count its references. If the same entry is
read by several sites, changing it changes all of them — self-consistent and therefore useless
for splitting behaviour. Prefer editing code over shared data.
- **Assert your addresses.** Patch with the original bytes asserted first and re-read the result
afterwards; an address that is off by one instruction will happily corrupt a snapshot into
something that still loads.
- **`libapp.so` inside an APK is usually deflate-compressed**, so in-place byte patching of the zip
entry is not possible — patch the extracted file, then replace the whole entry
(`repack-and-sign.md`).
## 11. Traps specific to this layer
- **Identifiers are ambiguous across layers.** A pool string that reads like a domain concept can
belong to a standard library or a wire protocol instead. A near-miss example worth remembering:
the token `expires` resolved to HTTP **response-header** parsing (its siblings were `date`,
`host`, `connection`), not to a subscription expiry field. **Read the surrounding strings and the
code shape before assigning meaning** — one misleading identifier can send a whole analysis down
the wrong branch.
- **Obfuscated names are not identities.** Symbol names are mangled (`_abc@12345` shapes are
common) and change between builds. String literals do survive, so anchor on those.
- **Clusters reveal structure.** Adjacent pool entries plus adjacent reference sites indicate one
subsystem; use it to avoid guessing what a function is for.
- **A write-only flag is usually an outbound parameter, not a local switch.** If a string is
referenced exactly once, at a site that stores a constant into a map that is then sent, the client
is *reporting* a value — changing it does not change local behaviour. Confirm whether the value is
ever read before treating it as a gate.
- **Full-file disassembly is not a debugging tool** (§4). Windowed disassembly is.
## 12. Verification
Static: asserted patch bytes, re-read from the built artifact, plus structural checks
(`verification.md`). Runtime: the changed behaviour must be observed in the app's own UI — for this
layer that means an actual launch, because nothing about Dart AOT code proves itself statically.
If the snapshot drives a decision that a server also enforces, see `membership-and-limits.md`:
getting the client side right does not make the outcome reachable.
references/framework-runtimes.md
+10 -4
View File
@@ -64,9 +64,11 @@ Establish layer ownership **before** investing in a patch direction.
| Change a decision permanently | `libapp.so` | most direct, hardest to locate |
| Change startup behaviour only | host native library constructor | survives runtime restarts |
Editing `libapp.so` is the most durable but locating Dart AOT code is genuinely hard: strings are the
most reliable entry, because Dart string objects and the code referencing them can be found by scanning
the snapshot rather than by pattern-matching machine code.
Editing `libapp.so` is the most durable but locating Dart AOT code is genuinely hard. This subject is
large enough to have its own reference: **`references/dart-aot.md`** covers pinning the Dart version,
building a decompiler for exactly that version, the object-pool model and how to index it, the register
and boolean conventions, the three assembly signatures that identify most business logic, the locating
workflow, and how to patch this layer safely.
## Locating logic without symbols
@@ -85,7 +87,11 @@ the snapshot rather than by pattern-matching machine code.
'utf-16le:', blob.find(phrase.encode('utf-16-le')))
```
Try both, and try a short distinctive substring rather than a long phrase.
Try both, and try a short distinctive substring rather than a long phrase. Which encoding applies
is predictable rather than random: in a Dart AOT snapshot ASCII literals are stored one-byte (a
UTF-8 search finds them) while CJK / non-Latin literals are UTF-16LE (a UTF-8 search returns zero).
`dart-aot.md` §7 gives the exact framing, and `scripts/dart_pool_strings.py --find` locates either
encoding by file offset.
- **A phrase that is absent may simply never exist as one literal.** UI text is often assembled from
fragments or templates, so "the whole sentence" can be missing while both halves are present.
Search the shortest distinctive token, and expect the interesting anchor to be a *label* rather
references/pitfalls.md
+38
View File
@@ -603,3 +603,41 @@ as a finding, and findings propagate into the plan.
`scripts/blob_decode.py`).
- **State a negative result with its scope**: "no UTF-8 or UTF-16LE literal match in this artifact"
is a finding. "The string does not exist" is a guess wearing a finding's clothes.
## P26. A self-built analysis script fails in a way that reads as a target finding
**Symptom**
A purpose-written script emits something that looks like a result -- an offset table, a count, a
"no matches" verdict -- and it is wrong. Nothing in the output says so, so it gets recorded as a
finding and the plan is built on top of it.
**Root cause**
Three variants, all hit in a single project:
- **A silent arithmetic error.** `(w >> 10) & 0xFFF << 12` binds as `& (0xFFF << 12)` in Python, so
the intended mask silently became a different one. The script ran clean and printed a plausible
table; every offset in it was wrong.
- **A helper script shadowing a stdlib module.** A local file named `dis.py` captures any
`import dis` performed inside a third-party package. The symptom appears as a circular-import
error *inside that package*, which reads like a broken dependency rather than a local name clash.
- **Measuring against an incomplete reference.** A "precision" number computed against a partial
listing reports the reference's gaps as your errors. Here it scored a working extractor at ~30%
when the reference itself was the incomplete side.
**Why it is hard to see**
The tool is trusted by default, because you wrote it for this exact job. Its output is well-formed
and arrives fast, which reads as competence.
**Do instead**
- **Sanity-check the shape before the content.** A ratio or distribution that is implausible for the
domain is a bug signal -- e.g. an average of 31 references per offset when the rest of the picture
implies ~3.
- **Cross-check against an independently built artifact.** Two independent producers agreeing to
~99% is evidence; one producer's own output never is.
- **Histogram before choosing a threshold, and re-measure after changing it.** If moving a knob does
not move the metric, the knob is not doing what you think (a run-length filter here barely changed
accuracy across a 10x range, which is how it was caught).
- **Exercise any fallback path against a case whose answer is already known.** An accuracy claim
derived only from the tool's own output means nothing.
- **Name helpers so they cannot shadow a module** (`dart_disasm.py`, not `dis.py`), and keep a
timeout on every scan (SKILL.md).
scripts/dart_disasm.py
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Annotated, windowed disassembly of a Dart AOT snapshot, plus a caller index.
WHY THIS EXISTS
---------------
Two things make Dart AOT code readable at all:
1. **Pool annotations.** A raw `ldr x2, [x27, #0x9d0]` is meaningless; annotated with the
pool entry it becomes `"vipflag"` and the surrounding code explains itself. This tool
resolves those loads and prints what they point at, including how many other sites
reference the same entry.
2. **Windowed disassembly.** Decoding an entire multi-MB `.text` in one pass is how a host
gets pinned at 1-2 GB and the run looks like a hang. Disassemble only the ranges you
care about -- that is what this tool does, and it is not a limitation but the method.
Booleans are annotated too: Dart materialises `true`/`false` as small offsets from the null
register, so `add x0, x22, #0x20` is marked TRUE and `#0x30` is marked FALSE. Those are the
cheapest patch sites in the whole snapshot.
The caller index answers the other half of every question -- "what calls this function".
It is built from B/BL target arithmetic on the raw words, so it needs no disassembler at all
and finishes in seconds.
USAGE
-----
# a window around one address (default: 0x180 before, 0x220 after)
python dart_disasm.py libapp.so --pp pp.txt --refs pp_refs.json 0x26e390
# an explicit range
python dart_disasm.py libapp.so --pp pp.txt --refs pp_refs.json --range 0x64f310 0x64f3e0
# widen / narrow the window
python dart_disasm.py libapp.so --refs pp_refs.json 0x64f1bc --before 0x80 --after 0x40
# caller index
python dart_disasm.py libapp.so --build-index callers.json
python dart_disasm.py libapp.so --index callers.json 0x26e230 0x64f0f0
INPUTS
------
* `--pp` is the decompiler's pool listing (blutter `pp.txt`): lines shaped
`[pp+0x9d0] String: "vipflag"`. Without it, loads are annotated with the raw offset only.
* `--refs` is the pool-offset -> code-site index built by `dart_pprefs.py`.
Requires `capstone` for disassembly; the caller index works without it.
"""
import argparse
import json
import os
import re
import struct
import sys
POOL_REG = 27
BOOL_TRUE, BOOL_FALSE = 0x20, 0x30
PP_LINE = re.compile(r'^\[(pp\+0x[0-9a-f]+)\](.*)$')
def text_section(path):
blob = open(path, 'rb').read()
if blob[:4] != b'\x7fELF':
raise SystemExit('not an ELF file: %s' % path)
if blob[4] != 2:
raise SystemExit('not ELF64: %s' % path)
e_shoff = struct.unpack_from('<Q', blob, 0x28)[0]
e_shentsize = struct.unpack_from('<H', blob, 0x3A)[0]
e_shnum = struct.unpack_from('<H', blob, 0x3C)[0]
e_shstrndx = struct.unpack_from('<H', blob, 0x3E)[0]
def sh(i):
return struct.unpack_from('<IIQQQQ', blob, e_shoff + i * e_shentsize)
_, _, _, _, shstr_off, shstr_size = sh(e_shstrndx)
strs = blob[shstr_off:shstr_off + shstr_size]
for i in range(e_shnum):
name, _t, _f, addr, off, size = sh(i)
end = strs.find(b'\x00', name)
if strs[name:end].decode('ascii', 'replace') == '.text':
return addr, blob[off:off + size]
raise SystemExit('.text not found')
def load_pp(path):
pool = {}
if not path or not os.path.exists(path):
return pool
for line in open(path, encoding='utf-8', errors='replace'):
m = PP_LINE.match(line.rstrip('\n'))
if m:
pool[int(m.group(1)[5:], 16)] = m.group(2).strip()[:120]
return pool
def load_refs(path):
if not path or not os.path.exists(path):
return {}
return {int(k, 16): v for k, v in json.load(open(path, encoding='utf-8')).items()}
def disasm_window(sample, addr, before, after, pp, refs):
from capstone import Cs, CS_ARCH_ARM64, CS_MODE_LITTLE_ENDIAN
text_addr, blob = text_section(sample)
end_of_text = text_addr + len(blob)
start = max(text_addr, addr - before)
stop = min(end_of_text, addr + after)
md = Cs(CS_ARCH_ARM64, CS_MODE_LITTLE_ENDIAN)
md.skipdata = True # without this, one undecodable byte silently ends the range
md.detail = True # without this, operands are unavailable
print('range 0x%x..0x%x focus=0x%x' % (start, stop, addr))
pending = {}
for ins in md.disasm(blob[start - text_addr:stop - text_addr], start):
if ins.id == 0: # SKIPDATA pseudo-instruction: no operands
print('%x: %s' % (ins.address, ins.mnemonic))
continue
note = ''
ops = ins.operands
rn = ins.reg_name
if (ins.mnemonic == 'add' and len(ops) == 3 and ops[1].type == 1
and rn(ops[1].reg) == 'x27' and ops[2].type == 2
and ops[2].shift.value == 12):
pending[rn(ops[0].reg)] = ops[2].imm << 12
elif (ins.mnemonic == 'add' and len(ops) == 3 and ops[1].type == 1
and rn(ops[1].reg) == 'x22' and ops[2].type == 2):
note = {BOOL_TRUE: ' ; TRUE', BOOL_FALSE: ' ; FALSE'}.get(ops[2].imm, '')
elif ins.mnemonic in ('ldr', 'ldur') and len(ops) == 2 and ops[1].type == 3:
base = rn(ops[1].mem.base) if ops[1].mem.base else ''
off = None
if base == 'x27':
off = ops[1].mem.disp
elif base in pending:
off = pending.pop(base) + ops[1].mem.disp
if off is not None:
text = pp.get(off, '')
sites = refs.get(off, [])
note = ' ; pp+0x%x %s [%d refs]' % (off, text, len(sites))
mark = ' <<<' if abs(ins.address - addr) < 4 else ''
print('%x: %-8s %s%s%s' % (ins.address, ins.mnemonic, ins.op_str, note, mark))
def build_index(sample, out):
"""B/BL target -> call sites, decoded arithmetically (no disassembler needed)."""
text_addr, blob = text_section(sample)
index = {}
words = struct.unpack_from('<%dI' % (len(blob) // 4), blob, 0)
for i, w in enumerate(words):
if w & 0xFC000000 in (0x14000000, 0x94000000):
imm = w & 0x03FFFFFF
if imm & 0x02000000:
imm -= 0x04000000
site = text_addr + i * 4
index.setdefault(site + (imm << 2), []).append(site)
with open(out, 'w', encoding='utf-8') as fh:
json.dump({hex(k): v for k, v in index.items()}, fh)
print('distinct targets : %d' % len(index))
print('call sites : %d' % sum(len(v) for v in index.values()))
print('written %s' % out)
def lookup_index(sample, path, addrs):
index = {int(k, 16): v for k, v in json.load(open(path, encoding='utf-8')).items()}
for spec in addrs:
target = int(spec, 16)
sites = sorted(index.get(target, []))
print('0x%x: %d caller(s)' % (target, len(sites)))
for s in sites[:80]:
print(' 0x%x' % s)
if len(sites) > 80:
print(' ... %d more' % (len(sites) - 80))
def main():
ap = argparse.ArgumentParser(
description=__doc__.split('\n')[0],
formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
ap.add_argument('sample', help='libapp.so')
ap.add_argument('addr', nargs='*', help='hex address(es) to disassemble around')
ap.add_argument('--pp', metavar='PP_TXT', help="decompiler pool listing (blutter pp.txt)")
ap.add_argument('--refs', metavar='REFS_JSON', help='index from dart_pprefs.py')
ap.add_argument('--range', nargs=2, metavar=('START', 'END'))
ap.add_argument('--before', default='0x180')
ap.add_argument('--after', default='0x220')
ap.add_argument('--build-index', metavar='OUT_JSON')
ap.add_argument('--index', metavar='CALLERS_JSON')
a = ap.parse_args()
if a.build_index:
return build_index(a.sample, a.build_index)
if a.index:
if not a.addr:
raise SystemExit('give at least one hex address to look up')
return lookup_index(a.sample, a.index, a.addr)
pp = load_pp(a.pp)
refs = load_refs(a.refs)
if a.range:
start, stop = int(a.range[0], 16), int(a.range[1], 16)
disasm_window(a.sample, start, 0, stop - start, pp, refs)
return 0
if not a.addr:
raise SystemExit('give an address, --range, --build-index, or --index')
for spec in a.addr:
disasm_window(a.sample, int(spec, 16),
int(a.before, 0), int(a.after, 0), pp, refs)
return 0
if __name__ == '__main__':
sys.exit(main())
scripts/dart_pool_strings.py
+273
View File
@@ -0,0 +1,273 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Recover string literals from a Dart AOT snapshot, and locate them by file offset.
WHY THIS EXISTS
---------------
Strings are the cheapest anchor for locating logic in a stripped AOT snapshot: field
names, endpoint paths and labels survive compilation even when every symbol name is gone.
Two things this gives you that a decompiler's own listing does not:
* **file offsets**, where the constant physically sits in the binary (what you need to
patch bytes, as opposed to a pool offset);
* **identifiers the decompiler's listing omits** -- measured on a real sample, most of
what this recovers was absent from the pool listing entirely. Treat the two as
complementary, not competitive.
THE FRAMING
-----------
[ tag ][ payload ] tag = 0x80 | (len << 1) | <two-byte flag>
* tag bit 7 set -> a string entry
* (tag >> 1) & 0x3F -> payload length in characters (this framing tops out at 63)
* the low bit -> nominally "payload is UTF-16LE"
ENCODING, AND WHY ONE-BYTE IS THE DEFAULT
-----------------------------------------
The two payload kinds are searched differently, and this is where most people lose time:
* **one-byte** payloads are ASCII / Latin-1 -> a **UTF-8 byte search finds them**
* **two-byte** payloads are **UTF-16LE** -> a UTF-8 search returns *zero hits*
So search ASCII as UTF-8 and CJK/non-Latin as UTF-16LE; forcing one encoding on both is
the classic false negative (`pitfalls.md` P25).
**But automatic two-byte detection does not work reliably.** The low tag bit is not a
dependable discriminator against ordinary data, and UTF-16LE decoding essentially never
fails -- *any* byte pair is a legal code unit -- so decoding success filters nothing. On a
real sample, two-byte output was ~10k entries of near-pure garbage that happened to form
long runs, while one-byte output was almost entirely genuine. Hence:
* default: one-byte only (reliable; verified against a decompiler listing on a real sample)
* `--two-byte`: opt in, and **verify every hit yourself**
* to find a *known* non-Latin literal, use `--find` -- a direct search is far more
reliable than enumerating candidates
WHY THE RUN LENGTH IS REPORTED
------------------------------
A per-byte scan for tag-like bytes hits constantly inside ordinary data; a multi-MB
snapshot yields hundreds of thousands of candidates. Most real string tables are laid out
consecutively, so a genuine entry usually begins a run of back-to-back entries (each
entry's tag+payload ends exactly where the next begins). `--min-chain` uses that to shed
the scattered tail.
Note what the measurement showed: raising this threshold barely changes one-byte
precision while steadily destroying recall. It is a **noise filter, not a confidence
score** -- do not tune it expecting accuracy to improve.
USAGE
-----
python dart_pool_strings.py libapp.so strings.tsv
python dart_pool_strings.py libapp.so strings.tsv --pp pp.txt # annotate cross-hits
python dart_pool_strings.py libapp.so strings.tsv --min-chain 2 --two-byte
python dart_pool_strings.py libapp.so --find /activateVipCode --find vipflag
python dart_pool_strings.py libapp.so --stats
OUTPUT
------
offset_hex <TAB> text <TAB> run <TAB> form [<TAB> in_pp]
`in_pp` is present with `--pp`: `1` means the decompiler's own pool listing contains this
exact string. Absence is **not** evidence of a false positive -- the decompiler's listing
is incomplete by design.
NOTES
-----
* `--max` is capped at 63 by the framing; longer literals are serialized differently and
will not appear here (a few hundred per sample at most).
* Never edit a string without counting its references first: a shared entry changes every
consumer at once (`references/dart-aot.md` section 10).
"""
import argparse
import collections
import re
import sys
TAG_STR = 0x80
BAD = re.compile(r'[\x00-\x08\x0b\x0c\x0e-\x1f]')
def decode_payload(raw, two_byte):
if two_byte:
if len(raw) % 2:
return None
try:
text = raw.decode('utf-16-le')
except UnicodeDecodeError:
return None
else:
try:
text = raw.decode('utf-8')
except UnicodeDecodeError:
return None
if BAD.search(text):
return None
return text
def extract(blob, min_len, max_len, want_two_byte):
"""Return {start: (end, text, form)} for every framed candidate of the wanted kind."""
n = len(blob)
cands = {}
for i in range(n - 2):
tag = blob[i]
if not (tag & TAG_STR):
continue
ln = (tag >> 1) & 0x3F
if not (min_len <= ln <= max_len):
continue
two = tag & 1
if two and not want_two_byte:
continue
end = i + 1 + (ln * 2 if two else ln)
if end > n:
continue
text = decode_payload(blob[i + 1:end], two)
if text is None or len(text.strip()) < min_len:
continue
cands[i] = (end, text, 'utf16le' if two else 'one-byte')
return cands
def run_lengths(cands):
"""For each candidate, how many back-to-back entries follow it (inclusive)."""
nxt = {s: cands[s][0] for s in cands}
lengths = {}
closed = set()
for start in sorted(cands):
if start in closed:
continue
seq = []
cur = start
guard = 0
while cur in nxt and cur not in closed:
seq.append(cur)
closed.add(cur)
cur = nxt[cur]
guard += 1
if guard > 1_000_000:
break
for i, item in enumerate(seq):
lengths[item] = len(seq) - i
return lengths
def load_pp(path):
"""Strings present in a decompiler pool listing, for cross-annotation."""
if not path:
return set()
out = set()
pat = re.compile(r'^\[pp\+0x[0-9a-f]+\] String: (.*)$')
for line in open(path, encoding='utf-8', errors='replace'):
m = pat.match(line.strip())
if not m:
continue
s = m.group(1).strip()
if len(s) >= 2 and s[0] == '"' and s[-1] == '"':
s = s[1:-1]
if s:
out.add(s)
return out
def histogram(lengths):
counts = collections.Counter(lengths.values())
for run in sorted(counts)[:10]:
print(' run=%-4d %d' % (run, counts[run]))
tail = sum(v for k, v in counts.items() if k > 10)
if tail:
print(' run>%-3d %d' % (10, tail))
def search(blob, needles):
total = 0
for needle in needles:
for label, enc in (('utf-8', needle.encode('utf-8')),
('utf-16le', needle.encode('utf-16-le'))):
offsets = []
start = blob.find(enc)
while start != -1 and len(offsets) < 8:
offsets.append(start)
start = blob.find(enc, start + 1)
if offsets:
total += len(offsets)
print('%-26s %-9s file offset %s' % (
needle, label, ' '.join(hex(o) for o in offsets)))
if not total:
print('no hits in either encoding. Try the shortest distinctive fragment -- text is\n'
'often assembled from parts, and a literal may only exist as fragments\n'
'(pitfalls.md P25).')
return 0
def main():
ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
ap.add_argument('sample', help='libapp.so or any artifact holding the snapshot')
ap.add_argument('out', nargs='?', help='output tsv')
ap.add_argument('--min', type=int, default=4, help='minimum payload length (default 4)')
ap.add_argument('--max', type=int, default=63,
help="maximum payload length; 63 is this framing's ceiling (default 63)")
ap.add_argument('--min-chain', type=int, default=3,
help='noise filter: minimum run of consecutive entries (default 3). '
'This does not improve accuracy -- see the module docstring.')
ap.add_argument('--two-byte', action='store_true',
help='ALSO emit UTF-16 candidates. Detection is unreliable; verify each.')
ap.add_argument('--pp', metavar='PP_TXT',
help='decompiler pool listing, to mark which strings it also contains')
ap.add_argument('--stats', action='store_true',
help='print the run-length histogram and exit')
ap.add_argument('--keep-isolated', action='store_true',
help='also write entries below --min-chain (noisy; triage only)')
ap.add_argument('--find', action='append', default=[], metavar='TEXT',
help='search a literal in BOTH encodings and print file offsets '
'(repeatable; the reliable way to locate a known literal)')
a = ap.parse_args()
blob = open(a.sample, 'rb').read()
if a.find:
return search(blob, a.find)
if not a.out and not a.stats:
raise SystemExit('give an output path, or --find / --stats')
cands = extract(blob, a.min, a.max, a.two_byte)
lengths = run_lengths(cands)
if a.stats:
print('candidates (two-byte included: %s): %d' % (a.two_byte, len(cands)))
print('run-length histogram:')
histogram(lengths)
return 0
pp = load_pp(a.pp)
kept = [s for s in cands if lengths[s] >= a.min_chain]
rows = kept + (sorted(s for s in cands if lengths[s] < a.min_chain)
if a.keep_isolated else [])
with open(a.out, 'w', encoding='utf-8') as fh:
fh.write('# offset_hex\ttext\trun\tform%s\n' % ('\tin_pp' if pp else ''))
for s in sorted(rows):
_end, text, form = cands[s]
flat = text.replace('\n', '\\n').replace('\t', ' ')
extra = ('\t%d' % (1 if text in pp else 0)) if pp else ''
fh.write('0x%06x\t%s\t%d\t%s%s\n' % (s, flat, lengths[s], form, extra))
one = sum(1 for s in kept if cands[s][2] == 'one-byte')
print('candidates : %d' % len(cands))
print('kept (run >= %d) : %d (one-byte %d / utf16le %d)'
% (a.min_chain, len(kept), one, len(kept) - one))
print('below threshold : %d%s' % (
len(cands) - len(kept), '' if a.keep_isolated else ' [--keep-isolated to keep them]'))
if pp:
hits = sum(1 for s in kept if cands[s][1] in pp)
print('also in --pp listing : %d / %d (absence is NOT a false positive: the'
% (hits, len(kept)))
print(' decompiler listing is incomplete by design)')
if a.two_byte:
print('WARNING: --two-byte detection is unreliable (see module docstring);')
print(' verify each UTF-16 hit before trusting it.')
print('written %s' % a.out)
return 0
if __name__ == '__main__':
sys.exit(main())
scripts/dart_pprefs.py
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Dart AOT object-pool reference index: pool offset -> the instructions that load it.
WHY THIS EXISTS
---------------
blutter's `pp.txt` tells you what is *in* the object pool (strings, types, closures,
field metadata) but not which code reads it. "Who uses this string" is the most frequent
question in Dart AOT work, and it needs an index you build once and query constantly.
Building that index with a general-purpose disassembler is the trap here: decoding every
instruction of a multi-MB `.text` with detail enabled and asking for per-instruction
register writes costs minutes of CPU and 1-2 GB of peak memory, and on a 16 GB host it is
indistinguishable from a hang -- no output, no progress, nothing to interrupt. Only three
encodings can reach the pool, so this decodes them from the raw 32-bit words instead:
seconds, and a flat memory profile.
Recognised forms (AArch64, little endian), with x27 = pool base:
add xD, x27, #imm12, lsl #12 -> xD = x27 + (imm12 << 12)
ldr xD, [xN, #imm12 * 8] -> 64-bit or 32-bit variant
ldur xD, [xN, #simm9] -> 64-bit or 32-bit variant
A reference is either a direct load whose base is x27, or a load consuming the register an
ADD just derived from x27. The compiler emits that pair back to back, so a small window
keeps it exact without full liveness analysis.
USAGE
-----
python dart_pprefs.py libapp.so pp_refs.json # build the index
python dart_pprefs.py --lookup pp_refs.json 0x1d1a8 0xc9d0
python dart_pprefs.py libapp.so out.json --window 5 # widen the ADD->LDR window
NOTES
-----
* Offsets are **pool** offsets -- the `pp+0x...` space used by pp.txt -- not file offsets.
Keep the two spaces apart in your notes; conflating them costs hours.
* One offset commonly has many reference sites. Count them before editing the entry:
a shared constant will change behaviour everywhere at once (see references/dart-aot.md).
"""
import argparse
import json
import struct
import sys
POOL_REG = 27 # x27: object-pool base in Dart AOT on arm64
DEFAULT_WINDOW = 3 # instructions an ADD-derived address may survive
def text_section(path):
"""Return (vaddr, bytes) of .text, parsing ELF64 section headers directly."""
blob = open(path, 'rb').read()
if blob[:4] != b'\x7fELF':
raise SystemExit('not an ELF file: %s' % path)
if blob[4] != 2:
raise SystemExit('not ELF64: %s' % path)
e_shoff = struct.unpack_from('<Q', blob, 0x28)[0]
e_shentsize = struct.unpack_from('<H', blob, 0x3A)[0]
e_shnum = struct.unpack_from('<H', blob, 0x3C)[0]
e_shstrndx = struct.unpack_from('<H', blob, 0x3E)[0]
def sh(i):
return struct.unpack_from('<IIQQQQ', blob, e_shoff + i * e_shentsize)
_, _, _, _, shstr_off, shstr_size = sh(e_shstrndx)
strs = blob[shstr_off:shstr_off + shstr_size]
for i in range(e_shnum):
name, _typ, _flags, addr, off, size = sh(i)
end = strs.find(b'\x00', name)
if strs[name:end].decode('ascii', 'replace') == '.text':
return addr, blob[off:off + size]
raise SystemExit('.text not found; pass --text with its virtual address')
def sx(value, bits):
m = 1 << (bits - 1)
return (value ^ m) - m
def scan(text_addr, blob, window):
refs = {}
pending = {}
words = struct.unpack_from('<%dI' % (len(blob) // 4), blob, 0)
for i, w in enumerate(words):
addr = text_addr + i * 4
# add xD, x27, #hi, lsl #12
if w & 0xFFC00000 == 0x91400000 and (w >> 5) & 0x1F == POOL_REG:
# Keep the shift out of the mask expression: `a & m << n` binds as
# `a & (m << n)` in Python, which silently produces a wrong base.
imm = (w >> 10) & 0xFFF
pending[w & 0x1F] = [imm << 12, i]
continue
ldr64 = w & 0xFFC00000 == 0xF9400000
ldr32 = w & 0xFFC00000 == 0xB9400000
ldur64 = w & 0xFFE00C00 == 0xF8400000
ldur32 = w & 0xFFE00C00 == 0xB8400000
if ldr64 or ldr32 or ldur64 or ldur32:
rn = (w >> 5) & 0x1F
rd = w & 0x1F
if ldr64 or ldr32:
disp = ((w >> 10) & 0xFFF) << 3
else:
disp = sx((w >> 12) & 0x1FF, 9)
if rn == POOL_REG:
refs.setdefault(disp, []).append(addr)
elif rn in pending:
base, seq = pending.pop(rn)
if i - seq <= window:
refs.setdefault(base + disp, []).append(addr)
pending.pop(rd, None) # this load overwrote rd
continue
if w & 0xFFC00000 in (0x91000000, 0x91400000):
pending.pop(w & 0x1F, None)
for r in [r for r, (_o, s) in pending.items() if i - s > window]:
del pending[r]
return refs
def build(so, out, window):
text_addr, blob = text_section(so)
refs = scan(text_addr, blob, window)
with open(out, 'w', encoding='utf-8') as fh:
json.dump({hex(k): v for k, v in refs.items()}, fh)
total = sum(len(v) for v in refs.values())
print('.text vaddr=0x%x bytes=%d' % (text_addr, len(blob)))
print('distinct pool offsets : %d' % len(refs))
print('reference sites : %d' % total)
for off in sorted(refs, key=lambda k: -len(refs[k]))[:8]:
print(' pp+0x%-8x %4d sites' % (off, len(refs[off])))
print('written %s' % out)
def lookup(path, offsets):
refs = {int(k, 16): v for k, v in json.load(open(path, encoding='utf-8')).items()}
for spec in offsets:
off = int(spec, 16)
sites = sorted(refs.get(off, []))
print('pp+0x%x: %d ref site(s)' % (off, len(sites)))
for s in sites:
print(' 0x%x' % s)
def main():
ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
ap.add_argument('sample', nargs='?', help='libapp.so (build mode)')
ap.add_argument('out', nargs='?', help='output json (build mode)')
ap.add_argument('--lookup', metavar='REFS_JSON', help='query an existing index')
ap.add_argument('--window', type=int, default=DEFAULT_WINDOW,
help='instructions an ADD-derived address may survive (default %d)'
% DEFAULT_WINDOW)
a = ap.parse_args()
if a.lookup:
rest = sys.argv[sys.argv.index('--lookup') + 2:]
if not rest:
raise SystemExit('give at least one hex pool offset')
lookup(a.lookup, rest)
return 0
if not (a.sample and a.out):
raise SystemExit('usage: dart_pprefs.py <libapp.so> <out.json>')
build(a.sample, a.out, a.window)
return 0
if __name__ == '__main__':
sys.exit(main())