inliver233
0ea42265b7
Generalized from a long hardened-target engagement. Additions target the failure
modes that cost the most time and are not yet covered; nothing here is sample-specific.
New discipline
- SKILL.md: a third override rule -- deliver in the environment the request requires.
A root-only or live-instrumentation result can look like success while failing the
actual requirement (unrooted, installable, self-contained). Also adds two classify
questions: what form the deliverable must take, and which architecture/library is
actually executing.
- long-task-discipline.md: "the most expensive drift" -- a constraint table (privilege,
modification form, ABI, network, persistence, distribution), the checkpoint question,
and how to report a privileged workaround as a fallback rather than a deliverable.
Plus "bound every wait": timeouts on every command, budgets calibrated from
measurement so slow and hung are distinguishable, and sampling an observable you can
inspect instead of sleeping blind.
- environment.md: look at the screen instead of driving blind; which of the two screen
evidence kinds is trustworthy (an empty accessibility tree is common for
runtime-rendered UI, where the image is primary rather than a fallback); emulator
console control and adb recovery; preflight before every experiment block.
- native-and-so.md: what the device claims vs what the package manager chose vs what is
actually mapped; translation layers and their consequences for hooks and native
checks; fat-APK ABI selection.
- runtime-data.md: when the app rewrites your edit (server refresh vs local
regeneration), why chmod/chown cannot work because the app deletes and recreates the
file, and the immutable-attribute fix; decoding a value that only looks encrypted.
- framework-runtimes.md: string encoding traps (a UTF-8 scan of a UTF-16 snapshot
returns nothing and reads as "the strings were stripped"), and non-ASCII obfuscated
identifiers -- resolve by shape, not by name.
- pitfalls.md: P23 the fix lives in a file the app rewrites; P24 the artifact changed
but the wrong one is executing; P25 "search found nothing" treated as "not there".
New scripts (parameterized, path-agnostic, all with explicit timeouts)
- preflight.py environment check: device, root, ABI/translation, clock skew, leftover
proxy/forwards, dead device server, chosen ABI, free space
- lib_map.py what is actually mapped into a live process: path, base, architecture
from the ELF header, and whether it came from the APK or was
materialized at runtime
- blob_decode.py search (don't guess) the framing of a stored value, then re-encode the
edited payload with the same parameters
- snap.py bounded burst screenshots + control-tree capture, with a stall detector
and an explicit verdict on whether the tree is usable
All four were exercised against a live device while writing: preflight correctly
flags multi-device ambiguity and detected the ARM translation layer; lib_map read real
ELF headers and identified the translation shim; blob_decode round-tripped a
rotation+deflate+base64 blob at realistic size in ~0.2s; snap detected a stalled screen
across three hashes.
190 lines
9.6 KiB
Markdown
190 lines
9.6 KiB
Markdown
# Runtime data — DataStore, SharedPreferences, SQLite, protobuf
|
|
|
|
Sometimes the cheapest fix is not code at all, but a value in app data. And sometimes that is exactly the wrong fix, because it does not survive a fresh install (`pitfalls.md` P11).
|
|
|
|
## Decide first: data or code?
|
|
|
|
| Question | Answer |
|
|
|---|---|
|
|
| Does the deliverable need to work after a clean install? | Then it must be **code**, not data |
|
|
| Are you iterating on your own device? | Data edits are a fast way to test a hypothesis **before** writing the patch |
|
|
| Is the value written by the app itself (a token, a cache, a counter)? | Data — and the server may re-assert it |
|
|
|
|
**Recommended workflow:** use a data edit to *prove the hypothesis* (cheap), then implement the same effect in code (durable), then verify with a clean install.
|
|
|
|
## Where app data lives
|
|
|
|
```
|
|
/data/data/<pkg>/ (== /data/user/0/<pkg>)
|
|
├── files/
|
|
│ ├── datastore/*.preferences_pb AndroidX DataStore (protobuf)
|
|
│ └── ...
|
|
├── shared_prefs/*.xml SharedPreferences
|
|
├── databases/*.db SQLite (Room, etc.)
|
|
└── <sdk caches>/ crash logs, download state, ad configs
|
|
```
|
|
|
|
Read with root:
|
|
```bash
|
|
adb shell "su -c 'ls -la /data/data/<pkg>/files/datastore/'"
|
|
adb shell "su -c 'xxd /data/data/<pkg>/files/datastore/<name>.preferences_pb | head -20'"
|
|
```
|
|
|
|
## AndroidX DataStore (protobuf)
|
|
|
|
Format:
|
|
```proto
|
|
PreferenceMap { map<string, Value> preferences = 1; }
|
|
Value {
|
|
oneof value {
|
|
bool boolean = 1; float float = 2; int32 integer = 3; int64 long = 4;
|
|
string string = 5; StringSet string_set = 6; double double = 7; bytes bytes = 8;
|
|
}
|
|
}
|
|
```
|
|
|
|
**Encoding an entry requires TWO tag levels:**
|
|
```
|
|
outer (map field 1, wire type 2) : 0A <len(inner)>
|
|
inner (one map entry) : 0A <len(key)> <key> 12 <len(value)> <value>
|
|
value payload (e.g. int64) : 20 <varint> # field 4, wire type 0
|
|
value payload (e.g. int32) : 18 <varint> # field 3, wire type 0
|
|
value payload (e.g. bool) : 08 <0x00|0x01> # field 1, wire type 0
|
|
value payload (e.g. string) : 2A <len> <utf8> # field 5, wire type 2
|
|
```
|
|
|
|
Omitting the **outer** tag produces a file the app cannot deserialize, and the failure is usually a **stack-less crash** (`pitfalls.md` P8). Use `scripts/datastore_inject.py` rather than hand-rolling it.
|
|
|
|
Procedure:
|
|
1. `am force-stop <pkg>` (DataStore caches in memory and writes back)
|
|
2. Write the file (as root), preserving ownership — `cp -f` over the existing file keeps its owner
|
|
3. `restorecon` if SELinux is enforcing
|
|
4. Start the app and observe
|
|
|
|
Verify by reading the file back with `xxd` before launching.
|
|
|
|
## SharedPreferences (XML)
|
|
|
|
Simple XML. Edit with the app stopped. Types are explicit (`<boolean>`, `<int>`, `<long>`, `<string>`). Same ownership rules.
|
|
|
|
## SQLite
|
|
|
|
```bash
|
|
adb shell "su -c 'sqlite3 /data/data/<pkg>/databases/<db>.db \".tables\"'"
|
|
adb shell "su -c 'sqlite3 ... \"select * from <table> limit 5;\"'"
|
|
```
|
|
Useful for: auth sessions (tokens), user profile caches (often a raw JSON blob), and any server state the app persists. A JSON column often contains the exact server payload — the fastest way to learn field names.
|
|
|
|
## Ownership — the silent killer
|
|
|
|
The app runs as its own uid. A file written by root with the wrong owner is unreadable, and directory permissions are `drwx------`.
|
|
|
|
```bash
|
|
uid=$(adb shell "dumpsys package <pkg> | grep userId=" | tr -dc '0-9')
|
|
adb shell "su -c 'chown -R $uid:$uid /data/user/0/<pkg>'"
|
|
adb shell "su -c 'restorecon -R /data/user/0/<pkg>'"
|
|
```
|
|
Symptom of getting this wrong: crash in a database-init path right after launch, often `Cannot open database ... Directory ... doesn't exist`.
|
|
|
|
Note: after every reinstall the app's uid **increments**, so a restored data directory needs this again.
|
|
|
|
## Introducing a value that does not exist yet
|
|
|
|
If the key is absent, the app uses a default. Two options:
|
|
|
|
1. **Add the key** with the exact protobuf shape above.
|
|
2. **Confirm the default first** — sometimes removing a key already yields the behavior you want (e.g. clearing an expiry so it reads as "not set").
|
|
|
|
## Making the fix durable
|
|
|
|
If the effect must ship inside an APK, move it into code. Patterns, in order of preference:
|
|
|
|
- **Patch the read path** so the value is always the desired one. Find where the Flow/getter is built and make it emit a constant.
|
|
- **Patch the decision**, not the data: find the comparison that consumes the value and make it resolve the way you want.
|
|
- **Do not patch the generic encoder/boxer** used by the whole app (`pitfalls.md` P6).
|
|
|
|
Reference case: a promo popup was gated by `<key>_expires_at`. Writing a large value via DataStore suppressed it on the test device, but a fresh install lost it. The durable fix replaced the dedicated map-lambda that produces the value so it always yields a far-future timestamp — a single-purpose class, safe to patch, no effect on the shared serialization helpers.
|
|
|
|
## When the app rewrites your edit
|
|
|
|
A very common sequence: you change the value, relaunch, and the value is **back** — often byte for
|
|
byte identical to what it was before. That is not a failed write. It is the app re-establishing the
|
|
value from a source of truth you have not touched yet.
|
|
|
|
Two causes look identical and need different fixes:
|
|
|
|
| Cause | How to tell | Fix |
|
|
|---|---|---|
|
|
| The app **re-fetches from the server** and re-persists | take the app offline and relaunch: does your value survive? | lock the file, block the refresh request, or move the change into code |
|
|
| The app **rewrites the file on every start** from its own defaults | your value does not survive even with the network down | the value is not authoritative — patch the read path instead |
|
|
|
|
Do the offline test first. It costs one launch and separates the two cases.
|
|
|
|
### Making a data edit stick (in order of durability)
|
|
|
|
1. **Immutable file attribute** — the reliable, reversible way to stop a rewrite:
|
|
```bash
|
|
su -c "chattr +i <path/to/prefs.xml>"
|
|
su -c "lsattr <path/to/prefs.xml>" # expect the 'i' flag, e.g. -----i-------
|
|
# to undo:
|
|
su -c "chattr -i <path/to/prefs.xml>"
|
|
```
|
|
Verify with `lsattr`, not by assuming the command worked.
|
|
|
|
**Why permissions do not achieve this.** Tightening the mode or changing the owner looks like the
|
|
obvious move and does not work, because the app does not open-and-write the existing file — it
|
|
**deletes the file and creates a new one**. The new file is created by the app, with the app's own
|
|
mode and owner, so a restrictive `chmod` or a `chown root` is simply not inherited. The immutable
|
|
attribute is enforced by the filesystem against the delete itself, which is why it holds.
|
|
|
|
2. **Keep the app offline** so the refresh source is unreachable. Effective for a test, and
|
|
sometimes acceptable in use — but be explicit about it, because a fix that only works offline
|
|
fails the "works under normal conditions" constraint. See the deliverable-drift section in
|
|
`long-task-discipline.md` before presenting this as a result.
|
|
|
|
3. **Block the refresh request** at the runtime or network layer — durable while your instrumentation
|
|
is running, and requires you to have identified the endpoint (`references/server-api.md`).
|
|
|
|
4. **Move the change into code** — patch the read path or the decision. This is the only form that
|
|
ships inside an APK (`§Making the fix durable` above).
|
|
|
|
### Two cautions about locking
|
|
|
|
- **Verify the feature, not the value.** Blocking a write the app depends on can make it misbehave
|
|
or fail loudly. After locking, exercise the feature you care about — "the file still has my value"
|
|
is not the same as "the app still works".
|
|
- **A lock is device state; it does not travel with an APK.** If the recipe requires it, record it as
|
|
an environment requirement, so nobody later mistakes it for something baked into the artifact.
|
|
|
|
## Encoded values: do not assume "encrypted"
|
|
|
|
Preference values that carry server configuration are frequently stored as a single opaque string
|
|
rather than readable XML. Before concluding the value is encrypted, check whether it is merely
|
|
**framed**: a common shape is `base64( rotate( deflate( json ) ) )`, where the cyclic rotation exists
|
|
precisely so a naive decode fails.
|
|
|
|
`scripts/blob_decode.py` searches the parameter space instead of guessing it — outer encoding
|
|
(base64 / base64url / hex), rotation offset, and compression type — and reports every combination
|
|
that yields a structured document, plus the exact parameters to re-encode your edited payload:
|
|
|
|
```bash
|
|
# pull the value straight out of a preferences XML and decode it
|
|
python scripts/blob_decode.py --prefs-xml prefs.xml --name <key> # (see --help for exact name)
|
|
|
|
# after editing the decoded payload, rebuild the value with the winning parameters
|
|
python scripts/blob_decode.py --encode --file decoded.bin --outer base64 --inner raw --cut <N>
|
|
```
|
|
|
|
The rotation search is cheap — a few tens of thousands of candidates resolve in well under a second
|
|
— so there is no reason to guess. **Distinguish framing from real encryption before spending time
|
|
on a key:** a framed blob becomes structured the moment you remove the framing, whereas a keyed blob
|
|
stays random-looking. If it stays random, treat it as opaque and move to the code path that consumes
|
|
it.
|
|
|
|
## Cautions
|
|
|
|
- Editing data while the app is running is unreliable: in-memory caches win. Force-stop, edit, then launch.
|
|
- The server may overwrite your value on next sync — see *When the app rewrites your edit* above.
|
|
- Do not edit a token you do not own and expect it to be accepted; tokens are validated server-side (`references/server-api.md`).
|
|
- A restored data directory needs ownership fixed again after every reinstall, because the uid increments.
|