You can't tail -f a user's Snapdragon.
TL;DR
Qualcomm has a new on-device LLM runtime and SDK, GenieX, and it leaves it to app developers to turn its per-call telemetry into fleet-wide visibility. WildEdge fills that gap, aggregating TTFT, decode speed, and hardware fallback across an install base without the prompt or completion ever leaving the device.
LLM product engineering reads like a slow retreat from "just call the API." Apple Intelligence routes between on-device models and Private Cloud Compute rather than defaulting to a server round trip. Google's Gemini Nano runs directly on Pixel and Samsung hardware, escalating to cloud Gemini only when it has to. Practitioners are increasingly making the same case in public: the 3B–30B (billion parameters) "Goldilocks zone" of models is now good enough that latency, privacy, connectivity, and per-inference cost stop being reasons to default to a server, and start being reasons not to. Some production teams are reportedly routing the majority of their LLM traffic on-device already, escalating only the requests that genuinely need it. Aggressive quantization is pushing that zone upward, too. PrismML's Bonsai 27B compresses a 27B model down to a 3.9 GB, phone-sized 1-bit variant while claiming to retain 90% of full-precision performance, which is exactly the kind of result that keeps moving "local" and "capable" closer together instead of trading one for the other.
None of that runs through one runtime, either. The on-device landscape has splintered along hardware lines almost as fast as the models have shrunk to fit it. llama.cpp is the mature, community-driven default, minimal and portable across CPU and GPU with no intermediate representation. Meta's ExecuTorch exports PyTorch models straight from torch.export() through an ahead-of-time compiled .pte graph, no ONNX or TFLite detour required. Apple's MLX is an array framework built specifically for Apple Silicon, running eager or graph-mode across the M-series CPU, GPU, and ANE. And Qualcomm AI Engine Direct (QNN) is the low-level counterpart on Snapdragon, trading portability for direct access to the Kryo CPU, Adreno GPU, and Hexagon NPU as heterogeneous compute. Each one optimizes for a different slice of hardware, which is exactly why a runtime that can move between them, rather than committing to just one, matters.
Snapdragontorch.export(), no intermediate ONNX/TFLite, with delegation backends.gguf (primary), .bin (legacy).pte (native), .onnx / .pt (imported).npy, safetensors, MLX-specific weights.dlc (native), .onnx / .tflite (imported)Qualcomm is making the same bet from the silicon side, and saying so in public: their own OnQ blog has argued directly that shifting inference from the cloud to the phone can cut AI costs, and framed OpenAI shipping gpt-oss-20b to run on-device on Snapdragon as evidence the frontier itself is moving to the edge. Snapdragon's Hexagon NPU has been getting faster and more efficient generation over generation specifically so that the 3B–30B "Goldilocks zone" runs comfortably on the chip already in the phone, rather than a server somewhere. GenieX is Qualcomm stepping in front of that shift with an actual runtime and SDK, rather than leaving it to app developers to wire up llama.cpp and QNN by hand: one Kotlin/C API that dispatches to llama.cpp or Qualcomm AI Engine Direct depending on hardware and model, across Android, Windows ARM64, and Linux ARM64.
But moving inference onto the device inherits the same visibility problem the industry-wide shift creates: the moment a model runs inside someone else's app on someone else's silicon, you lose the thing server-side ML teams take for granted: a log line per request. You can't tail -f a user's Snapdragon. That gap is exactly why this post exists: it's where WildEdge picks up what GenieX deliberately leaves on the table.
Further reading on the shift:
- On-Device LLM – Future is EDGE AI (Navay Singh Gill, LinkedIn)
- Shifting AI inference from the cloud to your phone can reduce AI costs (Qualcomm OnQ blog)
- OpenAI's gpt-oss-20b: Its first open-weights reasoning model to run on devices with Snapdragon (Qualcomm OnQ blog)
- The On-Device LLM Revolution: Why 3B-30B Models Are Moving to the Edge (Edge AI and Vision Alliance)
- On-Device LLMs in 2026: What Changed, What Matters, What's Next (Edge AI and Vision Alliance)
- Apple Intelligence
- On-Device LLM Revolution (Quadric blog)
Qualcomm's SDK (GenieX) and its multiple runtimes
GenieX is Qualcomm's runtime and SDK for running LLMs and VLMs locally on-device (described as "the community version of Qualcomm GENIE"), with bindings for Android (Kotlin), Python, a CLI, an OpenAI-compatible server, and Docker, all dispatching to the same underlying engine. The rest of this post focuses on the Android SDK specifically, since phones are the most common case for on-device inference, and on that SDK, "on-device" still isn't a single inference path. The same generate() call can end up running on llama.cpp (CPU or GPU, on any Android device, against a GGUF model pulled from Hugging Face) or on Qualcomm AI Engine Direct (NPU-only, Snapdragon-specific, against a per-chipset bundle pulled from AI Hub), and which one actually ran depends on the compute unit and runtime id resolved at load time, not just on what the developer requested. LlmCreateInput is the data class that makes that request: it's passed to LlmWrapper.Builder to load a model, and bundles the model path together with the two fields that decide dispatch, runtime_id and compute_unit:
val input = LlmCreateInput(
model_path = modelPath,
config = ModelConfig(),
runtime_id = RuntimeIdValue.LLAMA_CPP.value,
// Left null: picks the best-performing compute unit automatically.
compute_unit = null,
)runtime_id isn't something GenieX picks for you: the developer has to know upfront which runtime a given task and model should use, since the SDK doesn't auto-select llama_cpp versus qairt based on the file it's handed. In practice the choice depends on where you land on the tradeoff between the two. GGUF models through llama_cpp will use Hexagon NPU automatically when it's available. Its NPU performance has also gotten close to QAIRT's. But it degrades faster than QAIRT as the KV cache grows, so long-context or long-running sessions are where that gap reopens. For now, QAIRT still posts the best raw NPU performance, so if peak throughput on a fixed-size prompt matters more than convenience, pin qairt and a chipset-specific AI Hub bundle instead of llama_cpp. That gap isn't just throughput, either: Qualcomm advertises the Snapdragon 8 Elite's Hexagon NPU as delivering 45% better performance per watt generation-over-generation, so a qairt build compiled against that NPU path is also the one positioned to make the most of that power efficiency, which matters for any sustained on-device inference workload, not just burst latency. Either way, it's worth benchmarking on your own model and target chipset with geniex infer against both a llama.cpp and a qairt build before committing one to production. Picking qairt also means confirming the device is actually a Snapdragon in the first place, and there's no public API on LlmWrapper/LlmCreateInput for that check either: GenieX's own model-manager does it internally by reading the Android system properties ro.soc.manufacturer and ro.soc.model (see detect.rs), which is the same technique an app would have to fall back on to gate its own qairt versus llama_cpp choice.
Picking runtime_id and compute_unit once, at build time, only answers the question for the device you tested on. Across a real install base, the same choice raises another operational question you can't answer from a single benchmark: what fraction of sessions actually resolve to a compute_unit of NPU versus falling back to CPU, does one runtime post worse TTFT or more truncated completions than the other, and are load failures concentrated on one runtime/compute_unit combination? GenieX's runtime_id/compute_unit fields carry exactly the raw signal needed to answer that fleet-wide. Pairing GenieX with wildedge-android is what turns "multiple runtimes" from an invisible implementation detail into a dimension you can slice dashboards by:
handle = wildEdge.registerModel(
File(modelPath).nameWithoutExtension,
ModelInfo(
modelName = File(modelPath).nameWithoutExtension,
modelSource = "local",
modelFormat = "gguf", // this is where "multiple runtimes" becomes visible: "qairt" here means a different runtime dispatched entirely
inputModality = InputModality.Text,
outputModality = OutputModality.Generation,
)
)From ProfilingData to Product Telemetry
GenieX already normalizes away the multi-runtime split at the output layer. Whichever runtime actually handled a call, llama_cpp or qairt, it reports back through the same ProfilingData shape: time-to-first-token, prompt/decode timing, prompt and generated token counts, prefill and decode speed, real-time factor, stop reason, structured and per-call on every platform GenieX targets, including the Android SDK's ProfilingData data class:
data class ProfilingData(
val ttftMs: Double,
val promptTimeMs: Double,
val decodeTimeMs: Double,
val promptTokens: Long,
val generatedTokens: Long,
val prefillSpeed: Double,
val decodingSpeed: Double,
val realTimeFactor: Double,
val stopReason: String,
)This is genuinely excellent inference telemetry data: it's exactly the raw material you need. What GenieX intentionally doesn't do is decide what happens to it next. It hands you the struct after every call and gets out of the way: there's no network client bundled into the SDK, no phone-home behavior, nothing to opt out of. That's the right default for an on-device inference runtime.
A ProfilingData instance sitting in a log statement tells you about one inference, on one device, at one moment. It doesn't tell you:
- Whether TTFT on Snapdragon 8 Elite NPU builds is trending up after your last model swap.
- Whether decode speed craters on a specific subset of devices (older HTP driver, thermal throttling, a chipset your app's model catalog never accounted for).
- What fraction of sessions hit
stopReason: "length"instead of a natural stop (a proxy for truncated, unsatisfying answers). - Whether load failures cluster on non-Qualcomm hardware where an NPU codepath was requested by mistake.
Answering any of that requires turning single-run counters into a queryable, aggregatable dataset across your install base, without shipping the prompt or the completion anywhere. That's the specific job the WildEdge does, and it's also exactly the gap wildedge-android is built to fill: it "tracks latency, confidence, drift, and hardware metrics without ever sending raw inputs."
Here's what the glue code between WildEdge and GenieX looks like, picking up right where the LlmCreateInput snippet above left off:
lateinit var handle: ModelHandle
val modelPath = File(applicationContext.filesDir, "models/llama-3.2-3b-instruct-q4_k_m.gguf").absolutePath
// llmWrapper is the LlmWrapper built from the LlmCreateInput above, and handle is
// the ModelHandle returned by wildEdge.registerModel for this model.
val genStart = System.currentTimeMillis()
llmWrapper.generateStreamFlow(prompt, GenerationConfig()).collect { result ->
if (result is LlmStreamResult.Completed) {
val profile = result.profile
handle.trackInference(
durationMs = System.currentTimeMillis() - genStart,
outputMeta = GenerationOutputMeta(
timeToFirstTokenMs = profile.ttftMs.toInt(),
tokensIn = profile.promptTokens.toInt(),
tokensOut = profile.generatedTokens.toInt(),
tokensPerSecond = profile.decodingSpeed,
stopReason = profile.stopReason,
).toMap(),
)
}
}The Missing Accelerator
There's a field conspicuously absent from that flow: trackLoad accepts an accelerator parameter, but nothing here ever sets it. That's not an oversight. The Android bindings don't expose which compute unit (CPU/GPU/NPU/hybrid, resolved to a concrete device_id like HTP0/GPUOpenCL/NPU) a created LlmWrapper/VlmWrapper is actually running on. geniex_resolve_device computes this once at create time to dispatch the call correctly, but that resolved value isn't surfaced back out: it isn't stored on the C++ plugin classes, and there's no getter API exposing it, in core or in either plugin. Reporting the request that was made instead of the request that was honored would misattribute NPU-tagged inference metrics to devices that silently fell back to CPU, so leaving it unset is the honest choice given what GenieX's public API can actually attest to.
That's not a blocker, though: wildedge-android already collects deviceModel = "${Build.MANUFACTURER} ${Build.MODEL}" and the device's available accelerators independently of GenieX, so chipset- and hardware-level slicing keeps working even without GenieX handing back which compute unit ran a given call.
Back to the bigger picture
Stepping back, GenieX itself is doing exactly what this post opened with: it's billed as "an on-device Gen AI inference runtime built for Qualcomm platforms," promising to "Run Any Gen AI Model On Device" across phones, PCs, IoT, and automotive, dispatching across Hexagon NPU, Adreno GPU, and CPU. It's Qualcomm shipping the on-device half of the industry-wide shift away from "just call the API," not asking app developers to assemble it from llama.cpp and QNN themselves, and doing it well: a runtime and SDK are exactly the right thing for Qualcomm to own here.
What a runtime SDK isn't in the business of shipping is the fleet-wide observability layer that sits above it: whether TTFT holds up across chipsets, whether load failures cluster on specific hardware, whether sessions are actually landing on NPU. Any team moving inference on-device needs that layer from somewhere once a runtime like GenieX is embedded in an app on a device they don't control, and it's a natural complement rather than a gap in GenieX itself. That's the layer WildEdge sits on top of GenieX to provide, turning per-call ProfilingData structs into a dataset you can query across an install base, without the prompt or completion ever leaving the device.
Links
- GenieX (GitHub)
- llama.cpp (GitHub)
- GenieX (Qualcomm AI Hub)
- wildedge-android (GitHub)
- WildEdge
detect.rs(GenieX chipset detection)- Snapdragon 8 Elite Mobile Platform (Qualcomm specs)