Offline LLM SDK Architecture
Four-layer architecture for on-device LLM inference on Android using llama.cpp, JNA, and Kotlin business modules.
Overview
The Offline LLM SDK is designed as a production-grade on-device inference solution for Android. It wraps llama.cpp through a C wrapper and JNA binding, with all business logic implemented in pure Kotlin.
Four-Layer Architecture
Layer 1: App UI
The UI layer contains three main screens:
- ChatActivity — Multi-turn streaming conversation
- ModelManagerActivity — GGUF model download and management
- CustomerServiceActivity — Offline knowledge-base Q&A
These screens only interact with the SDK facade — they never touch JNA handles, model pointers, or vector computations directly.
Layer 2: OfflineLlmSdk Facade
A global singleton that aggregates five business modules:
init()— Initializes the C backend viallm_backend_init()releaseAll()— Unloads models, releases vector resources, clears caches- Read-only module accessors:
sessionManager,ragEngine,customerService,modelRuntime,downloadManager
Layer 3: Core Business Modules
| Module | Responsibility |
|---|---|
| LlmSessionManager | Multi-session isolation, SQLite persistence, sliding-window truncation |
| RagEngine | Local document vector retrieval |
| CustomerServiceEngine | Q&A flow orchestration |
| ModelRuntimeManager | Model load/unload, KV cache management |
| ModelDownloadManager | GGUF shard download and checksum validation |
Layer 4: JNA + Native
- LlmWrapperLib.kt — JNA interface mapping
- TokenCallback — Streaming token bridge from C to Kotlin
- llm_wrapper.c — Atomic C API wrapping llama.cpp
- llama.cpp — GGUF loading, tokenize, decode, embedding
Key Design Decisions
Session persistence via Room/SQLite — Chat history survives app restarts. Sliding-window truncation prevents OOM on long conversations.
KV cache lifecycle — Switching sessions clears the underlying KV cache. App backgrounding triggers session caching and KV release; foregrounding restores context.
Streaming callbacks — Native layer emits tokens through a callback bridge, marshaled to the main thread for UI updates.
Module Dependencies
OfflineLlmSdk → SessionManager, RagEngine, CustomerService, ModelRuntime, DownloadManager
CustomerService → SessionManager, RagEngine, ModelRuntime
RagEngine → ModelRuntime
SessionManager → ModelRuntime
All modules → JNA → llm_wrapper.c → llama.cpp
This architecture separates concerns cleanly: UI knows nothing about native code, business modules know nothing about llama.cpp internals, and the native layer contains zero business logic.