--- title: "Token efficiency" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Token efficiency} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- gptr logo Keep data and computation in R, and send models enough context to choose useful actions. For inline work, an object around 4 GB can remain in the same R session across turns. The model receives size-limited descriptions, previews and tool output that may include data values, while its R tool computes on the object. No repeated export and reload cycle is required; R copies, computation time and request latency still depend on the workflow. Workers receive serialized context rather than sharing the parent workspace. Four practices help: - Refer to live objects with a compact description rather than printing large tables into a prompt. The model can compute on those objects with the `r` tool. - Compose several deterministic operations in one R call and return a bounded result. - Reuse a session's stable prompt and growing conversation when the provider supports prompt caching. Changing models, tools, compaction or rewind can change that prefix. - Inspect usage and set budgets before starting a costly workflow. The executed examples use fake providers. Their request counts and estimates demonstrate the interfaces; they are not measurements of a real model's speed, quality or bill. ## Inspecting the prompt ``` r library(gptr) view = gptr_prompt(preset = "minimal") attr(view, "tool_names") #> [1] "read" "r" "edit" "write" view$sections #> name tier tokens #> 1 preamble T0 42 #> 2 tools T0 86 #> 3 rules T0 348 #> 4 modes T0 84 #> 5 context T0 152 view$total_tokens #> [1] 1846 ``` `gptr_prompt()` inspects the system sections and tool definitions with estimated token counts; printing it shows their text. It does not query a provider. The `standard` preset describes most capabilities as functions in `peter$`, with the core model-visible `r`, `read`, `edit` and `write` tools and `ask` when applicable. `view$total_tokens` already counts the tool array and a preview of the first message; your prompt and the objects you attach add to it. ## Object descriptions and result size ``` r writeLines(gptr_describe(iris)) #> 150 x 5, 6.8 KB #> $ Sepal.Length min 4.3, median 5.8, max 7.9; no NA #> $ Sepal.Width min 2, median 3, max 4.4; no NA #> $ Petal.Length min 1, median 4.35, max 6.9; no NA #> $ Petal.Width min 0.1, median 1.3, max 2.5; no NA #> $ Species 3 levels, top: setosa (50), versicolor (50), virginica (50); no NA writeLines(gptr_describe(iris, budget = 60)) #> 150 x 5, 6.8 KB #> Sepal.Length:dbl Sepal.Width:dbl Petal.Length:dbl Petal.Width:dbl Species:fct ``` `gptr_describe()` summarizes class, shape and values within a budget (150 tokens by default). Packages can register methods for their own classes. Generative-model context normally uses these descriptions and names; **System 1 questions send their input states**, so their egress and size rules differ. See [Typed decisions](system-one.html). ``` r work = new.env() session = peter("Compare mpg and weight by cylinder count.", mtcars, model = fake, envir = work, mode = auto) work$slopes #> 4 6 8 #> -5.647025 -2.780106 -2.192438 ``` The calculation runs in R once the model requests it. Return a small table, statistic or plot rather than a full intermediate object. Long `r` output keeps a bounded head and tail; the full spilled text can be read through `peter$out(id)` when an output ID was supplied. ## Reported usage and estimated components ``` r gptr_usage(session, by = "model") #> group requests input output cache_read cache_write cost #> 1 fake/fake-1 2 3382 48 0 0 0 #> #> # 1 row #> 2 requests, 3.4k tokens in, $0.0000 ledger = gptr_usage(session, detail = TRUE) aggregate(tokens ~ component, data = ledger, FUN = sum) #> component tokens #> 1 attached 132 #> 2 environment 150 #> 3 images 0 #> 4 other 156 #> 5 project 0 #> 6 t0 2720 #> 7 t1 618 #> 8 tool_results 53 #> 9 tools 1926 #> 10 transcript 70 #> 11 workspace 58 ``` The usage summary contains provider-reported input, output, cache reads/writes and cost. `by = "session"`, `"agent"`, `"model"` or `"route"` groups it; child requests are included once. `gptr_usage()` without a session also includes the process-level System 1 log. The detailed ledger estimates components such as tools, workspace, attached objects, transcript and tool results. Its sum can differ from reported usage because tokenizers, provider accounting, images and fake-provider behavior differ. Unknown reported usage or cost stays `NA`, not zero. A local model's known zero metered charge does not imply zero hardware cost or latency. ## Budgets and partial results ``` r budget_error = tryCatch( peter("Summarize mpg.", model = small, budget = list(tokens = 500)), gptr_error_budget = function(e) e ) class(budget_error)[1] #> [1] "gptr_error_budget_tokens" budget_error$session$status #> [1] "budget" ``` The estimated input for the first request already exceeds 500 tokens, so this call stops before that request. A top-level run's defaults are 2 million tokens and 5 US dollars; `budget = list(tokens = ..., cost = ..., turns = ...)` supplies limits for that call. Change settings deliberately when different defaults are needed: ``` r gptr_config(budget = list(tokens = 20000, cost = 1, turns = 10), .scope = "session") session = peter("Review this analysis.", budget = list(tokens = 12000, turns = 6)) ``` Checks happen before requests using estimated next-input tokens and usage already recorded. The next output or final provider charge can exceed a remaining allowance; unknown usage cannot establish a hard monetary ceiling. A budget is a stopping policy, not a billing reservation. At the console the UI can ask to extend it; without a person present the call signals a budget condition carrying the partial session as `e$session`. Inspect that session's status, transcript, workspace and usage before deciding whether to continue. Teams and fan-outs share the root call's budget; launching more children does not give each one a fresh root allowance. With `gptr_parallel()`, member `budget =`, `tools =` and per-run `.opts` are not applied. Set the intended limits through the governing settings or enclosing run, cap concurrency and inspect the returned children's statuses. See [Teams and background work](teams-and-background.html). ## Compaction At the chat prompt, `/compact` replaces older model context with a summary. Add a focus when particular details must survive, for example `/compact retain the model comparison and next steps`. Automatic compaction follows the configured context and cold-cache thresholds; see [Configuration](configuration.html). Compaction can make a model request and add usage. It keeps the R objects already in your session, while the model no longer sees every older turn verbatim. Preserve important decisions in project files and named R objects. A changed request prefix can also affect provider cache reuse. ## Three different caches | Mechanism | What is reused | Inspect | |---|---|---| | Provider prompt cache | A repeated request prefix | `gptr_usage(session)` | | System 1 answer cache | Matching input, question and model | Metadata and `gptr_cache()` | | Replay / System 2 cache | Recorded code and saved answer | `gptr_blocks(path)`, `gptr_cache()` | A stable prefix makes provider cache reuse possible. Providers decide whether a cache hit occurs, subject to their support, minimum size, expiry and pricing; gptr does not guarantee a hit after the first request. Compaction, image elision, changed resources or rewind can break the prefix. Document replay makes no model request for a usable recorded call but still runs its R code. Clearing an answer cache changes what can be reused. ## Explicit ways to reduce context These live-call examples assume a configured model. For unattended use of a remote provider, save its egress acknowledgment first, as in [Language models](language-models.html). `context = "none"` still requires that consent because project instructions and environment details remain; offline providers and local loopback routes are exempt (Ollama only while local-only inference is on). ``` r peter("Summarize the model fit.", fit, .opts = list(context = "names")) peter("Find the mean of x. Return one number.", .opts = list(context = "none")) peter("Compute the mean.", x, .opts = list(preset = "minimal")) ``` `context = "names"` lists workspace names and uses headers for explicitly attached objects. `"none"` omits both built-in workspace and attached-object descriptions; the R bindings still exist and can be used by tool code, as `x` is in the second example. Prompt text, tool results, images and other context blocks can still reach the model. System 1 input states use a separate encoding route. [Execution model](execution-model.html) explains these boundaries. `preset = "minimal"` reduces the tool/prompt surface; it does not authorize actions or remove required safety checks. `{identifier}` in a literal prompt can insert a short atomic value when that is sufficient. For a long workflow, continue one session, keep intermediate data in R and request concise reports between stages. Use a new session or an explicit fork when a separate analysis is intended.