# Undercurrent Kit: local-inference

> Generated 2026-07-12 | Status: breakout | Emergence: 51.3
> Platforms: reddit, hn

## What's Happening

"local-inference" has 10 mentions across 2 platform(s) in the last 7 days.
Velocity: 10.0x week-over-week.

## Key Signals

### 24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)
- **Source:** reddit | [link](https://reddit.com/r/LocalLLaMA/comments/1tcc7h5/24_toks_from_30b_moe_models_on_an_old_gtx_1080_8/)
- **Author:** mdda ([profile](https://reddit.com/u/mdda))
- **Scores:** originality:75% depth:82% psychosis:15%
- **Engagement:** 53 upvotes, 14 comments, 0 stars
- **Summary:** Practitioner demonstrates specific, reproducible performance tuning for running 30B+ MoE models on commodity 8GB GPUs using TurboQuant/RotorQuant KV cache compression and CPU offloading, achieving 20-24 tok/s with 128k context window.

**Implementation:**
Use llama.cpp with TurboQuant (K) and RotorQuant/turbo3 (V) KV cache quantization flags to fit 128k context in 8GB VRAM. Key parameters: --n-cpu-moe (offload MoE experts to CPU, typically 20-30 depending on model), --flash-attn for attention optimization, and Q4_K_M quantization for base weights. Start with Qwen 3.6 35B-A3B or Gemma 4 26B-A4B (A3B/A4B variants optimized for these techniques). Benchmark on target hardware to find optimal --n-cpu-moe threshold balancing CPU/GPU utilization. Monitor token throughput with different K/V quantization pairs (turbo4/turbo3, etc.).

### Playing One Night Werewolf (Gemma4 & Qwen3.6)
- **Source:** reddit | [link](https://reddit.com/r/LocalLLaMA/comments/1tcjtmt/playing_one_night_werewolf_gemma4_qwen36/)
- **Author:** Some-Cauliflower4902 ([profile](https://reddit.com/u/Some-Cauliflower4902))
- **Scores:** originality:75% depth:65% psychosis:20%
- **Engagement:** 8 upvotes, 1 comments, 0 stars
- **Summary:** Practitioner successfully orchestrates multiple quantized local LLMs as game agents in One Night Werewolf, sharing concrete implementation details around model selection, quantization strategies, and reasoning suppression.

**Implementation:**
Use llama.cpp with model switching capability to run multiple quantized models (Gemma 4 31B Q4/26B Q5, Qwen 3.6 27B Q5/35B Q4) as game agents. Assign each model a role (werewolf, seer, villager, troublemaker) via role cards, suppress extended thinking outputs on Qwen models via inference settings to prevent token waste, and maintain separate observation logs per agent. The builder custom-coded a UI for seamless mid-chat model switching; replicate using llama.cpp's server mode with batched requests or build on top of llama-cpp-python bindings for role-specific prompting and state management across agents.

### Built a fully offline suitcase robot around a Jetson Orin NX SUPER 16GB. Gemma 4 E4B, ~200ms cached TTFT, 30+ sensors, no WiFi/BT/cellular. He has opinions.
- **Source:** reddit | [link](https://v.redd.it/9v5pmv1rgb1h1)
- **Author:** CreativelyBankrupt ([profile](https://reddit.com/u/CreativelyBankrupt))
- **Scores:** originality:75% depth:65% psychosis:15%
- **Engagement:** 215 upvotes, 44 comments, 0 stars
- **Summary:** A practitioner demonstrates a fully offline robotics system running local LLM inference on Jetson Orin NX with tight latency constraints and multi-sensor integration—concrete hardware implementation with measurable performance metrics.

**Implementation:**
The builder integrated Gemma 4 E4B (or similar quantized model) on Jetson Orin NX SUPER 16GB, achieving ~200ms cached time-to-first-token through local inference optimization (likely vLLM or TensorRT-LLM for batching/KV-cache). With 30+ sensors, the architecture likely uses sensor fusion (camera, IMU, possibly lidar) feeding into the local model for real-time decision-making. To replicate: start with Jetson's official CUDA environment, containerize with Docker, use quantization (4-bit/8-bit) and inference framework optimization, implement local sensor polling without network dependency. Specific tools: vLLM, TensorRT-LLM, or Ollama for model serving; ROS2 for sensor/actuator coordination; consider Triton inference server for batching across sensors.

### Where do you personally draw the line with AI access, read-only, file edits, running commands, browsing for you? Why there?
- **Source:** reddit | [link](https://reddit.com/r/ClaudeAI/comments/1tcflgd/where_do_you_personally_draw_the_line_with_ai/)
- **Author:** CarolusX74 ([profile](https://reddit.com/u/CarolusX74))
- **Scores:** originality:72% depth:35% psychosis:15%
- **Engagement:** 8 upvotes, 51 comments, 0 stars
- **Summary:** Practitioner-driven discussion of trust boundaries in agentic AI systems, grounded in real friction points rather than hype—identifies a genuine open question about where developers should gate AI capabilities.

**Implementation:**
The post doesn't prescribe a specific implementation, but frames a decision tree for tooling: assess your risk tolerance at each autonomy level (read-only via APIs like Claude Files, direct edits via file-writing APIs, shell execution via process managers like `subprocess` with strict allowlists, web browsing via tools like Playwright, or full agentic loops via frameworks like LangGraph or AutoGPT-style patterns). Practitioners should prototype at the lowest required capability level, implement audit logs for file/command changes, use sandboxing (Docker, VMs, or OS-level process isolation) before enabling shell access, and establish human checkpoints before irreversible operations. The discussion itself is the value—reading the 51 comments will surface domain-specific risk models from Android devs, backend engineers, and security-conscious users.

### Open-source, self-updating wiki for your codebase
- **Source:** reddit | [link](https://reddit.com/r/ClaudeAI/comments/1tcjv9b/opensource_selfupdating_wiki_for_your_codebase/)
- **Author:** ElectronicUnit6303 ([profile](https://reddit.com/u/ElectronicUnit6303))
- **Scores:** originality:72% depth:35% psychosis:25%
- **Engagement:** 38 upvotes, 1 comments, 0 stars
- **Summary:** Developer built Almanac, a self-updating markdown wiki that extracts institutional knowledge from conversations with Claude Code and repo structure to reduce context re-explanation overhead for coding agents.

**Implementation:**
Almanac appears to work by parsing Claude Code/Codex conversations and repo files to automatically generate and maintain markdown documentation in the repository root. To implement or extend this: (1) establish a hook or API integration with Claude Code conversations to capture context explanations, (2) build a parser that extracts rationale patterns ("we tried X but backed it out because Y") into structured wiki entries, (3) run periodic scans of codebase structure and git history to auto-populate architectural decisions, (4) store the wiki as markdown so it's version-controllable and readable. The tool likely feeds this self-maintained wiki back into agent prompts as context, reducing token waste on repetitive explanations. Specific implementation details (whether it uses AST parsing, LLM-based extraction, or pattern matching) are not provided; check the GitHub repo for architecture choices around markdown generation and conversation integration.

### Looking for fast vision-capable local models that handle tool calls well (open-source app, want to add local support)
- **Source:** reddit | [link](https://reddit.com/r/LocalLLaMA/comments/1tcl96f/looking_for_fast_visioncapable_local_models_that/)
- **Author:** yaboyskales ([profile](https://reddit.com/u/yaboyskales))
- **Scores:** originality:72% depth:68% psychosis:15%
- **Engagement:** 1 upvotes, 0 comments, 0 stars
- **Summary:** Developer seeking local vision-LLM alternatives for sub-2s latency in a practical cursor-overlay app, with clear technical constraints that distinguish this from generic model-comparison posts.

**Implementation:**
The asker should evaluate LLaVA-NeXT (7B), Qwen2-VL (7B), and Phi-3.5-vision for local deployment, benchmarking time-to-first-token on typical screenshot inputs (~1MB) using vLLM or Ollama for inference serving. LLaVA with quantization (GGUF format via llama.cpp) and batch-size=1 can achieve 1-2s latency on consumer GPUs; Qwen2-VL is faster but requires more VRAM. For tool-calling, ensure the model supports structured outputs (JSON mode) natively or use outlines.dev for token-level constraints. Test on actual cursor screenshots at various resolutions to simulate real UX.

### Show HN: Find the best local LLM for your hardware, ranked by benchmarks
- **Source:** hn | [link](https://github.com/Andyyyy64/whichllm)
- **Author:** andyyyy64 ([profile](https://news.ycombinator.com/user?id=andyyyy64))
- **Scores:** originality:62% depth:58% psychosis:35%
- **Engagement:** 117 upvotes, 13 comments, 0 stars
- **Summary:** A practical benchmark comparison tool for matching local LLM models to specific hardware constraints, addressing a genuine pain point in the local inference workflow.

**Implementation:**
The tool appears to be a GitHub repo (Andyyyy64/whichllm) that benchmarks popular open-source LLMs across different hardware profiles and provides ranked recommendations. To use this, a practitioner would: (1) identify their hardware specs (VRAM, CPU, GPU type), (2) run or review the benchmark results in the repo, (3) cross-reference model performance metrics with their constraints, and (4) download the recommended model via Ollama, llama.cpp, or similar local inference frameworks. The value lies in having pre-computed benchmarks rather than running individual benchmarks yourself—though the repo description itself is minimal, the GitHub presence suggests working code and runnable comparisons. Practitioners should verify benchmarks match their exact hardware setup.

### we really all are going to make it, aren't we? 2x3090 setup.
- **Source:** reddit | [link](https://reddit.com/r/LocalLLaMA/comments/1tcf2dt/we_really_all_are_going_to_make_it_arent_we/)
- **Author:** RedShiftedTime ([profile](https://reddit.com/u/RedShiftedTime))
- **Scores:** originality:35% depth:45% psychosis:15%
- **Engagement:** 49 upvotes, 35 comments, 0 stars
- **Summary:** Practitioner documents tangible improvements in local LLM inference on dual RTX3090 hardware using club-3090 project after bug fixes, with specific setup comparisons but incomplete throughput data.

**Implementation:**
User reports running club-3090 (GitHub: noonghunna/club-3090) on dual RTX3090 setup, initially on WSL2 vs LM Studio with improved results. To replicate: clone club-3090 repo, deploy Sonnet-patched fixes for SSE session drops and tool-calling bugs, benchmark against LM Studio baseline. User mentions tokens/second metrics but the post cuts off—check GitHub repo for supported models, exact setup requirements (bare metal vs WSL2 tradeoffs), and VRAM constraints. Relevant for hobbyists targeting 24GB×2=48GB total VRAM budgets.

## People to Watch

- **mdda**
- **Spiderboyz1**
- **RedShiftedTime**
- **CarolusX74**
- **chocofoxy**
- **Some-Cauliflower4902**
- **ElectronicUnit6303**
- **yaboyskales**

## Prompt: Hand This to Claude Code

```
I want to explore "local-inference" based on these emerging signals from the AI community.

Key implementations I've found:
- 24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context): Use llama.cpp with TurboQuant (K) and RotorQuant/turbo3 (V) KV cache quantization flags to fit 128k context in 8GB VRAM. Key parameters: --n-cpu-moe (offload MoE experts to CPU, typically 20-30 depend...
- Playing One Night Werewolf (Gemma4 & Qwen3.6): Use llama.cpp with model switching capability to run multiple quantized models (Gemma 4 31B Q4/26B Q5, Qwen 3.6 27B Q5/35B Q4) as game agents. Assign each model a role (werewolf, seer, villager, troub...
- Built a fully offline suitcase robot around a Jetson Orin NX SUPER 16GB. Gemma 4 E4B, ~200ms cached TTFT, 30+ sensors, no WiFi/BT/cellular. He has opinions.: The builder integrated Gemma 4 E4B (or similar quantized model) on Jetson Orin NX SUPER 16GB, achieving ~200ms cached time-to-first-token through local inference optimization (likely vLLM or TensorRT-...
- Where do you personally draw the line with AI access, read-only, file edits, running commands, browsing for you? Why there?: The post doesn't prescribe a specific implementation, but frames a decision tree for tooling: assess your risk tolerance at each autonomy level (read-only via APIs like Claude Files, direct edits via ...
- Open-source, self-updating wiki for your codebase: Almanac appears to work by parsing Claude Code/Codex conversations and repo files to automatically generate and maintain markdown documentation in the repository root. To implement or extend this: (1)...

Help me:
1. Assess which of these approaches fits my current stack
2. Build a minimal working prototype of the most promising one
3. Identify what these practitioners learned that isn't in the docs yet
```

---
*Kit generated by [Undercurrent](https://undercurrent-dashboard.pages.dev). Trend detection is people detection.*