Can Inkling-Small Run Privately? The 180 GB VRAM Catch

July 31, 2026

Thinking Machines released Inkling-Small with open weights and local runtimes, but its official quantized deployment still needs at least 180 GB of aggregated VRAM.

Inkling-Small can run privately on infrastructure you control, but “Small” does not mean laptop-sized. Thinking Machines released full open weights for the 276-billion-parameter model on July 30, 2026. The official vLLM recipe says the smallest validated NVFP4 serving path needs at least 180 GB of aggregated VRAM; the BF16 path needs at least 600 GB. Trying it in the hosted Tinker Playground is a different privacy boundary.

Watch The 30-Second Summary

Watch this video on YouTube

Checked July 31, 2026: OpenVeil inspected Thinking Machines’ release, both official Hugging Face checkpoints, their weight-index metadata, the current vLLM and SGLang recipes, Thinking Machines’ privacy notice, and same-day local-AI discussion.

Who This Article Is For

This article is for:

• people searching for Inkling-Small local hardware requirements
• privacy-conscious users who equate open weights with private AI
• developers deciding between self-hosting and a hosted playground or API
• teams that want local text, image, or audio processing
• buyers wondering whether “12B active” means the model fits like a normal 12B model
• anyone comparing a managed private AI workspace with a local model stack

The short answer is:

You can make an Inkling-Small workflow local, but the active-parameter count does not describe how much model data must be stored. The official quantized checkpoint is about 171 decimal GB, the official runtime calls for at least 180 GB of VRAM, and privacy still depends on the entire application—not just where the weights sit.

What Is Confirmed

Thinking Machines Released The Full Inkling-Small Weights

Thinking Machines’ July 30 announcement calls Inkling-Small an “efficient open-weights model” and says the full weights are available. The company describes it as a Mixture-of-Experts model with:

• 276 billion total parameters
• 12 billion active parameters per token
• text, image, and audio input
• text output
• up to a one-million-token context window
• variable reasoning effort
• an Apache 2.0 license on the published checkpoints

The official Hugging Face model card lists local deployment through SGLang, vLLM, TokenSpeed, Unsloth, and Hugging Face tooling. It also distinguishes those paths from the Tinker Playground, Tinker API, and third-party inference providers.

That distinction matters. Open weights make self-hosting possible. They do not make every place you can try the model local.

The Official Checkpoints Are Much Larger Than 12B

Inkling-Small activates 12 billion parameters for each token, but it has 276 billion parameters in the full mixture. Sparse activation can lower inference compute without shrinking the complete checkpoint to the size of a dense 12B model.

OpenVeil read the current model.safetensors.index.json metadata for both official repositories on July 31:

The file size and the runtime floor are related but not interchangeable. Serving also needs memory for the runtime, model state, caches, request context, multimodal inputs, and concurrent work.

The Current Validated Serving Recipes Are Multi-Accelerator Or Datacenter-Class

The current vLLM Inkling-Small recipe gives concrete hardware guidance:

• NVFP4 requires at least 180 GB of aggregated VRAM.
• The documented NVFP4 paths use one B300 or GB300, or two B200, GB200, or H200 accelerators depending on the mode.
• BF16 requires at least 600 GB of aggregated VRAM.
• The documented BF16 paths use four or eight high-memory datacenter accelerators.

The SGLang Inkling-Small cookbook provides verified deployment matrices, multimodal serving, tool calling, reasoning controls, cache offloading, and long-context options. At the time of this check, it said Inkling-Small support had merged into SGLang’s main branch but was not yet in a pip release.

That is a real local-deployment path. It is not a typical consumer-PC path.

Third-Party Quantizations Can Lower The Storage Floor

Same-day community tooling already offers smaller GGUF quantizations. The Unsloth Inkling-Small GGUF repository includes variants whose complete split-file totals range from roughly 70 GiB for an aggressive one-bit quantization to roughly 152 GiB for common four-bit variants, plus larger formats.

Those community formats can make CPU offload or large unified-memory systems possible. They do not establish the same performance, quality, multimodal behavior, long-context capacity, or support level as Thinking Machines’ official BF16 or NVFP4 checkpoints.

Quantization is a trade:

• less storage and memory pressure
• potentially broader hardware access
• possible accuracy or stability loss
• different runtime support
• slower generation when substantial work is offloaded to system memory or disk
• a separate provenance and update path from the official checkpoint

“Someone loaded it” and “this is practical for my workload” are different claims.

The Release Has Immediate Public Attention

The same-day r/LocalLLaMA release discussion passed 470 votes during OpenVeil’s scan. The most visible reaction was the mismatch between the word “Small” and a 276B model. Other current posts documented experimental GGUF runs and compared its reasoning results with much smaller local models.

That attention is useful evidence of the user question. It is not independent proof of speed, quality, or privacy on every setup.

What Is Still Unclear

There Is No Single Consumer Hardware Answer

Official documentation gives validated datacenter-class configurations, while community reports cover rapidly changing GGUF builds, CPU offload, and unified-memory machines. Actual usability depends on:

• the exact quantization
• available VRAM and system RAM
• memory bandwidth
• prompt length and KV-cache size
• whether image and audio inputs are enabled
• how many requests run at once
• how much of the model is offloaded
• the runtime version and model support
• acceptable tokens per second

A machine that can technically load a checkpoint may still be too slow for comfortable interactive chat.

Hosted Tinker Retention And Training Details Are Not Fully Specified On The General Privacy Page

Thinking Machines offers Inkling-Small through the hosted Tinker Playground and API. Its privacy notice says the company collects device, location, log, and usage data; may use information to provide, analyze, and improve services; may disclose information to vendors for cloud, storage, security, and analytics; and keeps personal information as long as reasonably necessary for stated purposes.

The public notice does not provide an Inkling-Small-prompt retention schedule or a model-specific statement saying whether Tinker prompts, files, images, or audio are used for training. Users with sensitive workloads should obtain a product-specific answer instead of inferring one from the availability of downloadable weights.

Open Weights Do Not Describe The Whole Runtime

A local checkpoint can still sit inside an application that uses:

• remote web search
• cloud speech recognition
• hosted image or document parsing
• telemetry and crash reporting
• automatic model downloads
• remote moderation
• third-party tools or MCP servers
• public media URLs
• an internet-reachable inference endpoint

The SGLang examples accept image and audio as either embedded data or HTTP URLs. Passing a public URL is not the same as keeping the source file inside a closed local system. Tool calls and web search create additional outbound paths even when inference itself remains local.

What “Run Privately” Should Mean

Use this five-layer test before calling any Inkling-Small deployment private.

1. Weight Layer

Are the exact model, tokenizer, configuration, multimodal components, and quantization stored locally? Pin the repository revision and verify the files you actually downloaded.

2. Inference Layer

Does generation run entirely on hardware you control? A local client that sends prompts to Tinker or another inference provider is still hosted inference.

3. Application Layer

Where does the chat UI keep history, uploads, audio, images, caches, traces, and error reports? Local weights do not override the application’s storage design.

4. Tool And Network Layer

Can the runtime perform web searches, fetch URLs, call tools, install packages, or reach third-party APIs? Inspect actual outbound connections rather than relying on the word “local.”

5. Operations Layer

Who can access the machine, logs, backups, model server, and network? A self-hosted service exposed without authentication can be less private than a carefully controlled hosted product.

Private AI is a property of the full data path. Open weights are one component of that path.

Local Inkling-Small Versus Hosted Access

This is not a ranking. Each option solves a different problem.

What This Release Does Not Prove

Inkling-Small’s release does not prove that:

• every Inkling-Small chat is local
• a 12B active count means 12B-model memory requirements
• a downloadable checkpoint is automatically secure
• a community quantization matches the official model
• hosted Tinker use has the same data path as self-hosting
• local inference prevents web search, tools, telemetry, or public URLs from sending data out
• benchmark results predict performance on your hardware or private documents
• an internet-exposed local server is safe

Thinking Machines also says its scores use defined harnesses and testing conditions. Treat them as model-evaluation results, not as proof of privacy or of a particular real-world deployment experience.

A Practical Pre-Download Checklist

Before downloading hundreds of gigabytes, answer these questions:

1. Which checkpoint will you use? BF16, official NVFP4, or a third-party GGUF quantization?
2. Can your hardware hold it with headroom? Count model weights, runtime overhead, KV cache, multimodal inputs, and concurrency.
3. What speed is acceptable? “Loads successfully” is not an interactive-performance target.
4. Will prompts ever leave the machine? Check search, URLs, tools, telemetry, crash reporting, and remote APIs.
5. Where will chat history live? The model does not define the UI’s persistence rules.
6. How will you secure the inference server? Bind it carefully, require authentication where appropriate, segment the network, and patch the runtime.
7. Can you reproduce the setup? Pin model and runtime revisions, preserve hashes, and document dependencies.
8. What is your failure plan? Account for corrupted downloads, runtime regressions, model updates, and hardware limits.

The decision should start with the privacy requirement, not with the model’s label.

Where OpenVeil Fits

OpenVeil does not offer Inkling-Small and is not a fully local or offline model runner.

OpenVeil is a paid privacy-focused AI chat workspace with browser-local chat history, no server-side chat-history record for private sessions, and web search, uploads, voice, and image tools where enabled. Active requests may still be processed by OpenVeil and necessary AI, hosting, routing, search, upload, security, billing, and infrastructure providers.

Choose self-hosted Inkling-Small when controlling inference hardware and the complete runtime is worth the equipment, setup, security, and maintenance cost. Consider OpenVeil when you want hosted private AI chat without operating a 180 GB-plus model-serving stack, while accepting the active-processing boundary in the OpenVeil privacy policy.

For the broader decision, compare private AI chat versus local AI, see why using Ollama does not make every mode local, and review the earlier Kimi K3 local-hardware reality check.

Frequently Asked Questions

Is Inkling-Small Open Source?

The model checkpoints are published as Apache-2.0 open weights. “Open weights” is the most precise description because downloadable weights do not necessarily include the complete training dataset, training code, and reproducible training process.

Can Inkling-Small Run Locally?

Yes. The official model card lists local SGLang, vLLM, TokenSpeed, Unsloth, and Hugging Face paths. The current official vLLM recipe requires at least 180 GB aggregated VRAM for NVFP4 or 600 GB for BF16.

Why Does A 12B-Active Model Need So Much Memory?

Inkling-Small is a sparse Mixture-of-Experts model with 276B total parameters. It routes each token through a subset of experts, reducing active compute, but the complete expert set still has to be stored or streamed by the serving system.

Can Inkling-Small Run On A Laptop?

The official validated configurations are not normal laptops. Aggressive third-party quantizations and CPU or unified-memory offload may run on unusually large-memory systems, but speed, quality, context, and multimodal support can differ. Verify a specific quantization and workload before buying hardware.

Does Using The Tinker Playground Keep Prompts Local?

No. Tinker is a hosted service, so prompts and attached media must be transmitted for processing. Its data handling should be evaluated under Thinking Machines’ terms, privacy notice, and any product-specific commitments.

Does Downloading The Weights Guarantee Privacy?

No. It makes local inference possible. The chat application, logs, uploads, search, tools, public URLs, telemetry, backups, and network exposure determine whether the complete workflow stays local.

Is Inkling-Small More Private Than OpenVeil?

A carefully isolated self-hosted Inkling-Small stack can keep inference inputs on infrastructure you control, which OpenVeil does not claim. OpenVeil offers a different tradeoff: hosted convenience with browser-local private-chat history and no server-side chat-history record, while active requests still require necessary provider processing.

The Bottom Line

Thinking Machines made Inkling-Small genuinely self-hostable by releasing the full weights and supporting local runtimes. But the name can hide the most important deployment fact: 12B active does not mean 12B-sized. The official NVFP4 checkpoint is about 171 GB and its validated vLLM path needs at least 180 GB of aggregated VRAM; the BF16 path needs at least 600 GB.

Inkling-Small can be private when the weights, inference, application, tools, storage, and network all stay under your control. Trying the same model through Tinker or another provider is hosted AI. If you do not want to operate that infrastructure, use a hosted service whose privacy boundaries you can understand—and do not confuse hosted convenience with fully local processing.

Sources

• Thinking Machines: Introducing Inkling-Small
• Official Inkling-Small BF16 model card
• Official Inkling-Small BF16 weight index
• Official Inkling-Small NVFP4 model card
• Official Inkling-Small NVFP4 weight index
• vLLM Inkling-Small deployment recipe
• SGLang Inkling-Small cookbook
• Thinking Machines privacy notice
• Unsloth Inkling-Small GGUF repository
• r/LocalLLaMA release discussion