huggingface-local-models

# Hugging Face Local Models Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with `llama-cli` or `llama-server`. ## Default Workflow 1. Search the Hub with `apps=llama.cpp`. 2. Open `https://huggingface.co/?local-app=llama.cpp`. 3. Prefer the exact HF local-app snippet and quant recommendation when it is visible. 4. Confirm exact `.gguf` filenames with `https://huggingface.co/api/models//tree/main?recursive=true`. 5. Launch with `llama-cli -hf :` or `llama-server -hf :`. 6. Fall back to `--hf-repo` plus `--hf-file` when the repo uses custom file naming. 7. Convert from Transformers weights only if the repo does not already expose GGUF files. ## Quick Start ### Install llama.cpp ```bash brew install llama.cpp winget install llama.cpp ``` ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp make ``` ### Authenticate for gated repos ```bash hf auth login ``` ### Search the Hub ```text https://huggingface.co/models?apps=llama.cpp&sort=trending https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending ``` ### Run directly from the Hub ```bash llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M ``` ### Run an exact GGUF file ```bash llama-server \ --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \ --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ -c 4096 ``` ### Convert only when no GGUF is available ```bash hf download --local-dir ./model-src python convert_hf_to_gguf.py ./model-src \ --outfile model-f16.gguf \ --outtype f16 llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M ``` ### Smoke test a local server ```bash llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M ``` ```bash curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{ "messages": [ {"role": "user", "content": "Write a limerick about exception handling"} ] }' ``` ## Quant Choice - Prefer the exact quant that HF marks as compatible on the `?local-app=llama.cpp` page. - Keep repo-native labels such as `UD-Q4_K_M` instead of normalizing them. - Default to `Q4_K_M` unless the repo page or hardware profile suggests otherwise. - Prefer `Q5_K_M` or `Q6_K` for code or technical workloads when memory allows. - Consider `Q3_K_M`, `Q4_K_S`, or repo-specific `IQ` / `UD-*` variants for tighter RAM or VRAM budgets. - Treat `mmproj-*.gguf` files as projector weights, not the main checkpoint. ## Load References - Read [hub-discovery.md](references/hub-discovery.md) for URL-first workflows, model search, tree API extraction, and command reconstruction. - Read [quantization.md](references/quantization.md) for format tables, model scaling, quality tradeoffs, and `imatrix`.