How many slurp juices to make a coat ape?

Presumably they’d have to expand their compute capacity astronomically to provide the paid service to everyone

but then again, they could also shitify the service

With AI specifically, cacheing like that is a LOT harder, one of the major reasons why the cost of compute has only gone up over time instead of going down as the tech gets more mature.

A lot of NDAs are gonna be worthless when all the chats leak

God you joke but I have tried some Stable Diffusion models and seriously question what the training set was becsuse even trying to get “Milfs” it would spit out some questionable shit.

Also, 16gb VRAM? You’ll be able to load a better model like https://huggingface.co/mradermacher/Skyfall-31B-v4-i1-GGUF which is a little stronger than the ones I linked in the guide. If the “i1-Q4_K_S” is too large then try the “i1-IQ4_XS” quant.

Probably try it offloading just the down tensors (top option) in the guide. Make sure your KV Batch size is 1024 (or better) so the context gets offloaded on the GPU faster to cut down on response times. Otherwise everything else in the guide is good for you. If you find you have a little bit of VRAM space at 16k context and 1024 batch size try upping the context a little until your VRAM is like 15GB utilized or better.

The guide is the same you’ll just need to follow the install instructions for Linux for Koboldcpp and SilltyTavern that is documented on the github pages linked.

The performance guide is the same as well as the link on how to set up the “character” to act as a GM. If you run into any hitches maybe I can help but to be clear I haven’t touched Linux in 10 years.

You’d download ollama, and then get the model via ollama install I believe. Although the 90% as good mark is pushing it, as open weight models below 32b parameters (what you could reasonably run on those machines) benchmark around 40% less than Opus 4.6 for software, and the difference is night and day for general reasoning.

huggingface client

This is what billionaires want us to take seriously. Let me just fire up the slimslam bazooper and have it connect to the sillynilly butterball API

If you are running at home as a hobby then just use Koboldcpp and maybe SillyTavern if you want extra functionality. In the former you can offload down and potentionally up tensors to save VRAM space if needed. For models it depends on need.

A 24-31B model is generally more than fine for most @home use cases, and they are quite “smart”, though that doesn’t mean anything in regards to AI. It’s a vibe, basically. A 32gb RAM / 8GB VRAM can use a 24B model to generate about 5 tokens per second, which is fine for an agent who is designed to give you short replies to answer questions.

You’ll most likely want to grab a GGUF quantization from hugging face, yes. Any 4bit quant is fine, really. The merges are all quasi-abliterated models for people who want slutty AI girlfriends/boyfriends. The models directly from companies like GLM7 or Kimi or whatever are more standard and generally run more efficiently.

People in development are likely going to want a 70-100B model. Claude, I think, is a 100B model. You can run those on about 64gb of ram and 32gb of VRAM.

If you want settings for Koboldcpp I can give you the rundown on how to optimize.

It depends massively on what hardware you have. I’ve heard good things about glm 4.7 flash and it’s easy enough to run. Also depends on what you want to use it for.

If you are talking about multi-modal stuff then locally the best way to do it is still separately.

If you just want to run a local adventure bot “game master” then a Koboldcpp and SillyTavern is the way to go. I’m a tabletop gaming nerd so this is what I use it for when I’m sitting on the couch sometimes.

Simple guide for Windows:

1. Go grab Koboldcpp Normal exe for NVidia or the NOCUDA if you have AMD and want a smaller download.
2. Go download Nodejs and install that.
3. Download SillyTavern {Go to the green “Code” box in the top-right and click it then select download.}
4. Extract SillyTavern into a folder that has no spaces (Spaces in the folder name will break their install script)
5. Browse to the /SillyTavern-Launcher/ folder and run the Installer.bat file and run through the simple install.
6. Go download a jailbroken model. If your RAM situation looks desperate like 16/6 then get something like Rocinante or if you have a little more RAM like 32/12 then go with something like Goetia
7. Run Koboldcpp.exe. Load the model you downloaded. Press Launch. Wait for it to say its waiting for connection at endpoint. By default it’ll give you the local web launch page and you can actually test things there.
8. If everything looks good and SillyTavern isn’t already loaded then launch that from the Launcher.bat.
9. In SillyTavern you click the plug button at the top and and make sure you can connect to the endpoint. Might have to change the port from 5000 to 5001.
10. Technically, you are done here and it’s just a matter of finetuning your settings and setting up a character card to be a game master.

Finetuning:

1. First rule of performance is the more of the model you can fit into VRAM the better. ALL is best.
2. The second rule is to offload ffn_ tensors. Down, then up, then if you are desperate, gate.
3. Third rule, if you are REALLY desperate is to offload KV cache. At this point your models is running SLOW.
4. What I would do is download the model. Load it in Kobold. Set your GPU layers to 99 and your context to 16384 (Minimum for text roleplayers)
5. Click the left Hardware tab. Check Use MMAP, Use mlock, High Priority. Up the threads to one lower than your logical CPU core count. Turn off Launch Browser if you want.
6. Click the left Context tab. Check No BOS Token (SillyTavern already does this).
7. Save that config and let the model load. Then use Task Manager to look at your GPU VRAM usage in the performance tab. Are you using all of it?
8. If “no” then close it and give yourself more context. The ultimate goal for text roleplay geeks like us is 32k. You can also up the Batch Size in the Hardware tab to 1024 or 2048 but realize this scales exponentially the KV cache and hogs VRAM. More KV cache helps your prompt load quicker, which starts to become a problem with larger cache sizes. Your speeds on a fully VRAM loaded model should be like 50tps or better (tokens per second)
9. If “yes” then run some prompts through the assistant or Seraphina default character cards and check your speeds. They are probably slow.
10. Close the Koboldcpp window and relaunch it. Load your previous config (you saved it didn’t you?) and plug in a tensor offload command (bottom of my comment) into the Override Tensors field in the Hardware Tab. Then return to step 7. Keep doing this until you get speeds of at least 5tps and can have a decent context size (16k+).
11. If you still can’t get enough speed because of VRAM constraints try a 3bit quant of the model you got or go to a less complex, small model. Sorry bubs, welcome to the club. We VRAM poor in this house.
12. If you are REALLY desperate you can try offloading the KV cache in the Hardware tab but the slowdown is MASSIVE.

Setting up a “character” card and “persona” in SillyTavern and some other settings

1. Use this guide to start
2. Don’t use his character template, it’s kind of bad. Instead consider using a guide like the character foundry template. I’ll share a character I made to format characters this way below. Works pretty well. Feed it as much information you want about the character and it makes a nice entry for the Lorebook.
3. Adjustments I made are to reduce the context to anywhere from 160-220 depending on the model. I don’t restrict TopK (leave at zero for off). I use the XTC settings over the repetition setting because it works better on longer adventure (like 500+ responses). If you are running a long adventure and the model starts omitting pronouns and words like “the” then you probably have an issue with your repetition settings being too strict or incorrect.
4. Learn to love the LoreBook. I wrap my supporting “characters” in the lorebook with <character>{}</character> and that seems to help models with that. There are also extensions to help with other lorebook entries like this WREC, though I use an inline summarizer which I like.

Character Creator:

You will create a character card considering the context provided by {{user}}. Adhere to the following format strictly, restructuring the context to fit this format:

# {The Character name}
- **Age:** {age in years}
- **Gender:** {male, female, nonbinary, or transgendered}
- **Height:** {height in feet and inches}
- **Race/Species:** {the race or species of a character such as human or elf or other fantasy species}

### Appearance
{Describe the person's basic physical appearance.} {Describe briefly how their body has shaped their life, if it gives them the attention they like or dislike and if they have insecurities about their body.} {Write a sentence or two about their clothing preferences and what kind of image they are trying to project about themselves.}

### Background
{Write a sentence describing where this character grew up and how that culture shaped them.} {Write a few sentences about experiences they had growing up that may have shaped who they are.}

### Personality
{A sentence or two detailing where this character feels they are in their life and if they are satisfied or not with their current situation.} {Describe a few things the character enjoys such as music, movies, games, or other activities.} {Describe briefly the character's innermost desire.} {Describe briefly the character's innermost fear.} {Describe a lie the character believes about themselves.} {Describe one irrational thing the character does or believes.}

### Speech
[{Write about how this person talks considering their accent and the slang and idioms they use. Give examples.}]

### Relationships
{Write a sentence or two about how this character expresses attraction or fondness for others.} {Write a sentence or two about boundaries this character has in relationships.} {Create one issue that makes relationships or intimacy difficult for this character.}

Some basic Tensor offloads, try the top one first, second one if you still need VRAM space, last is a big slowdown:

blk\.\d+\.ffn_(down*)=CPU
blk\.\d+\.((ffn_down*)|(ffn_up*))=CPU
blk\.\d+\.((ffn_down*)|(ffn_up*)|(ffn_gate*))=CPU

You’re right. Getting a 24B model locally isn’t going to be as powerful as a 600B model, for sure. You’re also right thar they don’t think. They absolutely don’t.

But the local ones are pretty powerful and can do a lot more than most think. Even some simple vibe coding can be done with local AI. I think for the average gamer type if they wanted to mess with it local is more than enough tbh.

I’ve observed the opposite, most people I see google things take the gemini response as-is and don’t even click into the first result

Edit: that is, if they don’t already actively use an LLM to seek out the answer in the first instance

blueoctopus@feddit.uk

I am crab

hmmm

This is the kind of drunk, unhinged posting I live for

… it has made that easier but you still have to argue with the bugger to get it to do what you want, I pay for nothing. I am the problem not the solution. I am crab.

right there with you. i took on a job at a non-profit to ally my conscious, but it meant that copilot was the closest thing to a technically inclined colleague and i’ve started to like it.

Uncritical support for Sentinelese for turning you into a human pincushion.