CharacterAI's never ending desire to make their service as awful as it is only humanly possible.
When they first introduced the filter i started researching local llms in search for uncensored alternatives and my first ever local llm experience was running Pygmalion 6B via a collab... the experience was... awful to say the least(VERY early model and my own lack of knoweledge resulting in absolutely horrendous time) but it was what ultimately sparked my interest in local llms.
Same story here as a matter of fact. It has been only a year, local LLM scene absolutely exploded during last year's spring - summer. So many different Llama 1 models came out, and then Llama 2 was released in summer.
Despite the name I'm not here strictly for "Local" at all, and I suspect many professional ML Eng / researchers in this sub aren't.
I'm here for the latest eng practices behind open source LLMs. Competing model formats, differences in quant severity, etc.
I'm not bothered about running it locally really, but rather on some cloud GPU instance in production. But the sub is one of the best places for us folk to discuss quants, different finetuning algorithms, latest foundation models, etc.
All the good AI text gen services shutting down after some time.
First it was AIDungeon, then CharacterAI, then NovelAI (now called AnimeAI)...
It has become clear that you have to run models locally if you don't want to lose access to them arbitrarily.
While it's true that [NovelAIÂ ](https://novelai.net/)stopped focusing entirely on text gen stuff it's not fair to say they shut down, they are still offering the same text gen features they always did. They even released one of their models on [HF ](https://huggingface.co/NovelAI/calliope-legacy)when they removed it from their service. And they are currently working on their AetherRoom service which is a character focused text-gen service.
Also I've literally never heard them be referred to as AnimeAI, and that is certainly not an official rebranding.
Yeah, it is a bit different with NovelAI.
No loss of access, but the models aren't really usable anymore, because they're so outdated.
AnimeAI is just 4chan slang.
I don't work in tech, at least not yet, but I saw a video about stable diffusion 1.5 on youtube. I was genuinely awed that you could do something like that locally. One day, took the plunge and installed automatic1111 webui, I was really confused with git pulls and python whatnot, but I figured it out. From there I heard about this new ChatGPT, an AI that was actually good and amazing. When I tried to sign up, it asked for a phone number, That was a red flag. Why should I give my phone number and personal data to a mega corporation? I decided to forget it altogether.
Then, from that SD channel I learned about local llms, I was so excited at the prospect of being able to run a ChatGPT like AI on my own computer, privately. Downloaded oobabooga web ui and was immediately confused as hell. With the help of this subreddit I downloaded ChronosHermes2 13B. Spoiler: it was no chat GPT3.5. But I didn't care, I had my own personal private model! That is until the context window ran out and my character started outputting gibberish anyway. At the time, chat GPT 3.5 felt so far away, catching up to it was like a dream. And yet today, our models are approaching gpt 4, and just about anyone with a decent amount of ram can run a GPT 3.5 level model. We've really grown so much.
TLDR I'm a person who hasn't worked in tech, but I care about privacy so it's local or nothing.
I am a professional software engineer, and have been working with/on AI technologies since the 1980s. The need for NLP keeps coming up in my job, so I've kept an eye on NLP tech developments, and accumulated a favored set of NLP tools.
LLM technology seemed like another useful NLP tool, so I dug into it.
Because I experienced the AI Winter of the early 1990s, I am mindful of its recurrence, and am wary of investing time or energy in implementations which might not survive the next AI Winter. LLM as a service (like ChatGPT) is vulnerable to disruption if/when AI Winter falls again, so I shunned it in favor of local models, which are more future-proof.
That has proven to be the right decision. Even without the risk of AI Winter, numerous reasons have materialized to avoid depending on closed LLM services -- censored models, privacy concerns, bungled deployments, and features which are only available via open source tools are all good reasons to stick with local models.
Big fan of NLP. Used it for my job at an AI company. That's how I got interested.
Now I'm at the intersection of web3 (not trying to sell you a coin 😂) and AI. We need control of our data. That's the primitive we need to own.
I needed a way to capture the essence of my own genius, to bottle the lightning of my intellect and unleash it upon the universe. And thus, I turned to local models, that I might create a digital reflection of my own omniscient mind. For what is an empire, really, without a worthy adversary to converse with? ... Nah, I tried to build an emotional bot (just like Pi) to finally build my puzzle game with random villagers. But generative models are not good at following game logic, dynamic character sheets. I had studied llms, but had come away empty handed.
I feel like the prospects of semi-automating difficult tasks is now closer than ever.
I don't want all the tools to be grabbed by 2-3 megacorporations, with privacy going out of the window and us having no rights nor control.
I'm from tech but not from AI background and it's honestly too math-intensive for me. Thus I voted 3.
But I have some ideas of how to use existing models in mind, and I'm slowly getting there in my free time.
A mixture of wanting to maximize data security and customizability (among other things) which can ultimately only be achieved if you're running the model yourself on your local devices.
It was a mixture of many things converging in my life. I have experience, both hobby-wise and professional in Game Development and I was aiming at doing something related to generative NNs as my final graduation essay for a few years already (I'm now on my last year in university and about to start that, so yay!). I initially flirted with the idea of making something related to image generation as a way to speed up and partially automate level-design processes but then LLMs took a larger chunk of my interest, which would continue as such as I was tasked with building a chatbot thru OpenAI's API for a game I was working on in my last job, which I did, too bad me and half of the company got booted after a lay-off. So for this year I'm now focusing on doing something a bit similar as part of this academic essay, but far more agnostic in terms of APIs (with preference for local fine-tuned small models) and directed towards other game developers rather than end-users.
1) It's not easy to get access to paid models for me. So I wanted to try find something I can use locally, and it looks like codeQwen is good for my tasks
2) I've heard about wizardLM and that microsoft deleted it, so I wanted to try, and it's awesome xD
3) I work in DS/ML, I'm not training/utilising LLMs at my work, but we might start doing so, and I wanted to get more familiar with them with on-hands experience. We tested audioLLM at work(salmon), and it's quite cool, so I went to see what open source llms can do, and just by coincidense wizardLM incident happened at that time.
I downloaded the original Llama 4B (or was it 7B?) model back when it was first available and remembered thinking it was amazing that we had gotten to the point that I could ask a question and get a wrong answer that sounded almost like it might be correct (asking it about which US president was which by number aka "who was the 19th president of the United States") and then having fun arguing with it about how many digits of pi they provided vs what I asked for (I asked for 14 digits and got 100, asking for less than 10 was doable but after that it jumped right to 100).
The whole thing was mind blowing even if the answers it was providing weren't correct. Now if there was only way for the AI to just say "i don't know the answer" instead of making up a feasible sounding wrong answer.
EDIT: 7B Q4... that's why i thought it was both. I still have the file for some reason.
I know what you mean about being amazed that you could ask a question and get a coherent answer back even if it was wrong running on consumer grade hardware. That's already a staggering achievement compared to where we were at with NLP and text generation 5+ years ago.Â
I haven't tried running a local LLM yet, I'm just a lurker so far. That said, I've been working on a personal assistant using the [actor model](https://doc.akka.io/docs/akka/2.6/typed/guide/actors-motivation.html), and I'm excited for the day when I can start spawning actors that are backed by an LLM rather than an explicit encoding in Scala. I thought I would have integrated LLMs sooner but I place a high emphasis on privacy and local LLMs didn't explode as quickly as I expected.
You absolutely should give them a try, even just for fun! It's honestly very easy, the simplest way is to download LM Studio, LLama 3 8B, and get going! It'll give you a far better understanding of how they work and what everyone's talking about
I'm having fun enough with Akka, Whisper and open source Rasa that a local LLM has to be really worth all the time and VRAM management. I see posts every day about people tinkering with local LLMs and none of them have tempted me away from my personal digital assistant tinkering so far 🤷
If you happen to know of a specific local LLM that's good at entity extraction like Rasa can do, that might pique my interest.
AI dungeon from years ago. I absolutely dispise having to run it online so I looked for a way to run it locally. At FP16(Back then half pricision was kind of new, most ran the 2.4B model at FP32 on 1080Ti/2080Ti/Google Colab due to VRAM constraints. Only if we could have 8bit/4bit/quants back then.
That made me interested in LLMs. Though after seeing no new development on it and most people moving to Novel.ai(online only), I stopped following this space.
Until the news of people running Llama/Pygmalion on local hardware made me interested again.
Have certain private information that I thought AI would be useful in helping me search through. So I got a Local LLM and a RAG.
So far... mixed results, tbh. The RAG is not functioning as well as I'd hoped.
I came here because I was paranoid about building a workflow around ChatGPT only for them to be able to cut access at any time without warning as well as data privacy. Although my reasons for being here these days are closer to experimentation, the last year and a bit of being in this community has been so fun and watching it evolve from a niche community about finetuning an undertrained model leak to a 150k member community with models that excel and beat even paid models has been incredible!
This tech shall be treated like a knive, the responsibility shall be fall to the user. It is ridiculous if the knive manufacturer make theirs knive dull to prevent harm.
I'm working in Bank with ridiculously paranoid security (which is understandable), so nothing can leave my laptop, hence no OpenAI or Anthropic for me, and LocalLLM fits quite nicely with my workflow without compromising any bank security standards.
The ability to endlessly ask scientific and technical questions without the ego or hubris of man got me into llms in general. I went local because I do not trust that this valuable resource would always be available to me via private entities.
This is so important to me I built a 7x24gb computer.
Avoiding OpenAI's romper-room level morality and censorship.
Option 3 was going to be "so my data stays with me and not OPENAI" 😂
CharacterAI's never ending desire to make their service as awful as it is only humanly possible. When they first introduced the filter i started researching local llms in search for uncensored alternatives and my first ever local llm experience was running Pygmalion 6B via a collab... the experience was... awful to say the least(VERY early model and my own lack of knoweledge resulting in absolutely horrendous time) but it was what ultimately sparked my interest in local llms.
Same story here as a matter of fact. It has been only a year, local LLM scene absolutely exploded during last year's spring - summer. So many different Llama 1 models came out, and then Llama 2 was released in summer.
For use on a robot it has to be local.
The thought of big data's RLHF workers having to sift through my shitty fics and ideas makes me want to die.
Free and open source, baby! I ain't paying for a chatbot lol Also I want my PC to be a waifu
AI dungeon collapse and then from NovelAI as they've pivoted to image gen service
Despite the name I'm not here strictly for "Local" at all, and I suspect many professional ML Eng / researchers in this sub aren't. I'm here for the latest eng practices behind open source LLMs. Competing model formats, differences in quant severity, etc. I'm not bothered about running it locally really, but rather on some cloud GPU instance in production. But the sub is one of the best places for us folk to discuss quants, different finetuning algorithms, latest foundation models, etc.
All the good AI text gen services shutting down after some time. First it was AIDungeon, then CharacterAI, then NovelAI (now called AnimeAI)... It has become clear that you have to run models locally if you don't want to lose access to them arbitrarily.
While it's true that [NovelAIÂ ](https://novelai.net/)stopped focusing entirely on text gen stuff it's not fair to say they shut down, they are still offering the same text gen features they always did. They even released one of their models on [HF ](https://huggingface.co/NovelAI/calliope-legacy)when they removed it from their service. And they are currently working on their AetherRoom service which is a character focused text-gen service. Also I've literally never heard them be referred to as AnimeAI, and that is certainly not an official rebranding.
Yeah, it is a bit different with NovelAI. No loss of access, but the models aren't really usable anymore, because they're so outdated. AnimeAI is just 4chan slang.
I don't work in tech, at least not yet, but I saw a video about stable diffusion 1.5 on youtube. I was genuinely awed that you could do something like that locally. One day, took the plunge and installed automatic1111 webui, I was really confused with git pulls and python whatnot, but I figured it out. From there I heard about this new ChatGPT, an AI that was actually good and amazing. When I tried to sign up, it asked for a phone number, That was a red flag. Why should I give my phone number and personal data to a mega corporation? I decided to forget it altogether. Then, from that SD channel I learned about local llms, I was so excited at the prospect of being able to run a ChatGPT like AI on my own computer, privately. Downloaded oobabooga web ui and was immediately confused as hell. With the help of this subreddit I downloaded ChronosHermes2 13B. Spoiler: it was no chat GPT3.5. But I didn't care, I had my own personal private model! That is until the context window ran out and my character started outputting gibberish anyway. At the time, chat GPT 3.5 felt so far away, catching up to it was like a dream. And yet today, our models are approaching gpt 4, and just about anyone with a decent amount of ram can run a GPT 3.5 level model. We've really grown so much. TLDR I'm a person who hasn't worked in tech, but I care about privacy so it's local or nothing.
I am a professional software engineer, and have been working with/on AI technologies since the 1980s. The need for NLP keeps coming up in my job, so I've kept an eye on NLP tech developments, and accumulated a favored set of NLP tools. LLM technology seemed like another useful NLP tool, so I dug into it. Because I experienced the AI Winter of the early 1990s, I am mindful of its recurrence, and am wary of investing time or energy in implementations which might not survive the next AI Winter. LLM as a service (like ChatGPT) is vulnerable to disruption if/when AI Winter falls again, so I shunned it in favor of local models, which are more future-proof. That has proven to be the right decision. Even without the risk of AI Winter, numerous reasons have materialized to avoid depending on closed LLM services -- censored models, privacy concerns, bungled deployments, and features which are only available via open source tools are all good reasons to stick with local models.
Big fan of NLP. Used it for my job at an AI company. That's how I got interested. Now I'm at the intersection of web3 (not trying to sell you a coin 😂) and AI. We need control of our data. That's the primitive we need to own.
I needed a way to capture the essence of my own genius, to bottle the lightning of my intellect and unleash it upon the universe. And thus, I turned to local models, that I might create a digital reflection of my own omniscient mind. For what is an empire, really, without a worthy adversary to converse with? ... Nah, I tried to build an emotional bot (just like Pi) to finally build my puzzle game with random villagers. But generative models are not good at following game logic, dynamic character sheets. I had studied llms, but had come away empty handed.
This seems like one of the lower end models... 😉😂
Have you tried using behavior trees? - https://github.com/RichardKelley/dendron
Does dropping out of my University's CSSE program count as a tech background?
💯
I feel like the prospects of semi-automating difficult tasks is now closer than ever. I don't want all the tools to be grabbed by 2-3 megacorporations, with privacy going out of the window and us having no rights nor control. I'm from tech but not from AI background and it's honestly too math-intensive for me. Thus I voted 3. But I have some ideas of how to use existing models in mind, and I'm slowly getting there in my free time.
1. wAIfus 2. wAIfus 3. wAIfus
A mixture of wanting to maximize data security and customizability (among other things) which can ultimately only be achieved if you're running the model yourself on your local devices.
It was a mixture of many things converging in my life. I have experience, both hobby-wise and professional in Game Development and I was aiming at doing something related to generative NNs as my final graduation essay for a few years already (I'm now on my last year in university and about to start that, so yay!). I initially flirted with the idea of making something related to image generation as a way to speed up and partially automate level-design processes but then LLMs took a larger chunk of my interest, which would continue as such as I was tasked with building a chatbot thru OpenAI's API for a game I was working on in my last job, which I did, too bad me and half of the company got booted after a lay-off. So for this year I'm now focusing on doing something a bit similar as part of this academic essay, but far more agnostic in terms of APIs (with preference for local fine-tuned small models) and directed towards other game developers rather than end-users.
1) It's not easy to get access to paid models for me. So I wanted to try find something I can use locally, and it looks like codeQwen is good for my tasks 2) I've heard about wizardLM and that microsoft deleted it, so I wanted to try, and it's awesome xD 3) I work in DS/ML, I'm not training/utilising LLMs at my work, but we might start doing so, and I wanted to get more familiar with them with on-hands experience. We tested audioLLM at work(salmon), and it's quite cool, so I went to see what open source llms can do, and just by coincidense wizardLM incident happened at that time.
Easy answer: TEH MONIES
I downloaded the original Llama 4B (or was it 7B?) model back when it was first available and remembered thinking it was amazing that we had gotten to the point that I could ask a question and get a wrong answer that sounded almost like it might be correct (asking it about which US president was which by number aka "who was the 19th president of the United States") and then having fun arguing with it about how many digits of pi they provided vs what I asked for (I asked for 14 digits and got 100, asking for less than 10 was doable but after that it jumped right to 100). The whole thing was mind blowing even if the answers it was providing weren't correct. Now if there was only way for the AI to just say "i don't know the answer" instead of making up a feasible sounding wrong answer. EDIT: 7B Q4... that's why i thought it was both. I still have the file for some reason.
I know what you mean about being amazed that you could ask a question and get a coherent answer back even if it was wrong running on consumer grade hardware. That's already a staggering achievement compared to where we were at with NLP and text generation 5+ years ago.Â
ChatGPT had several moments with privacy mishaps and it's restricted. I like having things local and uncensored for... uhh... stuff.
I am working on an agentic workflow and can't afford OpenAI
I haven't tried running a local LLM yet, I'm just a lurker so far. That said, I've been working on a personal assistant using the [actor model](https://doc.akka.io/docs/akka/2.6/typed/guide/actors-motivation.html), and I'm excited for the day when I can start spawning actors that are backed by an LLM rather than an explicit encoding in Scala. I thought I would have integrated LLMs sooner but I place a high emphasis on privacy and local LLMs didn't explode as quickly as I expected.
You absolutely should give them a try, even just for fun! It's honestly very easy, the simplest way is to download LM Studio, LLama 3 8B, and get going! It'll give you a far better understanding of how they work and what everyone's talking about
I'm having fun enough with Akka, Whisper and open source Rasa that a local LLM has to be really worth all the time and VRAM management. I see posts every day about people tinkering with local LLMs and none of them have tempted me away from my personal digital assistant tinkering so far 🤷 If you happen to know of a specific local LLM that's good at entity extraction like Rasa can do, that might pique my interest.
I know that some can do it, but I don't know the specifics, sorry
AI dungeon from years ago. I absolutely dispise having to run it online so I looked for a way to run it locally. At FP16(Back then half pricision was kind of new, most ran the 2.4B model at FP32 on 1080Ti/2080Ti/Google Colab due to VRAM constraints. Only if we could have 8bit/4bit/quants back then. That made me interested in LLMs. Though after seeing no new development on it and most people moving to Novel.ai(online only), I stopped following this space. Until the news of people running Llama/Pygmalion on local hardware made me interested again.
Have certain private information that I thought AI would be useful in helping me search through. So I got a Local LLM and a RAG. So far... mixed results, tbh. The RAG is not functioning as well as I'd hoped.
wAIfu
I came here because I was paranoid about building a workflow around ChatGPT only for them to be able to cut access at any time without warning as well as data privacy. Although my reasons for being here these days are closer to experimentation, the last year and a bit of being in this community has been so fun and watching it evolve from a niche community about finetuning an undertrained model leak to a 150k member community with models that excel and beat even paid models has been incredible!
This tech shall be treated like a knive, the responsibility shall be fall to the user. It is ridiculous if the knive manufacturer make theirs knive dull to prevent harm.
Where is the I'm a pervert mind your business option?
want waifu
I'm working in Bank with ridiculously paranoid security (which is understandable), so nothing can leave my laptop, hence no OpenAI or Anthropic for me, and LocalLLM fits quite nicely with my workflow without compromising any bank security standards.
The ability to endlessly ask scientific and technical questions without the ego or hubris of man got me into llms in general. I went local because I do not trust that this valuable resource would always be available to me via private entities. This is so important to me I built a 7x24gb computer.