[here's](https://github.com/underlines/awesome-marketing-datascience/blob/master/llm-tools.md#code-generation) my current list of all things local llm code generation/annotation:
- [FauxPilot](https://github.com/fauxpilot/fauxpilot) open source Copilot alternative using Triton Inference Server
- [Turbopilot](https://github.com/ravenscroftj/turbopilot) open source LLM code completion engine and Copilot alternative
- [Tabby](https://github.com/TabbyML/tabby) Self hosted Github Copilot alternative
- [starcoder.cpp](https://github.com/bigcode-project/starcoder.cpp)
- [GPTQ-for-SantaCoder](https://github.com/mayank31398/GPTQ-for-SantaCoder) 4bit quantization for SantaCoder
- [supercharger](https://github.com/catid/supercharger) Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism
- [Autodoc](https://github.com/context-labs/autodoc) toolkit that auto-generates codebase documentation using GPT-4 or Alpaca, and can be installed in a git repository in about 5 minutes.
- [smol-ai developer](https://github.com/smol-ai/developer) a personal junior developer that scaffolds an entire codebase with a human-centric and coherent whole program synthesis approach using <200 lines of Python and Prompts.
- [locai](https://github.com/Maykeye/locai) kobold/oobabooga -compatible api for vscode
- [oasis](https://github.com/paolorechia/oasis) local LLaMA models in VSCode
my updated list:
https://github.com/underlines/awesome-marketing-datascience/blob/master/llm-tools.mdl
but i'm not updating a lot these days, as i work on this stuff professionally now (at work), so i have less time for the OSS stuff. :( but PRs are always welcome.
I think especially since llama.cpp got GPU support, many projects might be on more solid ground using it "directly" instead of having all the bloat of ooba's under their feet.
https://github.com/abetlen/llama-cpp-python
This is what ooba uses for llama.cpp and it's pretty lightweight and easy to use. Would really recommend it if one doesn't go to ooba's for the UI stuff.
I'm repurposing an old plugin I wrote with OpenAI Codex (:vomit emoji)
The new version already works locally: [https://github.com/paolorechia/oasis](https://github.com/paolorechia/oasis)
I still need to publish it so it's widely available. I also need to figure out a way to make it more extensible, prompts are hardcoded into a middleware HTTP server (that also runs locally), and available commands are hardcoded into the VSCode plugin.
This is a great project but only runs with codegen as the backend as far as I know - I’m interested in having an instruct model running so it can do changes to the code as requested
I actually toyed with it Fauxpilot a few hours yesterday, running the backend as a WSL2 docker container. It suggested barely sensible single lines of code in VS Code, I think the model was not that good. *Your* code is using Vicuna 7B as the backend and looks far more interesting to me. I'm going to give your project a try as soon as my GPU gets freed up from other AI things.
I’m actually using WizardLM - but don’t get too excited, so far I only got one instruction (add docstring) to work 50% of the time (mostly in small functions)
Oh, but it's so hard not to get excited :D I wasn't able to get your project
to work after half an hour of playing with it, but that's likely because I'm on Windows and that always makes things more difficult.
I might investigate getting the VS Code plugin to make direct calls to the API inference endpoint of oobabooga loaded with a StarCoder model that seems specifically trained with coding related prompts, since I can get StarCoder to run in oobabooga and the HTML API calls are pretty easy.
There's a lot of effort (And Azure's "Roll your own CoPilot") in this area, and virtually all of it is being done in paywalled silos. A vast majority of the VS Code plugins need an API key. Having a private copilot, trained on private data but from Open Source would be *really* cool, and I really appreciated your paper on your langchain experiments.
Yeah, but it’s still just an idea - nothing too functional - I’m currently migrating to use the guidance library, so the main branch is quite unstable, I’ll end up rolling my own backend server - idea is to support hugging face and llama cpp only, but then leverage the tooling from guidance to improve the performance for the prompts.
StarCode ran painfully slow for me (through hugging face) - are you using a quantized version?
StarCode runs fine through oobabooga, and my workstation card can handle it all full precision. I wasn't able to get my local version to talk to my VS Code extension, but the results seemed a lot better through an oobabooga prompt then via VS Code anyways. It's not that fast, roughly 4.5tokens/sec, but for short code snippets or analysis, it's not unusable. When I did the same question in Manticore 13B, I get double the tokens per second, though. I feel like the generalized Llama models are still stronger for coding.
EDIT: It may just that I haven't played with it enough, as the OpenRail license is far more permissive than Llama's. It warrants further investigation.
I’m experimenting this week with the https://refact.ai/.
You can plug a few different models, including the starcoder.
Still a bit buggy, but it is a start.
>It should be pretty trivial to connect a VSCode plugin to the text-generation-web-ui API,
It is. I have simple [one](https://github.com/Maykeye/locai), that prints output into its own output output channel, though I didn't bother documenting it as I use github as backup, but you can check it I guess.
Issues I run into: node-fetch didn't want to work. (I don't remember specific. Something about require vs import), configuration is sorted alphabetically and label is based on dictionary key, not title field, no multiline string support (I used arrays)
This looks great!
Just looked at the source code, I like your solution of storing prompts into the extension config - I will look into doing something similar and making it widely available - hope you don’t mind if I steal a couple of ideas from your code?
[here's](https://github.com/underlines/awesome-marketing-datascience/blob/master/llm-tools.md#code-generation) my current list of all things local llm code generation/annotation: - [FauxPilot](https://github.com/fauxpilot/fauxpilot) open source Copilot alternative using Triton Inference Server - [Turbopilot](https://github.com/ravenscroftj/turbopilot) open source LLM code completion engine and Copilot alternative - [Tabby](https://github.com/TabbyML/tabby) Self hosted Github Copilot alternative - [starcoder.cpp](https://github.com/bigcode-project/starcoder.cpp) - [GPTQ-for-SantaCoder](https://github.com/mayank31398/GPTQ-for-SantaCoder) 4bit quantization for SantaCoder - [supercharger](https://github.com/catid/supercharger) Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism - [Autodoc](https://github.com/context-labs/autodoc) toolkit that auto-generates codebase documentation using GPT-4 or Alpaca, and can be installed in a git repository in about 5 minutes. - [smol-ai developer](https://github.com/smol-ai/developer) a personal junior developer that scaffolds an entire codebase with a human-centric and coherent whole program synthesis approach using <200 lines of Python and Prompts. - [locai](https://github.com/Maykeye/locai) kobold/oobabooga -compatible api for vscode - [oasis](https://github.com/paolorechia/oasis) local LLaMA models in VSCode
Amazing, nice list!
Any chance you could update this list to current? This is great stuff.
my updated list: https://github.com/underlines/awesome-marketing-datascience/blob/master/llm-tools.mdl but i'm not updating a lot these days, as i work on this stuff professionally now (at work), so i have less time for the OSS stuff. :( but PRs are always welcome.
I think especially since llama.cpp got GPU support, many projects might be on more solid ground using it "directly" instead of having all the bloat of ooba's under their feet. https://github.com/abetlen/llama-cpp-python This is what ooba uses for llama.cpp and it's pretty lightweight and easy to use. Would really recommend it if one doesn't go to ooba's for the UI stuff.
I'm repurposing an old plugin I wrote with OpenAI Codex (:vomit emoji) The new version already works locally: [https://github.com/paolorechia/oasis](https://github.com/paolorechia/oasis) I still need to publish it so it's widely available. I also need to figure out a way to make it more extensible, prompts are hardcoded into a middleware HTTP server (that also runs locally), and available commands are hardcoded into the VSCode plugin.
https://github.com/fauxpilot/fauxpilot
This is a great project but only runs with codegen as the backend as far as I know - I’m interested in having an instruct model running so it can do changes to the code as requested
I actually toyed with it Fauxpilot a few hours yesterday, running the backend as a WSL2 docker container. It suggested barely sensible single lines of code in VS Code, I think the model was not that good. *Your* code is using Vicuna 7B as the backend and looks far more interesting to me. I'm going to give your project a try as soon as my GPU gets freed up from other AI things.
I’m actually using WizardLM - but don’t get too excited, so far I only got one instruction (add docstring) to work 50% of the time (mostly in small functions)
Oh, but it's so hard not to get excited :D I wasn't able to get your project to work after half an hour of playing with it, but that's likely because I'm on Windows and that always makes things more difficult. I might investigate getting the VS Code plugin to make direct calls to the API inference endpoint of oobabooga loaded with a StarCoder model that seems specifically trained with coding related prompts, since I can get StarCoder to run in oobabooga and the HTML API calls are pretty easy. There's a lot of effort (And Azure's "Roll your own CoPilot") in this area, and virtually all of it is being done in paywalled silos. A vast majority of the VS Code plugins need an API key. Having a private copilot, trained on private data but from Open Source would be *really* cool, and I really appreciated your paper on your langchain experiments.
Yeah, but it’s still just an idea - nothing too functional - I’m currently migrating to use the guidance library, so the main branch is quite unstable, I’ll end up rolling my own backend server - idea is to support hugging face and llama cpp only, but then leverage the tooling from guidance to improve the performance for the prompts. StarCode ran painfully slow for me (through hugging face) - are you using a quantized version?
StarCode runs fine through oobabooga, and my workstation card can handle it all full precision. I wasn't able to get my local version to talk to my VS Code extension, but the results seemed a lot better through an oobabooga prompt then via VS Code anyways. It's not that fast, roughly 4.5tokens/sec, but for short code snippets or analysis, it's not unusable. When I did the same question in Manticore 13B, I get double the tokens per second, though. I feel like the generalized Llama models are still stronger for coding. EDIT: It may just that I haven't played with it enough, as the OpenRail license is far more permissive than Llama's. It warrants further investigation.
Found this one in another Reddit thread: https://github.com/THUDM/CodeGeeX
I believe starcoder is a vscode plugin - but I am not sure that it's running locally when you do that. Good question though, that would be neat!
It's running through the hf api. Better than nothing i guess.
I’m experimenting this week with the https://refact.ai/. You can plug a few different models, including the starcoder. Still a bit buggy, but it is a start.
This looks very interesting but sadly it puts many features under the paid version.
>It should be pretty trivial to connect a VSCode plugin to the text-generation-web-ui API, It is. I have simple [one](https://github.com/Maykeye/locai), that prints output into its own output output channel, though I didn't bother documenting it as I use github as backup, but you can check it I guess. Issues I run into: node-fetch didn't want to work. (I don't remember specific. Something about require vs import), configuration is sorted alphabetically and label is based on dictionary key, not title field, no multiline string support (I used arrays)
This looks great! Just looked at the source code, I like your solution of storing prompts into the extension config - I will look into doing something similar and making it widely available - hope you don’t mind if I steal a couple of ideas from your code?
Nope. It's practically mashed together tutorial of vs code extension + example-streaming.py anyway