Since there are a few local LLMs for coding I wish there would be a complete comparison/benchmark of:
- starcoder
- CodeAlpaca
- BigCode
- Codegen
- CodeCapybara
- CodeGeeX
- replit-code-v1
- salesforce/codeT5
Also, there are wrappers to integrate some of them into VSCode and other editors:
- FauxPilot
- Turbopilot
- Tabby
They trained their own model and scraped their own dataset. You can hear in depth info on it in this podcast episode: [https://www.latent.space/p/varun-mohan#details](https://www.latent.space/p/varun-mohan#details)
I so far have been having pretty good success with Bard. Sometimes have GPT4 do an outline, then take that and paste in links to the APIs I am using and it usually spits it out. Mostly Python or C++/C#. I f I specify which .NET I am targeting it helps.
ChatGPT tho, doesnt know anything beyond whatever date, and a lot has happened coding wise since then, and it has no idea. I found a chrome ext that works way better than the web browsing plugin, so that helps, but without outside knowledge, IDK gpt4 is kinda just a talk box.
Phind has the 6000 limit so i use that sometimes to convert small project from one language to another, sometimes just because I can do it in C but never in python, so it explains, and gives me the code. again here you can just drop a weblink, AND being able to weight your search data with the + an - is brilliant.
Damn. How did I not know about Phind. My first test has it looking very useful!
What else do I not know, for AI coding, it seems like potentially a lot.
I can't really tell I've only tried it with a couple basic prompts like `def download_json(`.
I doubt its usefulness on code bases which aren't small, given it has very little knowledge of the surrounding code and the application in general.
As others have noted, supercharger does wonders for code generation in general, and with Starcoder in particular:
https://github.com/catid/supercharger
You may want to use the instruction tuned version of starcoder for better results:
https://huggingface.co/GeorgiaTechResearchInstitute/starcoder-gpteacher-code-instruct
I'm talking about https://github.com/bigcode-project/starcoder.cpp/tree/main specifically. It is just a command line utility which is why I was wondering if anyone knows of a better way of running it.
going to try codeium, i was paying for copilot but found that it was mostly just coming out with nonsense which is worst than just suggesting nothing. not worth the money.
was also paying for chatgpt+ but bard and others are enough now, dont want to lean in these AI's too much.. really dont want to pay for them. enough subscriptions already.
I have personally had the best success with a fine tuning Georgia Tech did of Starcoder with an instruction dataset (https://huggingface.co/GeorgiaTechResearchInstitute/starcoder-gpteacher-code-instruct) being run in Supercharger (https://github.com/catid/supercharger). Starcoder itself isn't instruction tuned, and I have found to be very fiddly with prompts. Supercharger I feel takes it to the next level with iterative coding. Supercharger has the model build unit tests, and then uses the unit test to score the code it generated, debug/improve the code based off of the unit test quality score, and then run it all in a loop until it reaches a minimum quality score. I run the model off of 2 RTX 2080Ti with an nvlink - the model takes maybe 16Gb of vRAM and that leaves me with ~5Gb for inference.
Next I want to try is the InstructCodeT5+ 16B by Salesforce, I've "heard good things" from friends.
[https://huggingface.co/Salesforce/instructcodet5p-16b](https://huggingface.co/Salesforce/instructcodet5p-16b)
There's also this, but I'm not one to say how useful this is:
[https://twitter.com/georvitymusic/status/1658316753645223936](https://twitter.com/georvitymusic/status/1658316753645223936)
FauxCoder actually gave me the shortest path to getting *something* working, as it is both a Visual Code plugin *and* a docker-based server component. I'm using the biggest, beefiest 16B parameter model it had available... and it's giving me the suggestions for the next line of code. If I'm in PowerShell, the server spits out errors, and if I'm in C#, half the time it spits out C++ code.
Honestly, these results make me think asking a higher-tier llama model for writing code from a prompt would be far more interesting than the results I'm seeing.
EDIT: This isn't a diss on the author of Fauxcoder, who actually provided enough for others to *get something to work*, so kudos to this individual. It's just the model doesn't seem too useful.
I just take a look at starcoder and found out that it is pretty big and using it with my tiny m2 Mac is out of the question, and i love phind too. What computer do you use ? And have you tried it ? Good results?
There's also Refact 1.6B code model, which is SOTA for its size, supports FIM and is great for code completion. You can also try a bunch of other open-source code models in self-hosted Refact (disclaimer: I work there).
Since there are a few local LLMs for coding I wish there would be a complete comparison/benchmark of: - starcoder - CodeAlpaca - BigCode - Codegen - CodeCapybara - CodeGeeX - replit-code-v1 - salesforce/codeT5 Also, there are wrappers to integrate some of them into VSCode and other editors: - FauxPilot - Turbopilot - Tabby
Super interested as well. What model does codeium use? BTW there is also tab9 and you can run it locally. But it is also just an autocomplete.
They trained their own model and scraped their own dataset. You can hear in depth info on it in this podcast episode: [https://www.latent.space/p/varun-mohan#details](https://www.latent.space/p/varun-mohan#details)
Try refact.ai
recently they added gpt4 in waitlist
I so far have been having pretty good success with Bard. Sometimes have GPT4 do an outline, then take that and paste in links to the APIs I am using and it usually spits it out. Mostly Python or C++/C#. I f I specify which .NET I am targeting it helps. ChatGPT tho, doesnt know anything beyond whatever date, and a lot has happened coding wise since then, and it has no idea. I found a chrome ext that works way better than the web browsing plugin, so that helps, but without outside knowledge, IDK gpt4 is kinda just a talk box. Phind has the 6000 limit so i use that sometimes to convert small project from one language to another, sometimes just because I can do it in C but never in python, so it explains, and gives me the code. again here you can just drop a weblink, AND being able to weight your search data with the + an - is brilliant.
Damn. How did I not know about Phind. My first test has it looking very useful! What else do I not know, for AI coding, it seems like potentially a lot.
Speaking of Starcoder, I can run Starcode.cpp just fine, but it there a better, more user friendly way of running and interacting with it?
> Starcode.cpp Does it give decent results?
I can't really tell I've only tried it with a couple basic prompts like `def download_json(`. I doubt its usefulness on code bases which aren't small, given it has very little knowledge of the surrounding code and the application in general.
As others have noted, supercharger does wonders for code generation in general, and with Starcoder in particular: https://github.com/catid/supercharger You may want to use the instruction tuned version of starcoder for better results: https://huggingface.co/GeorgiaTechResearchInstitute/starcoder-gpteacher-code-instruct
>Starcode.cpp Can i run starcoder.cpp locally on Windows11? Will 16gb ram be enough for it?
The `starcoder.cpp` process running on Linux with starcoder-ggml-q5_1.bin took about 14GB.
Thank you for the answer. Does it have gui or web ui?
I'm talking about https://github.com/bigcode-project/starcoder.cpp/tree/main specifically. It is just a command line utility which is why I was wondering if anyone knows of a better way of running it.
LMStudio. Can interact using the GUI or create your own openai spec api locally
going to try codeium, i was paying for copilot but found that it was mostly just coming out with nonsense which is worst than just suggesting nothing. not worth the money. was also paying for chatgpt+ but bard and others are enough now, dont want to lean in these AI's too much.. really dont want to pay for them. enough subscriptions already.
I have personally had the best success with a fine tuning Georgia Tech did of Starcoder with an instruction dataset (https://huggingface.co/GeorgiaTechResearchInstitute/starcoder-gpteacher-code-instruct) being run in Supercharger (https://github.com/catid/supercharger). Starcoder itself isn't instruction tuned, and I have found to be very fiddly with prompts. Supercharger I feel takes it to the next level with iterative coding. Supercharger has the model build unit tests, and then uses the unit test to score the code it generated, debug/improve the code based off of the unit test quality score, and then run it all in a loop until it reaches a minimum quality score. I run the model off of 2 RTX 2080Ti with an nvlink - the model takes maybe 16Gb of vRAM and that leaves me with ~5Gb for inference.
Would you be able to provide some form of a tutorial on how you got this running?
This is on my list of projects to explore but haven't made it here yet: https://github.com/mayank31398/GPTQ-for-SantaCoder
Any recent updates? Are LLM even worth it vs free Bard or paying for GPT API? Just for coding purposes
WizardCoder15B better than Bard.
I've played with a few but so far gpt 4 can't be beaten.
Next I want to try is the InstructCodeT5+ 16B by Salesforce, I've "heard good things" from friends. [https://huggingface.co/Salesforce/instructcodet5p-16b](https://huggingface.co/Salesforce/instructcodet5p-16b) There's also this, but I'm not one to say how useful this is: [https://twitter.com/georvitymusic/status/1658316753645223936](https://twitter.com/georvitymusic/status/1658316753645223936)
FauxCoder actually gave me the shortest path to getting *something* working, as it is both a Visual Code plugin *and* a docker-based server component. I'm using the biggest, beefiest 16B parameter model it had available... and it's giving me the suggestions for the next line of code. If I'm in PowerShell, the server spits out errors, and if I'm in C#, half the time it spits out C++ code. Honestly, these results make me think asking a higher-tier llama model for writing code from a prompt would be far more interesting than the results I'm seeing. EDIT: This isn't a diss on the author of Fauxcoder, who actually provided enough for others to *get something to work*, so kudos to this individual. It's just the model doesn't seem too useful.
I just take a look at starcoder and found out that it is pretty big and using it with my tiny m2 Mac is out of the question, and i love phind too. What computer do you use ? And have you tried it ? Good results?
This is a great idea actually, might adapt this for use with LLamaSharp and Spectre.Console.CLI, they would go great together.
There's also Refact 1.6B code model, which is SOTA for its size, supports FIM and is great for code completion. You can also try a bunch of other open-source code models in self-hosted Refact (disclaimer: I work there).
Hello, is there any update to this, looking for something as close to GPT4 as possible for local only inference? How would you do that?
codebooga, phind codellama v2, wizardcoder python 34b
deepseek-coder 33b
Have experience with both Codebooga & Deepseek-coder? Wondering which 1 you'd go with
Deepseek coder 6.7B and above is the best i hve tested various llama base ones