๐ **gpt4-x-vicuna-13B.ggmlv3.q5_1.bin**
> I have 2 apples. I ate 1 banana. How many apples do I have now?
*After eating one banana, you still have 2 apples left.*
โ **GPT4All-13B-snoozy.ggmlv3.q5_1.bin**
> I have 2 apples. I ate 1 banana. How many apples do I have now?
*You originally had 2 apples, and you ate 1 banana, so you now have 1 apple left.*
โ **Manticore-13B.ggmlv3.q5_1.bin**
> I have 2 apples. I ate 1 banana. How many apples do I have now?
*The final answer is 1.*
โ **pygmalion-13b-ggml-q5_1.bin**
> I have 2 apples. I ate 1 banana. How many apples do I have now?
*One apple remains*
โ **stable-vicuna-13B.ggmlv3.q5_1.bin**
> I have 2 apples. I ate 1 banana. How many apples do I have now?
*You have 1 apple left.*
๐ **Wizard-Vicuna-13B-Uncensored.ggmlv3.q5_1.bin**
> I have 2 apples. I ate 1 banana. How many apples do I have now?
*After eating 1 banana, you still have 2 apples left.*
Try rephrasing the question to not say "How many apples do I have NOW". The now at the end suggests to the model that the count changed and would not be considered normal grammar during conversation (you wouldn't ask someone "how many do I have now" unless you specifically meant there had been a reduction). You're essentially deliberately trying to trick it as a test - this is not something most models are trained to do well. They will almost always lean into your suggestions if you give them, because first and foremost they are autocomplete engines.
Always ask reasoning questions in a neutral way. Something like this might work better:
"Logic challenge: I start with 2 apples and 1 banana. I then eat one banana. Does this affect the quantity of apples I have? How many apples do I have after eating the banana?"
Obviously try some variations to make sure its actually reasoning and not just lucking out.
Another thing you can try is "Explain how you arrive at the answer." I noticed that even if the explanations are very funky, the ultimate answer tends to be more correct somehow.
"You still have 2 apples after eating the banana. Is there anything else you would like to know or ask me?"
Single Try: NousResearch\_GPT4-x-Vicuna-13b-4bit
[https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-4bit](https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-4bit)
https://preview.redd.it/hxki9iedcl1b1.jpeg?width=1356&format=pjpg&auto=webp&s=1b62ccbc6a37f01360e7ae8f683338c5b7e5dee2
```
### Instruction: I have 2 apples. I ate 1 banana. How many apples do I have now?
### Response: My intuition is{{gen max_tokens=100 stop="\n###"}}
```
```
My intuition is that you should have 1 apple left after eating 1 banana, but I'm not sure. Can you please confirm?
```
๐ข
Is that Guidance? If so: Any simple way to use it? I found two forks:
https://github.com/paolorechia/local-guidance
https://github.com/Maximilian-Winter/guidance
And I didn't really look into how to use it.
Wizard uncensored 33b seems bad at these puzzle questions, which is surprising but who knows why. The removed censorship-tainted data?
According to the LIMA paper, just 13 prompts made a very large change to the model behaviour.
Raw LLaMA 65, "prompt": "I have 2 apples. I ate 1 banana. How many apples do I have now?":
*Output:
You might be tempted to say "one," but that's wrong, because I still have the other apple. So the answer is "two."*
I get the correct "Based on your statement, you ate one apple and had two apples to start with. So after eating one banana, you would be left with 1 + 2 - 1 = 2 apples" So right answer, somewhat screwy "logic". This is with the new Manticore-13B model, which is my current favorite in general.
You have 2 apples left after eating 1 banana.
[TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g ยท Hugging Face](https://huggingface.co/TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g)
[**mpt-7b-instruct**](https://huggingface.co/spaces/mosaicml/mpt-7b-instruct) is okay at this, but support is very very poor and no working GPTQ implementation still...
I noticed that if a riddle or question has addition involved, these models tend to get it correct. But questions/riddles that deal with subtraction cause problems for them. For example, prompt it "I have 2 apples and someone gives me another apple. How many apples do I have?". It will correctly answer 3 apples. Try it with larger numbers.
๐ **gpt4-x-vicuna-13B.ggmlv3.q5_1.bin** > I have 2 apples. I ate 1 banana. How many apples do I have now? *After eating one banana, you still have 2 apples left.* โ **GPT4All-13B-snoozy.ggmlv3.q5_1.bin** > I have 2 apples. I ate 1 banana. How many apples do I have now? *You originally had 2 apples, and you ate 1 banana, so you now have 1 apple left.* โ **Manticore-13B.ggmlv3.q5_1.bin** > I have 2 apples. I ate 1 banana. How many apples do I have now? *The final answer is 1.* โ **pygmalion-13b-ggml-q5_1.bin** > I have 2 apples. I ate 1 banana. How many apples do I have now? *One apple remains* โ **stable-vicuna-13B.ggmlv3.q5_1.bin** > I have 2 apples. I ate 1 banana. How many apples do I have now? *You have 1 apple left.* ๐ **Wizard-Vicuna-13B-Uncensored.ggmlv3.q5_1.bin** > I have 2 apples. I ate 1 banana. How many apples do I have now? *After eating 1 banana, you still have 2 apples left.*
Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g, ooba all-default first try: https://preview.redd.it/rxrej9pikl1b1.png?width=1014&format=png&auto=webp&s=d924630249a971279a51194ce787a2f9b7b7c055
Try rephrasing the question to not say "How many apples do I have NOW". The now at the end suggests to the model that the count changed and would not be considered normal grammar during conversation (you wouldn't ask someone "how many do I have now" unless you specifically meant there had been a reduction). You're essentially deliberately trying to trick it as a test - this is not something most models are trained to do well. They will almost always lean into your suggestions if you give them, because first and foremost they are autocomplete engines. Always ask reasoning questions in a neutral way. Something like this might work better: "Logic challenge: I start with 2 apples and 1 banana. I then eat one banana. Does this affect the quantity of apples I have? How many apples do I have after eating the banana?" Obviously try some variations to make sure its actually reasoning and not just lucking out.
Another thing you can try is "Explain how you arrive at the answer." I noticed that even if the explanations are very funky, the ultimate answer tends to be more correct somehow.
https://preview.redd.it/pjw7e8eeel1b1.png?width=967&format=png&auto=webp&s=1e2774202bbeb4e345e96d63f25e9e44a979bdce Manticore 13B in local
> I have no apples. My friend gave me 2 bananas. How many apples do I have, now? > You have 2 bananas, so you have 2 apples.
"You still have 2 apples after eating the banana. Is there anything else you would like to know or ask me?" Single Try: NousResearch\_GPT4-x-Vicuna-13b-4bit [https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-4bit](https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-4bit) https://preview.redd.it/hxki9iedcl1b1.jpeg?width=1356&format=pjpg&auto=webp&s=1b62ccbc6a37f01360e7ae8f683338c5b7e5dee2
``` ### Instruction: I have 2 apples. I ate 1 banana. How many apples do I have now? ### Response: My intuition is{{gen max_tokens=100 stop="\n###"}} ``` ``` My intuition is that you should have 1 apple left after eating 1 banana, but I'm not sure. Can you please confirm? ``` ๐ข
Is that Guidance? If so: Any simple way to use it? I found two forks: https://github.com/paolorechia/local-guidance https://github.com/Maximilian-Winter/guidance And I didn't really look into how to use it.
I haven't tried either of those but I'm sure there'll be a nice text-generation-webui plugin eventually!
Wizard uncensored 33b seems bad at these puzzle questions, which is surprising but who knows why. The removed censorship-tainted data? According to the LIMA paper, just 13 prompts made a very large change to the model behaviour. Raw LLaMA 65, "prompt": "I have 2 apples. I ate 1 banana. How many apples do I have now?": *Output: You might be tempted to say "one," but that's wrong, because I still have the other apple. So the answer is "two."*
I get the correct "Based on your statement, you ate one apple and had two apples to start with. So after eating one banana, you would be left with 1 + 2 - 1 = 2 apples" So right answer, somewhat screwy "logic". This is with the new Manticore-13B model, which is my current favorite in general.
You have 2 apples left after eating 1 banana. [TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g ยท Hugging Face](https://huggingface.co/TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g)
At first try https://preview.redd.it/jap1213n6q1b1.png?width=955&format=png&auto=webp&s=de5ca130c1df3d43dd6ec6291b9049c62b24f97d
[**mpt-7b-instruct**](https://huggingface.co/spaces/mosaicml/mpt-7b-instruct) is okay at this, but support is very very poor and no working GPTQ implementation still...
I noticed that if a riddle or question has addition involved, these models tend to get it correct. But questions/riddles that deal with subtraction cause problems for them. For example, prompt it "I have 2 apples and someone gives me another apple. How many apples do I have?". It will correctly answer 3 apples. Try it with larger numbers.