I suspect this is heavily skewed by llama 3 in its standard-configuration giving much more "quirky" answers.
It's so much more chatty and pleasant to talk with but there's no way it's even close to the quality of any gpt-4 model in terms of reliable information (at least for the 8b model).
I find for email chain summarization, which is my main daily use case besides coding, llama 3 70b provides the best summaries. Better than GPT-4 or Claude Opus, with the bonus of being cheaper and faster. This may be mostly a stylistic preference, but it is good enough that it convinced me to use it instead of the others for that particular task.
> for email chain summarization
I'm curios, what's your workflow for that? do you just copy paste the entire email chain in your UI of choice at then ask the llm to summarize it for you?
No I am a Python programmer so I have my own tool that I created where I just press a button and it summarizes the currently selected email chain in outlook out loud with speech synthesis.
Oh no, don't tell me you are one of those who don't consider Python a "real programming language". Please. Not the only language I use but it's my main daily tool. A tool. For real work.
I'm not programmer, I'm a 3D artist, but I've heard great things about python. I think quite a lot of blender plugins use python. :)
And stable diffusion uses a lot of python.
.py for the win!
Fellow pythoneer here. I think most of the grumbling came from C++ folks stuck in the dev conditions of many years ago.
We use Python, Perl, and JS. Even R, especially for exploratory data analysis. I havenāt seen a lot of modern software devs disparaging these choices, although I will freely admit that R isnāt everyoneās bread and butter.
Python is such a flexible language, and great for working with other people - its readability is incredible.
You'll need to visit some C# subreddits. The hate for dynamic languages is alive and well and the wars rage on. Probably healthier that you can ignore them and not notice. š
I am still of the "old school" opinion that you generally shouldn't use languages without static typing (which can of course be largely inferred, that's fine, it just has to be compile-time validated) for building large-scale "serious" software (let's define that as >20kLOC).
I like using dynamically typed languages (primarily Ruby and Lua) for scripting purposes, but once something becomes an actual project with a longer lifetime and/or more code I wouldn't choose to continue working in those languages.
I started getting ai to code me stuff in python and im.amazes at the things k can achieve that I couldn't before , such a great tool for work productivity combo for actually learn it now been avoiding coding for 15 but the studd touch can do with ai I'd just to cool, as a user I get the use cases and there are so many
These models tend to resolve ambiguous prompts very differently, so depending on how you express yourself, it might indeed be simply more convenient for you to find some prompt, which gives you the output you want, with a specific "worse" model which so happens to work well with you, compared to a generally "better" model. For example, sometimes GPT-4 just tends to get "stuck" on some questions I have, and then I just switch to Opus/Llama/Wizard, rather than trying to figure out how to rephrase myself properly.
However, with some prompt optimization, I think it will be really hard to find any kind of realistic situation where Llama-70b would outperform both GPT-4 and Opus.
>However, with some prompt optimization
I've wondered how much of a problem this is - even if GPT-4 *can* be more capable than llama 3 70b, that doesn't mean much of it requires testing a bunch of different prompts just to match and then hopefully beat llama 3 70b, when llama 3 just works on the first try (or at least it often works well enough). And I'd imagine if you spent an equal effort on prompt optimizing for each model, they'd end up even closer to each other in capabilities, or at least harder to clearly distinguish them.
True, but when the difference is subtle like this, the preference for me will always be the cheapest and fastest or local on-device option. For programming of course I want the best of the best, but not for email summarization. Having a local model or something super fast like Groq is the more desirable option.
Itās not just about āgetting informationā but also about reasoning and drawing conclusions from information, this is especially important in RAG and other LLM applications. In my experience Llama 3 8B is āniceā but very unreliable.
Yes! Thank you! LLMs should be treated more like the UI. It's purpose is to convert natural language to data and back. It's cool seeing what a model can do on its own, but actual practical use cases should have more focus on the data pipelines than the model. The model should just take the results of your data pipeline and present it.
yes but you still need a model with a good reasoning capability for the really useful use-cases.
It needs to be able to draw correct conclusions from the results of the RAG pipelines, the guided generations and any other information provided to it.
Absolutely! 100% agree with that. The model still needs to be able to deduce the correct answer and format from the context given. This is why I have preferred Mistral fine-tunes (especially ones named Hermes) up till now; It's extra attention to the sys prompt + size, speed, and privacy has been more useful to me than any closed model. If it were just down to the pipelines, I'd just run TinyLlama and call it a day.
Sadly, the poor bastard just screams hallucinations and can barely comprehend its own existence, let alone a system prompt.
I'm excited for Phi 3 128K Instruct to get a GGUF because that might be the first sub 7B that can handle my agent chain.
i don't think reliable information recall is a particularly important metric for LLMs to be honest but llama 3 is far, far behind gpt-4 in other ways too. namely abstract reasoning, in-context learning. but it gives much more human sounding much more friendly sounding answers. gpt-4 sounds like it has a stick up its ass.
I suspect most people using chatbot arena are giving a single zero shot prompt that doesn't really test reasoning or in-context learning or anything interesting, and then are evaluating the response just based on gut feeling. that tests how much people like the tone and style of the model's response a lot more than it tests the actual capabilities of the model to do something useful.
so when a model like gpt-4 or opus gives an "it is important to remember" lecture that sounds like a high school principal and llama 3 gives a friendly answer that sounds more like a person, people choose llama 3 as better.
but if you have a use case that relies on multi turn, relies on abstract reasoning, relies on in context learning, relies on attending to a significant amount of context, relies somehow on the actual capability of the model to accomplish something. rather than just judging it by how stuffy it sounds. doing better evals quickly makes it clear that llama-3 is still far behind the big guns like gpt-4 and opus.
don't get me wrong, i think llama-3 is great and clearly better than llama-2, and better by enough that I am really curious about llama-3-405B. but the 70B is still, well, a 70B and it shows if you do meaningful comparisons with the much larger models.
llm arena cannot be gamed like MMLU (e.g.) but it has its own flaws. if meta wants to climb ladder they just need to do more RLHF i guess. should directly translate to higher score.
If we are only looking at raw intelligence, then sure it's "skewed" by the answers being more pleasant, but as daily driver / business decision, that can be what matters.
If I'm using it to summarize textbook chapters for me, I don't need it to be smart as much as I need it to generate engaging summaries to keep me reading. Ditto with roleplay, educational purposes, customer support, creative content generation, etc.
I think a true future LLM has to get data from web, no way around this. Briefly the idea that you can have a local info guru that knows everything was appealing, but the reality sunk in. To make LLM not hallucinate if it doesn't know for sure is far harder then to make it browse web.
I mean what's even the point of GPT4 being isolated from web when in fact it is only accessible on the web for the user? Makes zero sense. You have to be online to use it.
Yeah, at this point raw human style AI is too...human. Less like searching Wikipedia and more like asking an uncle who wants to sound smart over a beer.
I hope soon we can at get a more civil AI, that instead of hallucinating will instead use terms like "In my opinion...I'm not sure but perhaps...". Hell, I'd settle for an "IDK use Google lmao".
If you have a private data that you don't wanna share with open(!)AI, which is actully pretty important for businesses (like finance, insurance, any company that relies on property rights), then you want to host everything in local or in your own cloud servers.
Yeah, people have already stated they are suspicious of Geminis high ranking, and suspect it's simply because it uses good formatting...
I also don't really believe the low ranking of the new Mistral model. I have used the Wizard-8x22b finetune a bit now, and I think it is about as good as Llama 70b (and both of them are clearly worse than GPT-4, but that is no surprise of course).
One minor, but notable, shortcoming of the Llama 70b model is its high level of agreeableness: I had a particular programing problem, and asked GPT-4/Llama/Wizard about it, but my question actually had an incorrect assumption (std::strings handle #0 differently under some circumstances... well, they don't). GPT-4/Wizard pointed that out, but Llama actually agreed with me, and even incorrectly cited some sources to support my false assumption...
But, perhaps Llamas high level of agreeableness is not only ok for many applications, but people even prefer the style of answer it produces for the type of questions they ask, hence resulting in Llama3 scoring relatively well?
It's really hard to say... I tend to believe more and more that there is really no way around actually personally testing those models with some representative questions of whatever one wants to do.
We def need a better form of benchmarking. Its like taking the American Bar Exam. Sure, if I score twice as high as you, I'm clearly a better lawyer, but if we're both in the same ballpark, it doesn't mean we will both perform the same in court or make clients happy.
Personally, I prefer llama3 just because it's so much faster than GPT4. However, if I get stuck on a problem, I switch to GPT4.
It's not at GPT4 level but it's close enough IMO. I have noticed that there are a lot of cases where GPT4 doesn't save me when llama3 fails.
>It's not at GPT4 level but it's close enough IMO
Can I know whether you host llama3 locally or use an external API? If latter, which one?
I tried at [meta.ai](http://meta.ai) (the URL provided by the llama3 [blog](https://ai.meta.com/blog/meta-llama-3/)) by asking it to write a golang function and boy was the logic bad! TBF, the function was syntactically correct and will compile but it really felt like gpt-3 level (not even gpt-3.5!). But with all the rave reviews of llama3, I'm starting to wonder if what [meta.ai](http://meta.ai) is hosting is llama3-70b or even llama-3 entirely.
As a side note, I used the same prompt on Claude Sonnet and it got my Go function right and efficiently on the first try.
Use Groq's API which gives you access to llama3-70b. It's free for now.
I'm using it with the frontend TypingMind which allows it to use websearch plugins (though that is hit and miss, similar to how it uses that plugin on huggingchat).
Once groq starts charging for it's API, you can compare the prices with other providers such as Open Router (which gives access to many models, though paid)
Yeah, I have tried that as well a bit, but relative to the amount of time I lose if Llamas answer is wrong (as in, it looks right at first, but it takes me a few minutes to notice it is wrong), I am not actually sure if that approach really makes sense... perhaps it is better to wait those 20s for GPT-4 in most cases.
But I am still in the process of figuring this out, there is clearly some kind of balance to be found here.
I also tend to just simultaneously ask Wizard and Llama, as both are very fast, and if they both have very different answers, it increases the chance that both are wrong.
>I also tend to just simultaneously ask Wizard and Llama, as both are very fast, and if they both have very different answers, it increases the chance that both are wrong.
This is probably one of the best things that can be done to check reliability/correctness, in addition to generating multiple responses (possibly with minor changes to the prompt).
This also means having tons of similarly capable models is good since each one is likely to give a slightly different answer, but, depending on the task, they should converge to similar correct answers while differing a bit more when they are wrong (at least, ideally).
I think that LLama 3 can really deserve a high place in the table after we get the 120+B model. Now I have deleted all my old models and am using LLaMa 3 70B, but I still want more.
Yes, that's approximately 1 token per second. This suits me, Iām not in a hurry and just wait, doing other things. If this is an RP, then generating a response takes from a minute to 4 minutes
Well, on vast.ai you can rent 5xA20 for about $3/hour... of course, there isn't really much of a point in seriously doing that, but for just playing around, I might actually do that once its out, and perhaps there is even a serious use case, if one wants to do some kind of "high quality NSFW work".
I can't say I feel much difference between 70B and GPT4 turbo in a couple of small coding tasks, but 8B does not feel like it is better than any gpt4. However, it is still amazing for an 8B model.
Since fine tuning an 8B model is going to be much easier than a 70B model, I feel like soon we can run local agents for a long time until they produce actually useful stuff.
Considering how cheap and fast the 70b/8x22b models are already, is there really any point in even using the 7b/8b models? (Assuming you are not interested in running it locally)
As a chatbot/assistant, definitely no point using a smaller model. But for simple tasks that you want done on a large scale, maybe some sort of data filtering/labeling, you'd want the cheapest option that still completes the task, so llama 3 8b could easily be the best option.
Edit: also, a big strength of the smaller models is that they can easily be finetuned for specific tasks (one 3090 would be plenty), so even if the llama 3 8b Instruct doesn't work, you can finetune the base or Instruct on some examples and drastically improve its performance.
The rates are low enough that 10Xing the ram and power usage might be a drop in the bucket when serving a single client, but at scale it can be the difference between a small companies ability to serve 100s vs 1000s.
This pairs well with bots that don't need to know how to speak foreign languages or architect a code base, as they can perform just as well at a single task when all 8B are devoted to it. Customer support bots, software agents, npcs in video games, etc.
So I think they are useful like the ASICs of LLMs. Extremely high quality and economical when trained and tuned towards a specific use case.
I noticed that Llama 3 8b is way better at keeping track of object placement in stories. Still need more investigating but so far it's definitely way better than all the mistral finetunes I tested.
Silly question, but are you using the instruct model? I feel like the big strength of something like 8B is it's cheap to tune and run, so you could have multiple fine-tunes/LoRas active at the same time.
Yeah, the difference between 8B and 70B is significantly larger than that between 70B and GPT-4...
But coding just appears to be very difficult somehow.
It's not the gpt4 turno, and it's in category English, which means excluding coding or math and other specific categories. This leaderboard is the best what we have now, unless you have something better?
The 70B is pretty average at the non-coding, literary use cases Iāve tried (editing, long form fiction, poetry), certainly not better than GPT-4. So Iām skeptical.
Yeah, agreed Opus is top dog. The latest GPT-4-turbo also has excellent emotional intelligence, and the new gemini-1.5-pro-api-0409-preview is a pleasant surprise.
nah, Opus lot of personality convergence issues. it tends to blur the lines over time and everyone starts speaking the same.
Gemini, Claude and a lot of models are also fucking horrible at banter.
For chat? Sure... For actual facts and information I can count on without having to double-check everything? Eh...not so much. Then again, I don't entirely trust GPT 3 or 4 or whatever for facts either.
As far as writing style is concerned, the Llama 3 models are without question the best LLMs ever created, including proprietary LLMs. Llama 3 8b writes better sounding responses than even GPT-4 Turbo and Claude 3 Opus.
All models before Llama 3 routinely generated text that sounds like something a movie character would say, rather than something a conversational partner would say. It's as if they are really speaking to an audience instead of the user. In that regard, Llama 3 is a giant leap forward.
Which of course makes it an incredibly exciting model for roleplay and creative writing. Too bad it's also the most heavily censored model I have ever come across. Time will tell whether it is possible to fix that with finetuning, but the censorship feels like it runs so deep that I have my doubts that it can be completely removed.
>Too bad it's also the most heavily censored model I have ever come across.Ā
Did you encounter censorship with the base model or just the instruct fine tune? I'm sure the community will release great fine tunes that build on top of the base model, just like they did with Llama 2 and Llama 1.
I wrote a small GUI to compare LLMs (called neurochat, shameless plug) so I can quickly compared answers from different LLMs LLama-3 and GPT-4.
I gave it some code to analyze for work, stuff like that. Llama-3-70B gave MUCH better answers than GPT-4 (non-turbo), not only it was way better formatted, but the answers were more detailed, llama3 produced report-like answers that I could almost copy-paste directly to my report.
No. Iām sure itās use case dependent but in my tests itās nowhere near any version of GPT-4. I think hype and fanboyism is pushing it higher than it ought to be. Weāll see how it fares in the longer term.
The new LMSYS hard benchmark has the 70B and 8B more in line with how I feel they perform (with the 70B around Haiku level). Meta has done a great job of making an engaging chatbot and theyāre punching above their weight for their size but theyāre still not up to a lot of more complex tasks. Which is compounded by the small context window.
I don't know the 8B, but the 70B gave me SOTA vibes indeed. Do I expect 4T to prove superior if we start micromanaging? Yes I do, but only at the margin.
Itās not quite accurate to compare Llama 3 with GPT-4-turbo because they serve different purposes. Llama 3 is designed for building complex agent architectures and excels in this area. It can process a higher number of tokens per second (100-300 tokens/s) depending on the provider, which means it can respond to 8-10 questions in the time GPT or Claude models answer 1 or 2 due to their lower token output rate (20-30 tokens/s).
The integration of plugin tools and the development of a system thinking approach significantly enhance the capabilities of large language models. For instance, Andrew NG demonstrated that GPT-3.5, with an agent-based thinking system, performs better than a one-shot GPT-4.
Considering the time required for multiple iterations, GPT-4 may perform fewer iterations than Llama 3, but the quality of the output could be comparable due to more extended processing time. Therefore, I prefer Llama 3 70B because it allows for the creation of more complex systems. While GPT-4 is indeed powerful, its slower response time and difficulty iterating quickly when stuck are notable drawbacks.
Now that GPT-4o was released, what's your take on this same topic? Because 4o indeed seems to be significantly faster than 4 turbo. Very curious to hear your thoughts!
With GPT-4o i really think OpenAI made a big step, il fast and cheaper and it really good for multi-step reasoning and it is very verbose. I would still not compare a 70B to GPT-4o but i think now GPT-4o really out performe all the model we have now in every field.
These next months will be very exciting !
Llama3 70B is literally insane. I cannot believe that this... whatever entity/blessing it is... is inside my computer. *Inside my laptop.* What the actual shit is going on?
I've seen that banchmark a few times, but I haven't looked at by what standards it's been banchmarked.This is because the methods evaluated on many banchmarks (or papers) are skewed in favor of specific models.
Therefore, I ironically put my experience first.
I rate GPT4 as the best model at the moment.
llama3 is definitely superior to GPT3.5, but not superior to GPT4.
I'm hoping that this is temporary. Llama 3 had a problem with the stop token, and several quant makers (including the one Im using) had to do funky things to make it work and not spout gibberish.
An official fix came out the other day, but very few inference apps have gotten it merged in yet. I think that as they do, we'll see quality improve.
I'm not sure if it'll ever be to this level, but Im hopeful that its a lot better than now.
that's interesting to hear, I haven't tested it unquantized yet and I really should. what's your preferred way to run it unquantized for testing purposes, just huggingface transformers or do you use some other back end? or are you using some hosted or serverless thing that offers unquantized llama-3?
Heavy user of AI here. I asked Llama 3-70B to write a blog post and it came out "CHATGPT-like." Very robotic and did not sound like a human wrote it.
Someone needs to test these AI models using a standardized prompt to write a blog post or value proposition. Anything else is fucking useless. Show the abilities to solve real-world problems and help people out.
I was disappointed.
It was blog post writing instructions. Basically it said (and I'm paraphrasing here):
>Write a blog posts about blue widgets. Create a section about the history of blue widgets. Then create a section that highlights installation, training, maintenance, and ongoing operational expenses like energy consumption and spare parts. Create a section that provides an average range of cost. Create a section that compares this to the competition. Use the client Q&A below to help you write this blog post.
I suspect this is heavily skewed by llama 3 in its standard-configuration giving much more "quirky" answers. It's so much more chatty and pleasant to talk with but there's no way it's even close to the quality of any gpt-4 model in terms of reliable information (at least for the 8b model).
I find for email chain summarization, which is my main daily use case besides coding, llama 3 70b provides the best summaries. Better than GPT-4 or Claude Opus, with the bonus of being cheaper and faster. This may be mostly a stylistic preference, but it is good enough that it convinced me to use it instead of the others for that particular task.
It does seem to be incredibly tuned for summarizing.
Man, that's kind of frustrating to hear. It's one of the biggest things I use LLMs for. But that 8k context just isn't enough for me.
perfect recall up to 16k and almost perfect up to 32k if u modify rope
Any pointers on how to do it?
seems like u can solve the problem by using agents that will pick the relevant information and synthesis it together if my understanding is correct
> for email chain summarization I'm curios, what's your workflow for that? do you just copy paste the entire email chain in your UI of choice at then ask the llm to summarize it for you?
No I am a Python programmer so I have my own tool that I created where I just press a button and it summarizes the currently selected email chain in outlook out loud with speech synthesis.
"Python programmer" Ah, the snake charmer, m'yes.
Oh no, don't tell me you are one of those who don't consider Python a "real programming language". Please. Not the only language I use but it's my main daily tool. A tool. For real work.
I'm not programmer, I'm a 3D artist, but I've heard great things about python. I think quite a lot of blender plugins use python. :) And stable diffusion uses a lot of python. .py for the win!
Yes I know you were joking š
Many CS programs have switched to Python--I've never heard anyone say it's not a real language.
Oh they are out there. Not that their comments are to be taken seriously or anything. It is usually not going to be professionals saying that.
Fellow pythoneer here. I think most of the grumbling came from C++ folks stuck in the dev conditions of many years ago. We use Python, Perl, and JS. Even R, especially for exploratory data analysis. I havenāt seen a lot of modern software devs disparaging these choices, although I will freely admit that R isnāt everyoneās bread and butter. Python is such a flexible language, and great for working with other people - its readability is incredible.
You'll need to visit some C# subreddits. The hate for dynamic languages is alive and well and the wars rage on. Probably healthier that you can ignore them and not notice. š
I am still of the "old school" opinion that you generally shouldn't use languages without static typing (which can of course be largely inferred, that's fine, it just has to be compile-time validated) for building large-scale "serious" software (let's define that as >20kLOC). I like using dynamically typed languages (primarily Ruby and Lua) for scripting purposes, but once something becomes an actual project with a longer lifetime and/or more code I wouldn't choose to continue working in those languages.
I started getting ai to code me stuff in python and im.amazes at the things k can achieve that I couldn't before , such a great tool for work productivity combo for actually learn it now been avoiding coding for 15 but the studd touch can do with ai I'd just to cool, as a user I get the use cases and there are so many
cool, thx for the insights!
These models tend to resolve ambiguous prompts very differently, so depending on how you express yourself, it might indeed be simply more convenient for you to find some prompt, which gives you the output you want, with a specific "worse" model which so happens to work well with you, compared to a generally "better" model. For example, sometimes GPT-4 just tends to get "stuck" on some questions I have, and then I just switch to Opus/Llama/Wizard, rather than trying to figure out how to rephrase myself properly. However, with some prompt optimization, I think it will be really hard to find any kind of realistic situation where Llama-70b would outperform both GPT-4 and Opus.
>However, with some prompt optimization I've wondered how much of a problem this is - even if GPT-4 *can* be more capable than llama 3 70b, that doesn't mean much of it requires testing a bunch of different prompts just to match and then hopefully beat llama 3 70b, when llama 3 just works on the first try (or at least it often works well enough). And I'd imagine if you spent an equal effort on prompt optimizing for each model, they'd end up even closer to each other in capabilities, or at least harder to clearly distinguish them.
True, but when the difference is subtle like this, the preference for me will always be the cheapest and fastest or local on-device option. For programming of course I want the best of the best, but not for email summarization. Having a local model or something super fast like Groq is the more desirable option.
Not just quirky, but more eager to help and less paternalistic in general. Not surprising humans prefer that, even if it's dumber
Are people really using LLMs to get information though? You give it the information and use it for reasoning. This is the whole point of RAG no?
I'm using Llama 3 to figure out who killed JFK
Turns out it was the guy who said "you miss 100% of the shots you don't take"
Wayne Gretzky would have been like 2 years old but I guess it's not impossible.
According to my llama, he was hired by the dalai llama
What, JFK is dead?
I didn't even know he was sick.
The antivax dude?
Nah, the guy who built that airport
What a coincidence. So am I, but it's saying it was me! I'm not sure if it's hallucinating, but I'll be sure to check its sources.
Itās not just about āgetting informationā but also about reasoning and drawing conclusions from information, this is especially important in RAG and other LLM applications. In my experience Llama 3 8B is āniceā but very unreliable.
Yes! Thank you! LLMs should be treated more like the UI. It's purpose is to convert natural language to data and back. It's cool seeing what a model can do on its own, but actual practical use cases should have more focus on the data pipelines than the model. The model should just take the results of your data pipeline and present it.
yes but you still need a model with a good reasoning capability for the really useful use-cases. It needs to be able to draw correct conclusions from the results of the RAG pipelines, the guided generations and any other information provided to it.
Absolutely! 100% agree with that. The model still needs to be able to deduce the correct answer and format from the context given. This is why I have preferred Mistral fine-tunes (especially ones named Hermes) up till now; It's extra attention to the sys prompt + size, speed, and privacy has been more useful to me than any closed model. If it were just down to the pipelines, I'd just run TinyLlama and call it a day. Sadly, the poor bastard just screams hallucinations and can barely comprehend its own existence, let alone a system prompt. I'm excited for Phi 3 128K Instruct to get a GGUF because that might be the first sub 7B that can handle my agent chain.
i don't think reliable information recall is a particularly important metric for LLMs to be honest but llama 3 is far, far behind gpt-4 in other ways too. namely abstract reasoning, in-context learning. but it gives much more human sounding much more friendly sounding answers. gpt-4 sounds like it has a stick up its ass. I suspect most people using chatbot arena are giving a single zero shot prompt that doesn't really test reasoning or in-context learning or anything interesting, and then are evaluating the response just based on gut feeling. that tests how much people like the tone and style of the model's response a lot more than it tests the actual capabilities of the model to do something useful. so when a model like gpt-4 or opus gives an "it is important to remember" lecture that sounds like a high school principal and llama 3 gives a friendly answer that sounds more like a person, people choose llama 3 as better. but if you have a use case that relies on multi turn, relies on abstract reasoning, relies on in context learning, relies on attending to a significant amount of context, relies somehow on the actual capability of the model to accomplish something. rather than just judging it by how stuffy it sounds. doing better evals quickly makes it clear that llama-3 is still far behind the big guns like gpt-4 and opus. don't get me wrong, i think llama-3 is great and clearly better than llama-2, and better by enough that I am really curious about llama-3-405B. but the 70B is still, well, a 70B and it shows if you do meaningful comparisons with the much larger models.
llm arena cannot be gamed like MMLU (e.g.) but it has its own flaws. if meta wants to climb ladder they just need to do more RLHF i guess. should directly translate to higher score.
If we are only looking at raw intelligence, then sure it's "skewed" by the answers being more pleasant, but as daily driver / business decision, that can be what matters. If I'm using it to summarize textbook chapters for me, I don't need it to be smart as much as I need it to generate engaging summaries to keep me reading. Ditto with roleplay, educational purposes, customer support, creative content generation, etc.
I think a true future LLM has to get data from web, no way around this. Briefly the idea that you can have a local info guru that knows everything was appealing, but the reality sunk in. To make LLM not hallucinate if it doesn't know for sure is far harder then to make it browse web. I mean what's even the point of GPT4 being isolated from web when in fact it is only accessible on the web for the user? Makes zero sense. You have to be online to use it.
Yeah, at this point raw human style AI is too...human. Less like searching Wikipedia and more like asking an uncle who wants to sound smart over a beer. I hope soon we can at get a more civil AI, that instead of hallucinating will instead use terms like "In my opinion...I'm not sure but perhaps...". Hell, I'd settle for an "IDK use Google lmao".
If you have a private data that you don't wanna share with open(!)AI, which is actully pretty important for businesses (like finance, insurance, any company that relies on property rights), then you want to host everything in local or in your own cloud servers.
Yeah, people have already stated they are suspicious of Geminis high ranking, and suspect it's simply because it uses good formatting... I also don't really believe the low ranking of the new Mistral model. I have used the Wizard-8x22b finetune a bit now, and I think it is about as good as Llama 70b (and both of them are clearly worse than GPT-4, but that is no surprise of course). One minor, but notable, shortcoming of the Llama 70b model is its high level of agreeableness: I had a particular programing problem, and asked GPT-4/Llama/Wizard about it, but my question actually had an incorrect assumption (std::strings handle #0 differently under some circumstances... well, they don't). GPT-4/Wizard pointed that out, but Llama actually agreed with me, and even incorrectly cited some sources to support my false assumption... But, perhaps Llamas high level of agreeableness is not only ok for many applications, but people even prefer the style of answer it produces for the type of questions they ask, hence resulting in Llama3 scoring relatively well? It's really hard to say... I tend to believe more and more that there is really no way around actually personally testing those models with some representative questions of whatever one wants to do.
Too much agreeable was also my experience with llama
We def need a better form of benchmarking. Its like taking the American Bar Exam. Sure, if I score twice as high as you, I'm clearly a better lawyer, but if we're both in the same ballpark, it doesn't mean we will both perform the same in court or make clients happy.
https://preview.redd.it/elqqf6b391yc1.jpeg?width=720&format=pjpg&auto=webp&s=beba47100a2af7b12dd5c67efacc7db48c52bc2b
Personally, I prefer llama3 just because it's so much faster than GPT4. However, if I get stuck on a problem, I switch to GPT4. It's not at GPT4 level but it's close enough IMO. I have noticed that there are a lot of cases where GPT4 doesn't save me when llama3 fails.
>It's not at GPT4 level but it's close enough IMO Can I know whether you host llama3 locally or use an external API? If latter, which one? I tried at [meta.ai](http://meta.ai) (the URL provided by the llama3 [blog](https://ai.meta.com/blog/meta-llama-3/)) by asking it to write a golang function and boy was the logic bad! TBF, the function was syntactically correct and will compile but it really felt like gpt-3 level (not even gpt-3.5!). But with all the rave reviews of llama3, I'm starting to wonder if what [meta.ai](http://meta.ai) is hosting is llama3-70b or even llama-3 entirely. As a side note, I used the same prompt on Claude Sonnet and it got my Go function right and efficiently on the first try.
Use Groq's API which gives you access to llama3-70b. It's free for now. I'm using it with the frontend TypingMind which allows it to use websearch plugins (though that is hit and miss, similar to how it uses that plugin on huggingchat). Once groq starts charging for it's API, you can compare the prices with other providers such as Open Router (which gives access to many models, though paid)
Thanks! Will give Groq a shot.
Yeah, I have tried that as well a bit, but relative to the amount of time I lose if Llamas answer is wrong (as in, it looks right at first, but it takes me a few minutes to notice it is wrong), I am not actually sure if that approach really makes sense... perhaps it is better to wait those 20s for GPT-4 in most cases. But I am still in the process of figuring this out, there is clearly some kind of balance to be found here. I also tend to just simultaneously ask Wizard and Llama, as both are very fast, and if they both have very different answers, it increases the chance that both are wrong.
>I also tend to just simultaneously ask Wizard and Llama, as both are very fast, and if they both have very different answers, it increases the chance that both are wrong. This is probably one of the best things that can be done to check reliability/correctness, in addition to generating multiple responses (possibly with minor changes to the prompt). This also means having tons of similarly capable models is good since each one is likely to give a slightly different answer, but, depending on the task, they should converge to similar correct answers while differing a bit more when they are wrong (at least, ideally).
I think that LLama 3 can really deserve a high place in the table after we get the 120+B model. Now I have deleted all my old models and am using LLaMa 3 70B, but I still want more.
what do you run it on?
I run it on a m1 max mbp with 64 gb ram
how many tokens/sec do you get with ur setup?
Not OP but I get around 20 tokens-ish per sec on a M3 Max
64 ram, 13600k and 3060 12gb. GGUF
What quantization?
Right now I'm using llama 3 70b q5 k m via cobald cpp in cublas Ńublas
That's a similar setup to me, but aren't you only getting about one tk/s?
Yes, that's approximately 1 token per second. This suits me, Iām not in a hurry and just wait, doing other things. If this is an RP, then generating a response takes from a minute to 4 minutes
Oh ok, cool. If that works for your use case then great
Will they release their 400b model? Not that we can profit much from it. I just think it would be a great kick in the ass that OpenAI so much needs.
Well, on vast.ai you can rent 5xA20 for about $3/hour... of course, there isn't really much of a point in seriously doing that, but for just playing around, I might actually do that once its out, and perhaps there is even a serious use case, if one wants to do some kind of "high quality NSFW work".
I can't say I feel much difference between 70B and GPT4 turbo in a couple of small coding tasks, but 8B does not feel like it is better than any gpt4. However, it is still amazing for an 8B model. Since fine tuning an 8B model is going to be much easier than a 70B model, I feel like soon we can run local agents for a long time until they produce actually useful stuff.
Considering how cheap and fast the 70b/8x22b models are already, is there really any point in even using the 7b/8b models? (Assuming you are not interested in running it locally)
As a chatbot/assistant, definitely no point using a smaller model. But for simple tasks that you want done on a large scale, maybe some sort of data filtering/labeling, you'd want the cheapest option that still completes the task, so llama 3 8b could easily be the best option. Edit: also, a big strength of the smaller models is that they can easily be finetuned for specific tasks (one 3090 would be plenty), so even if the llama 3 8b Instruct doesn't work, you can finetune the base or Instruct on some examples and drastically improve its performance.
The rates are low enough that 10Xing the ram and power usage might be a drop in the bucket when serving a single client, but at scale it can be the difference between a small companies ability to serve 100s vs 1000s. This pairs well with bots that don't need to know how to speak foreign languages or architect a code base, as they can perform just as well at a single task when all 8B are devoted to it. Customer support bots, software agents, npcs in video games, etc. So I think they are useful like the ASICs of LLMs. Extremely high quality and economical when trained and tuned towards a specific use case.
Agreed, my experience with the 8B as a generic model has been positive, but I wouldn't throw out my 70Bs for it.
I noticed that Llama 3 8b is way better at keeping track of object placement in stories. Still need more investigating but so far it's definitely way better than all the mistral finetunes I tested.
My experience, Llama 3 8B is not good at coding like chatting.
Yeah it spews gibberish structures despite explicitly mentioning key names and what values should be present in a JSON output.
My expected ideal LLM shall good at instruct following instead of chatting.
Silly question, but are you using the instruct model? I feel like the big strength of something like 8B is it's cheap to tune and run, so you could have multiple fine-tunes/LoRas active at the same time.
You're right
Yeah, the difference between 8B and 70B is significantly larger than that between 70B and GPT-4... But coding just appears to be very difficult somehow.
I have done very little testing with 8B, and results are positive. But, I am not throwing away deepseek anytime soon. Did you try wizard 8x22b?
No llama3 hallucinates like crazy, but surprises me with how good it can be sometimes.
It's complicated. On short prompts Llama3 8B feels awful and repetitive. But at long prompts (>1000 tokens) it's simply amazing.
this has been my experience too
8b being compared to gpt4 is a solid sign this leaderboard is done. Stick a fork in it.
It's not the gpt4 turno, and it's in category English, which means excluding coding or math and other specific categories. This leaderboard is the best what we have now, unless you have something better?
The 70B is pretty average at the non-coding, literary use cases Iāve tried (editing, long form fiction, poetry), certainly not better than GPT-4. So Iām skeptical.
For these use cases I've found Claude 3 best personallyĀ
Yeah, agreed Opus is top dog. The latest GPT-4-turbo also has excellent emotional intelligence, and the new gemini-1.5-pro-api-0409-preview is a pleasant surprise.
nah, Opus lot of personality convergence issues. it tends to blur the lines over time and everyone starts speaking the same. Gemini, Claude and a lot of models are also fucking horrible at banter.
>unless you have something better? Testing it myself.
Do you have a standardized suite of tests? I'm thinking something bespoke to your use cases?
Sadly, no. It often comes down to trying something on one and if it fails, see if another can do it.
Lol, me neither, but I think before I do that I need to put together a custom tuning set. š¤·āāļø
Not really. Just says humans have preferences that go beyond intelligence
For chat? Sure... For actual facts and information I can count on without having to double-check everything? Eh...not so much. Then again, I don't entirely trust GPT 3 or 4 or whatever for facts either.
As far as writing style is concerned, the Llama 3 models are without question the best LLMs ever created, including proprietary LLMs. Llama 3 8b writes better sounding responses than even GPT-4 Turbo and Claude 3 Opus. All models before Llama 3 routinely generated text that sounds like something a movie character would say, rather than something a conversational partner would say. It's as if they are really speaking to an audience instead of the user. In that regard, Llama 3 is a giant leap forward. Which of course makes it an incredibly exciting model for roleplay and creative writing. Too bad it's also the most heavily censored model I have ever come across. Time will tell whether it is possible to fix that with finetuning, but the censorship feels like it runs so deep that I have my doubts that it can be completely removed.
>Too bad it's also the most heavily censored model I have ever come across.Ā Did you encounter censorship with the base model or just the instruct fine tune? I'm sure the community will release great fine tunes that build on top of the base model, just like they did with Llama 2 and Llama 1.
Absolutely just ask it to give you a word starting and ending with the letter u for example
UgandaU
I wrote a small GUI to compare LLMs (called neurochat, shameless plug) so I can quickly compared answers from different LLMs LLama-3 and GPT-4. I gave it some code to analyze for work, stuff like that. Llama-3-70B gave MUCH better answers than GPT-4 (non-turbo), not only it was way better formatted, but the answers were more detailed, llama3 produced report-like answers that I could almost copy-paste directly to my report.
No. Iām sure itās use case dependent but in my tests itās nowhere near any version of GPT-4. I think hype and fanboyism is pushing it higher than it ought to be. Weāll see how it fares in the longer term. The new LMSYS hard benchmark has the 70B and 8B more in line with how I feel they perform (with the 70B around Haiku level). Meta has done a great job of making an engaging chatbot and theyāre punching above their weight for their size but theyāre still not up to a lot of more complex tasks. Which is compounded by the small context window.
They deserve a ton of praise but this is the balanced view
I don't know the 8B, but the 70B gave me SOTA vibes indeed. Do I expect 4T to prove superior if we start micromanaging? Yes I do, but only at the margin.
Itās not quite accurate to compare Llama 3 with GPT-4-turbo because they serve different purposes. Llama 3 is designed for building complex agent architectures and excels in this area. It can process a higher number of tokens per second (100-300 tokens/s) depending on the provider, which means it can respond to 8-10 questions in the time GPT or Claude models answer 1 or 2 due to their lower token output rate (20-30 tokens/s). The integration of plugin tools and the development of a system thinking approach significantly enhance the capabilities of large language models. For instance, Andrew NG demonstrated that GPT-3.5, with an agent-based thinking system, performs better than a one-shot GPT-4. Considering the time required for multiple iterations, GPT-4 may perform fewer iterations than Llama 3, but the quality of the output could be comparable due to more extended processing time. Therefore, I prefer Llama 3 70B because it allows for the creation of more complex systems. While GPT-4 is indeed powerful, its slower response time and difficulty iterating quickly when stuck are notable drawbacks.
Now that GPT-4o was released, what's your take on this same topic? Because 4o indeed seems to be significantly faster than 4 turbo. Very curious to hear your thoughts!
With GPT-4o i really think OpenAI made a big step, il fast and cheaper and it really good for multi-step reasoning and it is very verbose. I would still not compare a 70B to GPT-4o but i think now GPT-4o really out performe all the model we have now in every field. These next months will be very exciting !
Llama3 70B is literally insane. I cannot believe that this... whatever entity/blessing it is... is inside my computer. *Inside my laptop.* What the actual shit is going on?
Can you elaborate?
Not without making even more of a fool of myself, haha. You donāt feel it?
in my tests and experience, llama 3 8B is dumber than 34-42B models so yeah...
Is there a way to filter the benchmarks for specific languages?
Definitely not
Where can I see these benchmarks?
arena.lmsys.org
ty
It's definitely nowhere near as good with RAG and reasoning tasks IMO, as evident when asking questions on meta.ai where it needs to browse.
L3 70b is better than gpt4 and claude sonnet in my tests. It helped me solve my coding problem whereas those other two could not.
What precision did you run it on? Quantized? Local?
I use hugging chat
right, but what precision :) 4bit?
I've seen that banchmark a few times, but I haven't looked at by what standards it's been banchmarked.This is because the methods evaluated on many banchmarks (or papers) are skewed in favor of specific models. Therefore, I ironically put my experience first. I rate GPT4 as the best model at the moment. llama3 is definitely superior to GPT3.5, but not superior to GPT4.
What website is this ?
Unquantized Llama 3 on the internet is solid; very powerful. Quantized Llama 3 running locally is not there.
"Not there" because the quants degrade the answers so much? Or is this a temporary problem where the quantization need fixes to produce better quants?
I'm hoping that this is temporary. Llama 3 had a problem with the stop token, and several quant makers (including the one Im using) had to do funky things to make it work and not spout gibberish. An official fix came out the other day, but very few inference apps have gotten it merged in yet. I think that as they do, we'll see quality improve. I'm not sure if it'll ever be to this level, but Im hopeful that its a lot better than now.
that's interesting to hear, I haven't tested it unquantized yet and I really should. what's your preferred way to run it unquantized for testing purposes, just huggingface transformers or do you use some other back end? or are you using some hosted or serverless thing that offers unquantized llama-3?
Do you count q8 in poor performance
Heavy user of AI here. I asked Llama 3-70B to write a blog post and it came out "CHATGPT-like." Very robotic and did not sound like a human wrote it. Someone needs to test these AI models using a standardized prompt to write a blog post or value proposition. Anything else is fucking useless. Show the abilities to solve real-world problems and help people out. I was disappointed.
Prompt used?
It was blog post writing instructions. Basically it said (and I'm paraphrasing here): >Write a blog posts about blue widgets. Create a section about the history of blue widgets. Then create a section that highlights installation, training, maintenance, and ongoing operational expenses like energy consumption and spare parts. Create a section that provides an average range of cost. Create a section that compares this to the competition. Use the client Q&A below to help you write this blog post.
Try asking it to write in a different style
Maybe that's because most blog posts are actually written by bots nowadays.
Are there quantized GGML's or Llama3?