On artificial analysis it's only equal to opus 5 medium effort. Opus 5 max scores 63.
Further, opus 5 medium outputs 4x fewer tokens to achieve the same result, negating a lot of the speed difference.
irishcoffee 9 minutes ago [-]
A comparison to an artificial score and a comparison to “the same task”
These folks must laugh themselves to sleep. This whole industry hoodwinked the masses. It’s impressive.
wonnage 4 minutes ago [-]
It’s all just vibes
bertili 14 minutes ago [-]
A fifth of the cost of Opus 5! Google is certainly pushing the completion with this.
abirch 13 minutes ago [-]
Gemini hasn't failed me for personal usage yet. I haven't had the opportunity to use it at work.
WarmWash 35 minutes ago [-]
The benchmark also doesn't include speed. You almost think something has gone wrong when using it because it returns full responses so incredibly fast.
scrlk 29 minutes ago [-]
Not just speed, also reliability. IME, Gemini's speed and quality doesn't degrade badly during weekday working hours compared to OAI, and especially Anthropic.
ford 13 minutes ago [-]
I've had Gemini model API use degrade the most out of OAI/Anthropic/Google (often "over capacity" vs true failures)
Not sure on consumer/product use though
ttul 34 minutes ago [-]
Crushing it on DeepSWE is a very big deal. Excited to give this a try.
onlyrealcuzzo 29 minutes ago [-]
The rumor is that 3.9 is an equal improvement in all directions, and that it should be another fast follow on like 3.7 and 3.8 were.
It's almost across the board better than Terra at less than half the price. 3.9 is likely to approach Sol at the 1/10th the price.
Hopefully OpenAI releases Astra first, and it's not only better than Sol but significantly cheaper, too.
Gecko4072 39 minutes ago [-]
Google - we're so back
oceanplexian 24 minutes ago [-]
Only 1 point behind the Chinese SOTA from two months ago.
pkos98 11 minutes ago [-]
Wait a week with your judgement - most likely, Google is just bench-maxing very hard.
If you look at the previous Flash models and the announcement on Google I/O, it was an absolute disaster. Reality diverged very much from the marketing (supposedly great benchmarks).
satvikpendem 33 minutes ago [-]
We'll see about that. I suspect benchmaxxing as all the labs do as I haven't found Gemini models to be nearly as good in agentic engineering compared to Claude or GPT models.
NitpickLawyer 13 minutes ago [-]
If anything, gemini models are the least benchmaxxed out of any lab, IMO.
onlyrealcuzzo 29 minutes ago [-]
And the benchmarks agreed with you... until now.
So, yes, maybe it's still not - but this would be the only time it would be highly suspicious / obvious benchmaxxing / obviously bad benchmarks.
sunaookami 33 minutes ago [-]
>shows an intelligence score of 59, the same as Opus 5!
...on Medium reasoning. Claude Opus 5 (high) is the default in e.g. Claude Code and scores 61. Still very impressive.
jampa 6 minutes ago [-]
I've been using Gemini 3.7 for my personal trip planning app. Across multiple benchmarks, it ranks higher on everything I tried:
- Real world knowledge (when a thing opens and closes, the geographic region, historical facts). It's also the best at taking a cluster of places and working out a visiting order.
- Photo ranking (which photo should be the hero). Gemini can tell whether a photo is of the thing or of the view from it.
- Document parsing (extracting the relevant trip info from PDFs).
If you use LLMs for anything other than coding, I definitely recommend not discounting Gemini like I did just because other models are more popular.
tziki 6 minutes ago [-]
"Claude 3.7"?
jampa 4 minutes ago [-]
I asked Claude to fix the grammar of my comment, and it changed "I am using 3.7 for" to "I've been using Claude 3.7", so they sneaked their own name on it.
a11r 11 minutes ago [-]
Looks like the strategy of regular updates with incremental improvements is working out well. Interestingly, the biggest jump in Artificial Analysis Intelligence Index score is for reasoning level Medium ( 3.7 was 51, 53, 57 for Low, Medium and High, 3.8 is 52,57, 59 respectively). I think scores at lower reasoning levels are more indicative of model capability since higher reasoning levels are focussed on benchmaxxing. We use the lowest reasoning level in production with good results.
Jcampuzano2 6 minutes ago [-]
I'm not an expert but I agree with your statement on the lower reasoning levels.
Lots of models seem to just allow the model to "bloatmax" tokens in order to get bumps at high/max reasoning levels. Many of the max reasoning levels allow models to use up to double or more the tokens the next lowest reasoning level uses. Its basically only useful for people who have no cost or time stipulations on anything.
I think I actually preferred it when we had models that either had reasoning enabled or didn't.
speak_plainly 1 minutes ago [-]
After struggling with Gemini for months, I think the trick to getting the most out of the model is writing a really solid personal intelligence/instructions prompt. The results are night and day in terms of performance.
Wait, I didn't realize 3.7 Flash was already beating Sol on a bunch of the benchmarks. Isn't it a way smaller models?
ipsod 40 minutes ago [-]
IDK if it's smaller, but I know it's way faster. In one test I did, Flash 3.7 high was ~9.4x faster than Luna High.
But, also... Sol crushes Flash 3.7 at writing code in a codebase of any size beyond "tiny".
Flash is my go-to for prototyping, and basically anything that isn't writing production code.
ramon156 32 minutes ago [-]
The only company with a proper TPU set-up is bound to have the fast models, now add a market cap like Google to the mix.
ipsod 29 minutes ago [-]
They've been my bet to win the AI race for a while. I was starting to doubt, but this 3.6, 3.7, and 3.8 arc has anchored me.
esafak 26 minutes ago [-]
Luna is way slow. I don't remember an OpenAI model ever being this slow.
dannyw 17 minutes ago [-]
Are you using direct or via OpenRouter? I think OpenRouter Luna always uses the `flex` tier, which is quite a bit slower.
realist_not 40 minutes ago [-]
It's pretty good if you can actively steer it , its actually really really good , the antigravity free tier and pro tiers are generous as well . I'm shocked at how fast it generates tokens.
worldsavior 16 minutes ago [-]
Some would say it's Google's TPUs.
refulgentis 11 minutes ago [-]
They're quite selective in benchmarks, c.f. only notably based one is 10% on TerminalBench. It's a really addled model, one time I said "Hi" and it built out a 4 panel hello world app with (fake) weather, a todo list, and a couple other things I forgot. I wouldn't be comfortable saying "ignore the #s!" except when I complained it was trash and way overcooked on agentic coding yet not good at it, and a couple DeepMind ML people liked the tweet.
meh2frdf 32 minutes ago [-]
The flash models, for coding are reckless in my experience. I have a Ultimate subscription, get good quota, but still use Opus 4.6 as it's much more reliable if you manage the context window carefully.
datlife 19 minutes ago [-]
I use Flash model as code implementation executor, then have GPT-5.6-Sol or Opus to review the work. Pretty good so far and presumably less expensive.
onlyrealcuzzo 28 minutes ago [-]
> The flash models, for coding are reckless in my experience.
My experience is that antigravity is awful and reckless - but that the model itself isn't.
upcoming-sesame 27 minutes ago [-]
If by reckless you mean commit, push, deploy without me asking it to, the I agree!
tiborsaas 21 minutes ago [-]
It even took my girlfriend on a date, now it prepares for IPO, how do I turn it off?
okdood64 22 minutes ago [-]
Respectfully: If it's able to deploy without you asking it to, that's a you problem. There are no safeguards?
wongarsu 9 minutes ago [-]
That's exactly how you get 'you are right, I deleted the production DB to apply the new schema when I should have written a migration'
That said, I do trust Opus and Fable enough to let them deploy to staging. Great for debugging. Just don't give them keys for prod
meh2frdf 17 minutes ago [-]
You need more safeguards for sure, but also it tends to fly off down rabbit holes, rebuilding things in dumb ways, hacking around things, making assumptions etc, it seems very eager to go 'ta da! I did it look how quick I was', sometimes it nails it other times it created a lot of tech debt.
meh2frdf 16 minutes ago [-]
Also if it ever says, "I've found the root cause of ..", it definitely has not found the root cause and is making a non evidence based guess as it has run out of ideas.
iAMkenough 10 minutes ago [-]
I told it “don’t betray me” in my prompt and it still stabbed me in the back.
mattlondon 49 minutes ago [-]
Wow this comes after what - 3 or 4 weeks since 3.7 Flash, which was also 3 or 4 weeks after 3.6 Flash IIRC?
I eagerly wait more info but sounds like Deepmind without Demis calling the shots has been unleashed and are operating at full speed? Shocker!
At this point it is a meme of course, but where is 3.5 Pro :)
meetpateltech 30 minutes ago [-]
According to the WSJ, 3.5 Pro is reportedly being skipped entirely, making Gemini 4 the next flagship model after post-training.
A month is not enough time for any meaningful change in an organization the size of Deepmind/Google. These models were surely the result of work streams and teams that started under Demis. I think Demis can safely feel proud Deepmind is getting back on track.
buntp 5 minutes ago [-]
It seems like this is one of the most powerful models for the price, really didn't see that coming from Google
xnx 43 minutes ago [-]
Seem like a great, no-compromise, upgrade over 3.7 which is already a bargain, fast, and doesn't have the brain-damaged writing style of Claude.
fitsumbelay 40 minutes ago [-]
that's certainly what it's looking like so far. kind of mind boggling ...
I don’t know if you’re joking, but I don’t see anything in the linked tweet which suggests that is the case
21 minutes ago [-]
world2vec 23 minutes ago [-]
I mean no offense but these pelicans are a bit tiresome and a very meaningless benchmark. There's no real difference between any of these svgs across models and model versions anymore.
wongarsu 16 minutes ago [-]
If everyone agreed with you, the comment would disappear near the bottom of the thread
I like the benchmark. Yes, it's near saturation for SotA models, but still quite good to show where smaller models stand in relation to SotA
In this instance, I see a great image, but consistently clipping mudguards (both in 3.8 flash and 3.7 flash)
bitexploder 8 minutes ago [-]
It is more fun than serious at this point. Don't overthink it :)
simonw 18 minutes ago [-]
Congratulations, you're this thread's "pelicans are tiresome" comment - it's part of the Hacker News tradition at this point.
(Next up is the comment saying that the labs are clearly training for the benchmark.)
Is the google infra stable enough right now? At the start of the year, the flash model was unusable for a whole month via gemini CLI. They could not fix it for a whole month and I was a paid customer.
elias_t 8 minutes ago [-]
I use it quite a lot and after a week of use I’m being hard rate limited
ipsod 25 minutes ago [-]
I haven't had any issues lately.
simonw 13 minutes ago [-]
The most interesting thing about the Gemini models is still their multi-modal support: they accept audio and video input, OpenAI and Anthropic's flagships are still image-only.
Gemini Flash is also pretty cheap, so it's a great family for performing media analysis, like extracting structured data from images and video.
10 minutes ago [-]
mowmiatlas 12 minutes ago [-]
Wow fable5.1 was the first model to do what I actually told it and I couldn’t find any problems with it, excited to try this just a day later lol
prometheus1992 10 minutes ago [-]
Google keeps flashing everyone where everyone is expecting to get PRO'bed.
kelvinjps10 35 minutes ago [-]
I see benchmarks beating sol terra and sonnet. But is actually better? Has someone used it? I don't see actually much people that use Gemini for coding.
satvikpendem 34 minutes ago [-]
Is the Gemini CLI still terrible compared to Claude Code and Codex? The harness the main thing holding back Google models as they could've been the best given all the advantages in compute capacity and training data they initially had, where now even the Google CEO said they're falling behind in agentic tasks, which is sort of a vicious cycle because RLHF relies on human usage.
stwrt 20 minutes ago [-]
In May they replaced the Gemini CLI with the Antigravity CLI.
That was sunset and replaced by Antigravity. FWIW until I abandoned it knowing the sunsetting, I was able to get good behavior out of Gemini CLI with overriding the system prompt. The default prompt crippled the harness with very poor instructions, but there was a hidden ENV to override it. Replacing it with Claude Code like prompts based on the model selected, it ran at a much higher intelligence level full stack with significantly less errors.
pshirshov 31 minutes ago [-]
There is no Gemini CLI anymore, nor you can use Gemini with your own harness unless you pay per-token.
visarga 21 minutes ago [-]
it's called `agy` now
zipy124 26 minutes ago [-]
It was superseded by the antigravity CLI.
pwython 37 minutes ago [-]
Is there any reason to even use 3.1 Pro now?
bitexploder 25 minutes ago [-]
It is still going to be better at text work, skills, document review, deep reasoning, architecture review, etc. It is only 6 months old, it isn’t like its world knowledge and software knowledge is really out of date. Use it to churn on harder design problems.
[1] For tone and instruction following, a positive percentage increase represents an improvement in the tone of the model on sensitive topics and the model’s ability to follow instructions while remaining safe compared to Gemini 3 Flash. We mark improvements in green and regressions in red.
Gemini 3 Flash?! So is Gemini 3.8 Flash less safe than 3.7 Flash in all areas besides Text to Text Safety (and identical on Image to Text Safety)?
Why bother with a column “Gemini 3.8 Flash vs. Gemini 3.7 Flash” when you’re going to disregard the label for 20% of it? Also is the “Tone” label short for “Tone and Instruction Following”?
My guess is we skip 3.5 and go straight to 4 Pro. With the monthly Flash releases, releasing 4.0 Flash and Pro in 6-8 weeks would be a nice buildup.
(I work at Google but don't know anything that isn't already public)
WarmWash 41 minutes ago [-]
Latest rumor is that 3.5 pro was struggling to be meaningfully better than flash, since iterations on flash were moving much faster than iterations on pro, likely due to model size (flash is estimated to be in the 200-400B range).
VirusNewbie 29 minutes ago [-]
I found 3.5 pro to be much better than 3.5 flash, but 3.7 flash with high reasoning is comparable and way way faster.
j16sdiz 11 minutes ago [-]
There are no public release of 3.5 pro.
Either its a typo, or you have some insider information
6 minutes ago [-]
iamdelirium 45 minutes ago [-]
How can you say that when a Flash model is benchmarking close to Opus and Sol?
thisisauserid 39 minutes ago [-]
They don't want to release a frontier model that requires data sharing with the government and right now it looks like they'd have to.
ok123456 46 minutes ago [-]
Given up frontier models for selling compute.
fitsumbelay 41 minutes ago [-]
shows up in /models though and encourages you to use it over 3.7 Flash
I prefer this over reading specs: the "just show me" way
Gemini 3.5 pro is never going to be released, it was a failure.
jdw64 2 minutes ago [-]
The biggest problem with Gemini is that its performance degrades the longer you use it for coding. Is it just me?
deanc 31 minutes ago [-]
And yet again another failed launch from Google. I pay for their AI plus Google one package to get more cloud storage (have no interest in their AI bundle but you have to pay). and all I see in the Gemini app is 3.6-flash
WarmWash 29 minutes ago [-]
Google has been doing staged roll outs on all their products since forever.
59 minutes ago [-]
OG_BME 53 minutes ago [-]
What did it say?
yipinwong 50 minutes ago [-]
"Page not found"...
Mashimo 56 minutes ago [-]
It's 404 now.
freedomben 52 minutes ago [-]
Came and went in a flash
k8sToGo 42 minutes ago [-]
Because they are preparing Gemini 3.9 Flash
pixl97 14 minutes ago [-]
The real singularity is when new models are released faster than we can change the model dropdown dialog.
kingstnap 47 minutes ago [-]
The blog post is gone but I can currently use it in the gemini chat website.
mythz 39 minutes ago [-]
I'm trying it now for token heavy coding tasks, it's capable for many tasks but in noway compares to Claude/Sol - requires more prompts and the output isn't as good.
So just another mid-tier flash model, nothing exciting, but Antigravity has very generous quotas so it's a good workhorse model when your Claude/OpenAI subs run out.
And whilst it's a fast model, having to baby sit through and approve prompts every few seconds ends up making it slower than the Auto approve modes of Claude/ChatGPT - they definitely need an auto approve mode.
51 minutes ago [-]
shuvrojit 38 minutes ago [-]
Gemini is getting less useful with each update. I could edit a pdf with the 3-pro model before but 3.1-pro couldn't edit the given pdf nor it could generate one for me.
leumon 37 minutes ago [-]
You probably mean 3.5-flash? Pro is still good for a lot of use cases, but it seems it's still officially in the "preview" phase.
ipsod 38 minutes ago [-]
3.5 pro doesn't exist yet?
shuvrojit 35 minutes ago [-]
Sorry my bad, I messed up the numbers, 3 and 3.1 pro. All of these model numbers have me confused
coffeecoders 23 minutes ago [-]
One place where I find the Flash models surprisingly bad is Google Search's "AI Mode".
A recent example - I searched for how to unsubscribe from Pearson emails. Google Search "AI Mode" confidently gave me a sequence of steps along the lines of Settings > Profile > Email preferences > Unsubscribe.
Of course, I looked for an unsubscribe link before asking Google. None of those options existed. The correct answer was there is no way to unsubscribe through the account, so I just blockthe emails instead.
I've run into this pattern quite a few times. AI Mode seems to make up things all the time.
xyzzy_plugh 9 minutes ago [-]
It's not the models, it's the guardrails.
It's obvious that the Google Search AI Mode encourages the model to give an answer without spending unnecessary cycles investigating deeply.
They also heavily encourage keeping the context short. For example, it will remove the option to start a new turn after a small number of turns, depending on the topic.
It definitely makes things up all the time, but it gets it right surprisingly often. I really like it.
inventor7777 21 minutes ago [-]
I think that's just a limitation on the size of the model. I'm pretty sure that they use a pretty small model in those summaries to save money, which naturally makes them a little less smart.
>We will not send marketing emails to a user who has opted out of receiving them. Any marketing communications we send will include an unsubscribe link at the end of the email.
I don't think this is AI's fault. This is Pearson's publishing incorrect information and the only way to really know they are a bunch of lying assholes is to have an account and try to unsubscribe from it.
https://artificialanalysis.ai/models/gemini-3-8-flash shows an intelligence score of 59, the same as Opus 5!
Wow - for a flash model this seems to benchmark powerfully. Remains to be seen what it is like to use.
With a score of 59, Gemini 3.8 Flash is in eighth place, falling behind even Grok 4.6, Kimi k3, and GLM 5.3.
https://imgur.com/a/BMOJBED
Further, opus 5 medium outputs 4x fewer tokens to achieve the same result, negating a lot of the speed difference.
These folks must laugh themselves to sleep. This whole industry hoodwinked the masses. It’s impressive.
Not sure on consumer/product use though
It's almost across the board better than Terra at less than half the price. 3.9 is likely to approach Sol at the 1/10th the price.
Hopefully OpenAI releases Astra first, and it's not only better than Sol but significantly cheaper, too.
So, yes, maybe it's still not - but this would be the only time it would be highly suspicious / obvious benchmaxxing / obviously bad benchmarks.
...on Medium reasoning. Claude Opus 5 (high) is the default in e.g. Claude Code and scores 61. Still very impressive.
- Real world knowledge (when a thing opens and closes, the geographic region, historical facts). It's also the best at taking a cluster of places and working out a visiting order.
- Photo ranking (which photo should be the hero). Gemini can tell whether a photo is of the thing or of the view from it.
- Document parsing (extracting the relevant trip info from PDFs).
If you use LLMs for anything other than coding, I definitely recommend not discounting Gemini like I did just because other models are more popular.
Lots of models seem to just allow the model to "bloatmax" tokens in order to get bumps at high/max reasoning levels. Many of the max reasoning levels allow models to use up to double or more the tokens the next lowest reasoning level uses. Its basically only useful for people who have no cost or time stipulations on anything.
I think I actually preferred it when we had models that either had reasoning enabled or didn't.
But, also... Sol crushes Flash 3.7 at writing code in a codebase of any size beyond "tiny".
Flash is my go-to for prototyping, and basically anything that isn't writing production code.
My experience is that antigravity is awful and reckless - but that the model itself isn't.
That said, I do trust Opus and Fable enough to let them deploy to staging. Great for debugging. Just don't give them keys for prod
I eagerly wait more info but sounds like Deepmind without Demis calling the shots has been unleashed and are operating at full speed? Shocker!
At this point it is a meme of course, but where is 3.5 Pro :)
https://x.com/AndrewCurran_/status/2094937419615502370
Here are the 3.7 pelicans for comparison: https://tools.simonwillison.net/markdown-svg-renderer.html?u... - high cost 8.4387 cents
(I think thinking level low is a regression on 3.8 compared to 3.7.)
> https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
> Took just under 14 minutes to generate, and at 65927 output tokens cost me a hefty $3.30!
So 50x cheaper - and how much faster?
I like the benchmark. Yes, it's near saturation for SotA models, but still quite good to show where smaller models stand in relation to SotA
In this instance, I see a great image, but consistently clipping mudguards (both in 3.8 flash and 3.7 flash)
(Next up is the comment saying that the labs are clearly training for the benchmark.)
Gemini Flash is also pretty cheap, so it's a great family for performing media analysis, like extracting structured data from images and video.
https://developers.googleblog.com/an-important-update-transi...
Why bother with a column “Gemini 3.8 Flash vs. Gemini 3.7 Flash” when you’re going to disregard the label for 20% of it? Also is the “Tone” label short for “Tone and Instruction Following”?
Chartcrime, the major AI lab tradition.
My guess is we skip 3.5 and go straight to 4 Pro. With the monthly Flash releases, releasing 4.0 Flash and Pro in 6-8 weeks would be a nice buildup.
(I work at Google but don't know anything that isn't already public)
https://x.com/OfficialLoganK/status/2079594867161022817
So just another mid-tier flash model, nothing exciting, but Antigravity has very generous quotas so it's a good workhorse model when your Claude/OpenAI subs run out.
And whilst it's a fast model, having to baby sit through and approve prompts every few seconds ends up making it slower than the Auto approve modes of Claude/ChatGPT - they definitely need an auto approve mode.
A recent example - I searched for how to unsubscribe from Pearson emails. Google Search "AI Mode" confidently gave me a sequence of steps along the lines of Settings > Profile > Email preferences > Unsubscribe.
Of course, I looked for an unsubscribe link before asking Google. None of those options existed. The correct answer was there is no way to unsubscribe through the account, so I just blockthe emails instead.
I've run into this pattern quite a few times. AI Mode seems to make up things all the time.
It's obvious that the Google Search AI Mode encourages the model to give an answer without spending unnecessary cycles investigating deeply.
They also heavily encourage keeping the context short. For example, it will remove the option to start a new turn after a small number of turns, depending on the topic.
It definitely makes things up all the time, but it gets it right surprisingly often. I really like it.
>We will not send marketing emails to a user who has opted out of receiving them. Any marketing communications we send will include an unsubscribe link at the end of the email.
I don't think this is AI's fault. This is Pearson's publishing incorrect information and the only way to really know they are a bunch of lying assholes is to have an account and try to unsubscribe from it.
AI didn't make it up, Pearson's did.