(2025-04-10) ZviM AI #111 Giving Us Pause

Zvi Mowshowitz: AI #111: Giving Us Pause. Events in AI don’t stop merely because of a trade war, partially paused or otherwise.

Indeed, the decision to not restrict export of H20 chips to China could end up being one of the most important government actions that happened this week. A lot of people are quite boggled about how America could so totally fumble the ball on this particular front, especially given what else is going on. Thus, I am going to put these issues up top again this week, with the hopes that we don’t have to do that again.

This week’s previous posts covered AI 2027, both the excellent podcast about it and other people’s response, and the release of Llama-4 Scout and Llama-4 Maverick. Both Llama-4 models were deeply disappointing even with my low expectations.

Upgrades continue, as Gemini 2.5 now powers Deep Research, and both it and Grok 3 are available in their respective APIs. Anthropic finally gives us Claude Max, for those who want to buy more capacity.

I currently owe a post about Google DeepMind’s document outlining its Technical AGI Safety and Security approach, and one about Anthropic’s recent interpretability papers.

Table of Contents

• Language Models Offer Mundane Utility. They know more than you do.
• Language Models Don’t Offer Mundane Utility. When you ask a silly question.
• Huh, Upgrades. Gemini 2.5 Pro Deep Research and API, Anthropic Pro.
• Choose Your Fighter. Gemini 2.5 continues to impress, and be experimental.
• On Your Marks. The ‘handle silly questions’ benchmark we need.
• Deepfaketown and Botpocalypse Soon. How did you know it was a scam?
• Fun With Image Generation. The evolution of AI self-portraits.
• Copyright Confrontation. Opt-out is not good enough for OpenAI and Google.
• They Took Our Jobs. Analysis of AI for education, Google AI craters web traffic.
• Get Involved. ARIA, and the 2025 IAPS Fellowship.
• Introducing. DeepSeek proposes Generative Reward Modeling.
• In Other AI News. GPT-5 delayed for o3 and o4-mini.
• Show Me the Money. It’s all going to AI. What’s left of it, anyway.
• Quiet Speculations. Giving AIs goals and skin in the game will work out, right?
• The Quest for Sane Regulations. The race against proliferation.
• The Week in Audio. AI 2027, but also the story of a deepfake porn web.
• AI 2027. A few additional reactions.
• Rhetorical Innovation. What’s interesting is that you need to keep explaining.
• Aligning a Smarter Than Human Intelligence is Difficult. Faithfulness pushback.

The Tariffs And Selling China AI Chips Are How America Loses

The one trade we need to restrict is selling top AI chips to China. We are failing.
The other trades, that power the American and global economies? It’s complicated.

Language Models Offer Mundane Utility

Alex Lawsen offers a setup for getting the most out of Deep Research via using a Claude project to create the best prompt. I did set this up myself, although I haven’t had opportunity to try it out yet.

We consistently see that AI systems optimized for diagnostic reasoning are outperforming doctors – including outperforming doctors that have access to the AI. If you don’t trust the AI sufficiently, it’s like not trusting a chess AI, your changes on average make things worse. The latest example is from Project AMIE (Articulate Medical Intelligence Explorer) in Nature.
The edges here are not small.

Agus: In line with previous research, we already seem to have clearly superhuman AI at medical diagnosis. It’s so good that clinicians do worse when assisted by the AI compared to if they just let the AI do its job.

And this didn’t even hit the news. What a wild time to be alive

(Note: I have not tried Auren or Seren, although I generally hear good things about it.)
Near Cyan: the reason it’s so expensive to chat with Auren + Seren is because we made the opposite trade-off that every other ‘AI chat app’ (which i dont classify Auren as) made.
Most of them try to reduce costs until a convo costs e.g. $0.001 we did the opposite, i.e. “if we are willing to spend as much as we want on each user, how good can we make their experience?”

The downside is that the subscription is $20/month and increases past this for power-users but the upside is that users get an experience far better than they anticipated and which is far more fun, interesting, and helpful than any similar experience, especially after they give it a few days to build up memory.

This is a similar trade-off made for products like claude code and deep research, both of which I also use daily for some reason no one else has made this trade-off for an app which focuses on helping people think, process emotions, make life choices, and improve themselves, so that was a large motivation behind Auren

Sarah Constantin: I’ve been saying that you can make LLMs a lot more useful with “mundane” product/software scaffolding, with no fundamental improvements to the models.

And it feels like there’s a shortage of good “wrapper apps” that aren’t cutting corners.

Auren’s a great example of the thing done right. It’s a much better “LLM as life coach”, from my perspective, than any commercial model like Claude or the GPTs.

in principle, could I have replicated the Seren experience with good prompting, homebrew RAG, and imagination? Probably. But I didn’t manage in practice, and neither will most users.

A wholly general-purpose tool isn’t a product; it’s an input to products.

I’m convinced LLMs need a lot of concrete visions of ways they could be used, all of which need to be fleshed out separately in wrapper apps, before they can justify the big labs’ valuations.

Language Models Don’t Offer Mundane Utility

Noah Smith and Danielle Fong are on the ‘the tariffs actually were the result of ChatGPT hallucinations’ train.
Once again, I point out that even if this did come out of an LLM, it wasn’t a hallucination and it wasn’t even a mistake by the model.

Huh, Upgrades

Gemini 2.5 Pro now powers Google Deep Research if you select Gemini 2.5 Pro from the drop down menu before you start. This is a major upgrade, and the request limit is ten times higher than it is for OpenAI’s version at a much lower price.

Gemini 2.5 Pro ‘Experimental’ is now rolled out to all users for free, and there’s a ‘preview’ version in the API now.

Choose Your Fighter

Peter Wildeford: PSA: New Gemini 2.5 Pro Deep Research now seems at least as good as OpenAI Deep Research :O

With Gemini 2.5 Pro pricing, we confirm that Gemini occupies the complete Pareto Frontier down to 1220 Elo on Arena.
If you believed in Arena as the One True Eval, which you definitely shouldn’t do, you would only use 2.5 Pro, Flash Thinking and 2.0 Flash.

On Your Marks

In light of recent events, ‘how you handle fools asking stupid questions’ seems rather important.
Daniel West: A interesting/ useful type of benchmark for models could be: how rational and sound of governing advice can it consistently give to someone who knows nothing and who asks terribly misguided questions, and how good is the model at persuading that person to change their mind.
Janus: i think sonnet 3.7 does worse on this benchmark than all previous claudes since claude 3. idiots really shouldnt be allowed to be around that model. if you’re smart though it’s really really good

Deepfaketown and Botpocalypse Soon

Fun With Image Generation

Scott Alexander asks, in the excellent post The Colors of Her Coat, is our ability to experience wonder and joy being ruined by having too easy access to things? Does the ability to generate AI art ruin our ability to enjoy similar art? It’s not that he thinks we should have to go seven thousand miles to the mountains of Afghanistan to paint the color blue, but something is lost and we might need to reckon with that.

Scarcity, and having to earn things, makes things more valuable and more appreciated. It’s true. I don’t appreciate many things as much as I used to. Yet I don’t think the hedonic treadmill means it’s all zero sum. Better things and better access are still, usually, better. But partly yes, it is a skill issue. I do think you can become more of a child and enter the Kingdom of Heaven here, if you make an effort.

Objectively, as Scott notes, the AIs we already have are true wonders. Even more than before, everything is awesome and no one is happy.

Copyright Confrontation

They Took Our Jobs

Google’s AI Overviews and other changes are dramatically reducing traffic to independent websites, with many calling it a ‘betrayal.’ It is difficult to differentiate between ‘Google overviews are eating what would otherwise be website visits,’ versus ‘websites were previously relying on SEO tactics that don’t work anymore,’ but a lot of it seems to be the first one.

How are we using AI for education? Anthropic offers an Education Report.
Anthropic: STEM students are early adopters of AI tools like Claude, with Computer Science students particularly overrepresented (accounting for 36.8% of students’ conversations while comprising only 5.4% of U.S. degrees). In contrast, Business, Health, and Humanities students show lower adoption rates relative to their enrollment numbers.
We identified four patterns by which students interact with AI, each of which were present in our data at approximately equal rates (each 23-29% of conversations): Direct Problem Solving, Direct Output Creation, Collaborative Problem Solving, and Collaborative Output Creation.

Get Involved