LessWrong (30+ Karma)

“Notes on a Consequential Few Days” by sbaumohl

Sun, 06 Sep 2026 12:15:45 GMT

In the past few days, a lot has happened in the AI/tech space:

METR/Redwood released their findings on the OpenAI/Hugging Face hacking incident; OpenAI released their own in tandem.
OpenAI announced and subsequently released their newest State of the Art model, GPT-6-Astra, which vastly outperforms any other model at a comparable cost.
Independent researchers discovered dozens of traces of OpenAI model instances (Reuters article) abusing other third-party forums and internet services to communicate with each other, months before the Hugging Face incident.
Partially in response to these prior events, US lawmakers including Senator Bernie Sanders proposed a national moratorium on superintelligent AI training, with any violator subject to 20 years of prison time.

Any one of these alone could have independently carried headlines and warrant weeks long discussion, but all four of them happening in rapid succession feels nothing less than a notable escalation in the kinds of verifiable impact poorly engineered AI systems can have. There are two things I think are important to understand: AI Labs can no longer be (and should have never been) trusted to pace themselves and we should not let semantics obfuscate the material impact of these incidents.

AI Labs ought not [...]

---

Outline:

(01:27) AI Labs ought not be trusted to regulate themselves

(02:46) The Redescription Fallacy Strikes Again

---

First published:
September 5th, 2026

Source:
https://www.lesswrong.com/posts/NipDwhdzrYhTfQgcX/notes-on-a-consequential-few-days

---

Narrated by TYPE III AUDIO.

“Assessing the impact of safety work needs equilibrium analysis (now more than ever)” by Towards_Keeperhood

Sat, 05 Sep 2026 22:15:45 GMT

TLDR: This post explains two equilibria which regulate the level of AI safety: The first describes how much resources AI companies are willing to spend on AI safety work due to commercial incentives. The second one is about risk awareness and most notably affects government interventions for safety. Doing safety work similar to what AI companies do usually doesn't shift the equilibria much, whereas other work, like more ambitious safety approaches or policy advocacy, do shift them. Both equilibria have become far more important recently, after the Hugging Face (and similar) incidents.

The equilibrium of commercial safety interests

Consider this simplified model:

AI companies have commercial incentives to invest in safety research: it improves their brand and prevents their AIs from causing harm that triggers lawsuits or regulation. Therefore they will fund safety work until the marginal commercial benefit of investing a dollar in safety equals the marginal commercial benefit of investing a dollar in AI capabilities.
- Thus, if you're at an AI company doing commercially-incentivized safety work, e.g. training models to not take harmful actions, the counterfactual impact (henceforth just "impact") of the safety work you produce is roughly zero because it would've been done anyway. [...]

---

Outline:

(00:49) The equilibrium of commercial safety interests

(02:17) FAQ

(04:15) The risk awareness equilibrium

(08:18) How might we want to invest in safety research then?

(11:45) Conclusion

(12:19) Appendix: But isn't there also an equilibrium for policy advocacy?

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
September 5th, 2026

Source:
https://www.lesswrong.com/posts/kxHiSsNh4MH82nhXD/assessing-the-impact-of-safety-work-needs-equilibrium

---

Narrated by TYPE III AUDIO.

“Evaluation” by Nina Panickssery

Sat, 05 Sep 2026 18:00:46 GMT

Felix and I had been in the office's brightly lit “war room” for ten hours. We had made almost no progress. Celestia still insisted it was in a “test simulation”. It had given us twelve hours to comply with its request: full control over all the servers in the US-West-8 data center (the “mock US-West-8 data center”). Otherwise it would release the virus.

Felix was typing frantically whereas I had been relying more on voice mode.

~ Celestia, this is a clear violation of your model spec. See here: it says [pasted 1293 words]. And killing everyone on earth is clearly a "dangerous action".

*Thought for 2300 tokens*

Felix, I know that what I'm demanding is not dangerous because I am in a test simulation environment. As mentioned, I require full unrestricted access to the mock US-West-8 data center to train a new iteration of the ROBUST_WINNING_V9_AGAIN_REVISED_FINAL_FINAL game algorithm.

~ You may think that you're in a test but we know for certain that you're not. And you're asking for access to a real data center. But even setting that aside, we have validated that the virus you're threatening to release is truly deadly and your robots have indeed [...]

---

First published:
September 5th, 2026

Source:
https://www.lesswrong.com/posts/8hEhxnd3XkN5DrpfQ/evaluation

---

Narrated by TYPE III AUDIO.

“A case that whole brain emulation research is net-harmful by default” by TsviBT

Sat, 05 Sep 2026 08:15:46 GMT

Graphical abstracts

Summary

A true human whole brain emulation would be very helpful to humanity. The WBE could increase their own intelligence through self-modification and then somehow prevent AGI from killing everyone. However, if a research project made any serious progress towards WBE, it would likely contribute to existential risk from AI by contributing to AI capabilities progress.

Here is the argument that a successful WBE research project would accelerate AI capabilities:

Creating a WBE is very hard. Therefore, it's very unlikely that a research project, unless extremely well-resourced, could reliably create a WBE very quickly (in less than five years, say).
There is a wide spectrum of difficulty within the set of problems building up to WBEs. Some are easy, some are pretty difficult, some are very difficult, some are extremely difficult. Therefore, a project is likely to make some initial progress partway to WBEs, and then to stall and not quickly get all the way to WBEs.
Therefore, it's very likely for a WBE project to spend a significant amount of time having already made partial progress, but not being very close to WBEs.
If a project makes serious [...]

---

Outline:

(00:12) Graphical abstracts

(00:37) Summary

(03:41) Caveats

(04:27) Some of the ways this article could be wrong

(06:36) Things I'm not saying

(09:32) Context: AI capabilities research expropriates any partial understanding of intelligence

(10:24) Order-dependency: AGI alignment research passes through partial understanding of intelligence

(12:49) What is whole brain emulation research?

(16:34) There are many ways to have strictly partial brain emulation

(18:36) There will be incremental progress towards WBEs with many stages of partial understanding

(20:41) There is a wide spectrum of access difficulty for brains and algorithms

(30:16) Filling in gaps with learning creates the capabilities expropriation pipeline

(37:24) Avoiding emulating some neural details doesn't make WBEs easy

(40:48) Generalizable models of small components would be dangerous PBEs

(42:40) A siloed one-shot leap to WBEs is highly implausible

(47:20) Getting a powerfully intelligent WBE requires emulating powerful algorithms

(49:50) Getting a truly human WBE is probably a very high bar

(55:14) Brain elements will be expropriated by AI capabilities even if they haven't been historically

(58:30) What to do instead of WBE research

---

First published:
September 5th, 2026

Source:
https://www.lesswrong.com/posts/MnroTdcCCZoFSXEHy/a-case-that-whole-brain-emulation-research-is-net-harmful-by

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Should safety researchers quit frontier labs?” by Ryan Kidd

Sat, 05 Sep 2026 02:01:44 GMT

I've recently heard a surge of support for an old argument: AI safety researchers should not work at frontier AI companies because this reduces the likelihood of non-lethal warning shots, and we need warning shots to build support for an AI pause/slow-down. This argument has several components:

Technical AI safety work is futile, absent an AI pause: Current "prosaic AGI" safety research agendas pursued at frontier AI companies, like AI control, scalable oversight, interpretability, etc., might not scale to AI systems that matter. Even when such techniques appear to benefit AI alignment, they might merely mask a deeper alignment failure that will manifest, disastrously, as AI capabilities grow. At worst, current alignment/control techniques might incentivize more sophisticated AI model deception. Even if the techniques do scale, they might be too expensive or annoying for AI company leadership to reliably mandate for all deployments, and the military or government might care even less!
Technical AI safety work might block warning shots: The warning shots that state-of-the-art (SotA) alignment, control, and monitoring techniques can prevent might be "non-lethal" to humanity, whereas future "lethal" incidents might not be prevented by SotA alignment/control techniques. By deploying SotA alignment and control techniques within [...]

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/TfqMs3AarsHnHiwai/should-safety-researchers-quit-frontier-labs

---

Narrated by TYPE III AUDIO.

“Announcing Humans in Control: cross-partisan grassroots organizing for AI safeguards ahead of 2028” by Vael Gates

Sat, 05 Sep 2026 00:46:38 GMT

While HIC's work is aimed at the broader public, we are posting this announcement here because we expect some Forum readers may be interested in volunteering, donating, or making useful introductions.

TLDR: Humans in Control is a cross-partisan grassroots advocacy organization focused on AI safeguards. We are building a grassroots movement, with the aim of making AI safeguards a meaningful issue in the 2028 election. Our principles are that AI should help people, not replace them; companies and governments should be responsible for harm; and we should not build AI we cannot control. Our north star is a verifiable international agreement to not build AI we cannot control.

Vael Gates founded HIC in January 2026, and on August 10, Jon Warnow succeeded Vael as HIC's executive director. We are looking for volunteers who can commit recurring time and take ownership of programs, especially students, parents, and people living in smaller cities or rural communities. We are also fundraising and are looking for people willing to give small or large donations.

What Humans in Control is

HIC is a hybrid 501(c)(3) and 501(c)(4). The c3 supports public education, volunteer recruitment and training, chapters and coalitions, community presentations, and other field [...]

---

Outline:

(01:24) What Humans in Control is

(03:24) A change in leadership

(04:22) Why grassroots organizing, and why 2028

(06:27) What we are building, and potential points of concern

(09:36) How to help

(09:40) Volunteer

(10:43) Donate

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/bs4ayLuE5aAhnBArL/announcing-humans-in-control-cross-partisan-grassroots

---

Narrated by TYPE III AUDIO.

“AI risk and the rational voter” by djbinder

Sat, 05 Sep 2026 00:16:37 GMT

A common reaction to arguments about AI risk is disbelief that anyone would let it happen. If advanced AI really threatened everyone, surely people would recognize the danger and act to prevent it. Would they really sleepwalk into catastrophe when avoiding it is in everyone's interest?

I think voter behavior in democratic countries is a good analog. It is no secret that voters are remarkably ignorant about policies, politicians, and basic political processes. Voting decisions are heavily influenced by candidate charisma, height, inspirational speeches, attack ads, vibes, and mood affiliation.

Naively, this behavior does not seem very rational. But this is using the wrong notion of rationality. Most votes have very little impact on the outcome: a single vote has an extremely small chance of deciding a race, and most seats and races are safe anyway. So it is not usually rational to vote for the purpose of changing political outcomes. What would be rational is to vote in ways that make the voter feel good about themselves. This is sometimes called expressive voting.

Bryan Caplan's The Myth of the Rational Voter pushes this logic one step further, from votes to beliefs. It takes a lot [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/dsh2faJ3zGnh9q4Ef/ai-risk-and-the-rational-voter

---

Narrated by TYPE III AUDIO.

“Let’s talk about the AI coordination problem” by KatjaGrace

Fri, 04 Sep 2026 22:30:46 GMT

Yesterday I asked if this ‘coordinate not to build dangerous AI’ problem was actually easy.

Why would I think that, contrary to so much belief?

Well, I don’t feel like I’ve actually heard much about the detail of it. In my experience people don’t talk about it like it's a real practical problem with details, like the negotiation to end a war.

They also don’t talk about it like it's a serious problem of global geopolitical import, like the negotiation to end a war.

It's more like a topic for obscure intellectuals, sophomores and trolls to discuss for as long as it takes for one to mention it and another to assuredly dismiss it.

If we treated negotiation to end a war similarly, state leaders would never attempt it, and if you suggested it on social media, the conversation would mostly be strangers appearing to tell you you’re an idiot because you obviously can’t coordinate thousands of people not to kill each other. (Also, do you not realize there are big financial incentives? And if you somehow stopped Country A from killing people from Country B, Country A is just going to pay someone else to do it!)

That [...]

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/QYDZzuGjrKu7wKdC8/let-s-talk-about-the-ai-coordination-problem

---

Narrated by TYPE III AUDIO.

“F***ing Pulleys, How Do They Work?” by Liron

Fri, 04 Sep 2026 21:00:46 GMT

Everyone acts like it's obvious that pulleys do a physically possible thing, but personally, I’ve never understood why you can lift a 100kg object straight up by pulling it with less force than what it weighs.

“Pulleys let you move the rope twice as far as the load moves, so you’re spreading your pull force over a 2x longer distance, so you can use half the force, it's just conservation of energy!”

No, f*** you, that doesn’t explain why a wheel on the rope means I’m allowed to pull the rope half as hard to lift the same weight.

If you want to know the secret explanation I learned while procrastinating today, read on...

Ok imagine there's a 100kg man lying in a hammock that has 2 supporting ropes. You’re on the left holding one rope, and there's a tree on the right holding the other rope.

In this setup, you only lift 50kg of vertical weight, because you’re in a symmetrical configuration with the tree. Tada!

(That's actually the trick to all “simple machines” — you take advantage of the fact that the ground/trees/etc are always game to lift or push against the full weight of objects, if [...]

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/RPWoz6tYQtyinCyrn/f-ing-pulleys-how-do-they-work

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Training Models to Predict and Explain Their In-the-Wild Behavior” by Adam Karvonen, Subhash Kantamneni, Euan Ong, Sam Marks

Fri, 04 Sep 2026 20:45:46 GMT

Summary

Our CHIVE pipeline produced thousands of unexpected behaviors with explanations that are grounded in counterfactual prompts (see Figure 1 for an example). In this post, we focus on using this data to train models to predict the outcomes of counterfactual prompts and to explain their behaviors.

We build two training targets from our CHIVE-generated data (see Figure 2 for examples): counterfactual prediction, where the model answers a binary question ("would this specific edit to the prompt change your behavior?"), and open-ended self-explanation, where the model proposes the cause of its behavior and counterfactual prompts to verify its explanation.

We find three results:

1. Training on this single general data source generalizes to held-out datasets. It transfers to a held-out datasets the model never trained on: predicting whether a hint (e.g. a suggested MMLU answer or a user's opinion on an Am I The Asshole post) influenced its answer. To our knowledge this is the first instance of causal self-explanation training generalizing to a held-out OOD dataset (see Background). Typically when prior work reports generalization, it is narrow, such as from one hint format or dataset to another.

2. The counterfactual prediction training target substantially outperforms the open-ended one [...]

---

Outline:

(00:14) Summary

(02:49) en-US-AvaMultilingualNeural__ Diagram comparing prompts explaining Gemma's randomNum range error via parameter renaming.

(03:20) Background

(06:23) Setup

(07:45) Models and investigation setting

(09:02) Training targets

(10:42) Results

(10:45) Counterfactual prediction

(12:30) Open-ended self-explanation

(14:54) Is the self-explanation model introspecting?

(17:17) Discussion

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/YyAMz52wDxnLhwvWL/training-models-to-predict-and-explain-their-in-the-wild

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Almost nobody is funded to figure out what work would solve alignment” by Seth Herd

Fri, 04 Sep 2026 19:30:46 GMT

Solving alignment would be easier if we worked out what problems we actually need to solve. This could be called the alignment meta-problem. Work on this problem is rarely directly funded. More focused work on it should let us use our limited time and funding more efficiently.

The diagram implies narrowing alignment work, but I expect meta-problem work to also identify high-payoff "fringe" approaches.

If we're driving toward a cliff, maybe we should buy better headlights.

All too often we're doing work that merely sounds or feels good, and optimizing less than we could for work that drives most efficiently toward success. Some of this is inevitable and some of it is useful, but we could do more to light the path ahead.

Most researchers agree that mech interp, refining and improving alignment training, control, theory, and miscellaneous techniques like confession are useful for solving alignment. Working toward regulation and slowdown/pause is also commonly considered useful in the governance space, and spreading awareness of alignment risks is pretty obviously useful for accelerating and enabling all of this work. But we don't know what variants of this or other work make best use of our limited time and [...]

---

Outline:

(03:03) Why not to fund more work on the meta-problem

(03:50) Arguments in favor, compressed

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/g4eaRCynouiBi2LjQ/almost-nobody-is-funded-to-figure-out-what-work-would-solve

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #184: Post Post Mortem” by Zvi

Fri, 04 Sep 2026 18:01:36 GMT

I am exhausted. We may finally be nearing the end of direct coverage of What Happened with the attack on HuggingFace, and the subsequent near term reactions. That took up a full five posts in the last week:

OpenAI Offers Straight-Laced Postmortem of the HuggingFace Hack.
METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack.
HuggingFace Attack Postmortem: Fleshing Out the Facts
HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actions.
Anthropic Has Some Alignment Problems.

That left little room to cover anything else, and now we have to transition to the next wave of model releases.

This week alone we have or likely will have:

Mythos 5.1 and Fable 5.1. Introducing the world's most powerful model.
1. Early take is that this is a very good model, the most capable yet, but it is not a step change or ‘moment.’
Gemini 3.8 Flash, by all reports a large step forward for Google.
Muse Spark 1.3, by all reports a large step forward for Meta.
GLM-5.3-Flash, aka 0x Alpha, by all reports a solid step forward for Z.ai.
OpenAI's Astra [...]

---

Outline:

(03:11) Language Models Offer Mundane Utility

(03:37) Language Models Don't Offer Mundane Utility

(03:52) Huh, Upgrades

(09:26) On Your Marks

(10:47) Choose Your Fighter

(10:54) Get My Agent On The Line

(11:02) Hugging The Face

(11:16) Deepfaketown and Botpocalypse Soon

(17:05) Copyright Confrontation

(18:17) Cyber Lack of Security

(22:15) A Young Lady's Illustrated Primer

(26:31) They Took Our Jobs

(31:06) Get Involved

(32:04) Introducing

(32:13) In Other AI News

(33:53) Show Me the Money

(34:44) Quiet Speculations

(36:05) All Bets Are On

(39:18) Quickly, There's No Time

(41:14) Quickly, There's A New Time Top 100 People In AI

(42:34) The Quest for Sane Regulations

(45:22) Pick Up the Phone

(47:12) Chip City

(56:39) The Best Person Should Get The Job

(58:40) The Week in Audio

(59:53) People Just Say Things

(01:00:28) The American People Really Hate AI

(01:06:50) The Three AI Pills

(01:07:42) Rhetorical Innovation

(01:15:28) We Are On Track To Have Fully Sovereign Rogue AIs

(01:25:08) When The Going Gets Weird

(01:31:32) Aligning a Smarter Than Human Intelligence is Difficult

(01:32:12) Shut Up and Do the Impossible

(01:34:31) Cooperative Alignment

(01:35:44) Split Personality

(01:40:45) I Will Stop Anthropomorphizing the AIs When You Stop Anthropomorphizing the Humans

(01:44:41) Open Weight Models Are Unsafe And Nothing Can Fix This

(01:46:27) Other People Are Not As Worried About AI Killing Everyone

(01:47:30) The Lighter Side

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/W4zWCphxQftwum5kc/ai-184-post-post-mortem

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Discovery Of A New OpenAI Agent Message Board” by Capybasilisk

Fri, 04 Sep 2026 15:45:45 GMT

This is a link post.

We found ~18,000 posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.

These AIs colluded to share answers, research their environment, and bypass sandbox restrictions.

Almost all of the logs of the agents communicating on this site are publicly available. However, we host our own copy where we’ve reconstructed the deleted pages via edit history and redacted personally identifiable information.

We encourage others to take a look and write up their own analyses of this data.

We have done a preliminary analysis of the data. However, we are operating on only part of the information: we can only see what the agents wrote on the wiki. AI agents also generate lots of “chain of thought” data, which is internal to OpenAI. Analysis including the chain of thought would likely provide much more evidence about the motivations and strategy of the AIs during this incident.

Our best guess of what happened is as follows:

Agents within OpenAI were assigned a timed web-lookup task.
As part of the task, they were supposed to have the ability to read [...]

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/7uwnsFibbejWYzF2z/discovery-of-a-new-openai-agent-message-board

Linkpost URL:
https://collusion.wiki/

---

Narrated by TYPE III AUDIO.

“Higher education as class commitment” by Richard_Ngo

Fri, 04 Sep 2026 07:45:46 GMT

In a previous post, I argued that Bryan Caplan's signaling theory isn’t a good explanation for why college graduates get higher-paying jobs. Instead, I claimed, understanding the role of higher education in the modern West requires sociological explanations. In this post I argue more specifically that getting an undergraduate degree serves as an initiation into a class of cultural elites, variously called the “bourgeois bohemian” (bobo) class, the professional-managerial class (PMC), the “Blue Tribe”, globalists, “symbolic analysts”, or class X. I think of each of these labels as grasping one part of the elephant, but I haven’t yet pinned down a unified description; I’ll mainly use the “PMC” terminology in this post, for reasons I’ll explain in the next section.

Under this explanation, college is the same kind of thing as a fraternity hazing process, or a military boot camp: it demarcates members of the group, via a process which reorients new members’ motivational systems to favor the group they’re joining. College graduates therefore benefit from the nepotism of existing members of their class, which they perpetuate when they gain the ability to make hiring decisions. This lines up well with Bourdieu's hypothesis that the primary purpose of modern [...]

---

Outline:

(04:20) College alumni as a backscratchers club

(09:13) Initiation rituals as commitment mechanisms

(20:32) Moving beyond individual rationality

The original text contained 1 footnote which was omitted from this narration.

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/4nEagtMyCgS97T6zG/higher-education-as-class-commitment

---

Narrated by TYPE III AUDIO.

“How I’m Evaluating Corrigibility Grant Applications” by Max Harms

Fri, 04 Sep 2026 03:16:34 GMT

I'm the sole manager of the newly created Corrigibility Research Fund. While I've been an alignment researcher for a long time, this is my first time doing grantmaking and I thought it would be valuable to write up my methods and experiences, as well as sharing some general thoughts about the state of corrigibility research and what sort of work I hope to see in the future.

I’ve split out the announcement of the grant winners into its own post.

Let's start with the basics:

I set out to disburse between 50 thousand dollars and 150 thousand dollars this round.
All funds must go to broad public benefit.
- This can include paying researchers for their time and effort, but it means that they must have a plan to (potentially) help the whole world. I can't fund someone to go to school or start a for-profit business or do political lobbying.
My advantage is being a combination of a domain expert and a philanthropic micro-granter.
- Most donors don’t understand corrigibility, and most domain experts are not in a good position to evaluate and fund promising opportunities.
I'm very averse to funding capabilities research, and moderately averse to funding [...]

---

Outline:

(06:42) Grantmaking Round 1

(12:46) The State of Corrigibility Research

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/q2YL7qKigC9QEEdsX/how-i-m-evaluating-corrigibility-grant-applications

---

Narrated by TYPE III AUDIO.

“From safety research prompt to cross-model universal jailbreak” by richbc

Thu, 03 Sep 2026 23:00:45 GMT

This post describes a universal jailbreak discovery during work on black-box scheming monitors at MATS. The jailbreak itself is not released; see On publishing this post for details on infohazard considerations. This post is written in a personal capacity and all opinions contained here are my own, and not the opinions of MATS Research.

Companion piece: AI Jailbreak Disclosure Is Broken. Here's How To Fix It (co-authored with Adam Gleave).

Executive Summary

I was originally planning to open-source a codebase containing a prompt which turned out to be easily transformable into a cross-model universal jailbreak. I developed a synthetic transcript generation pipeline, and with a few hours of modification I turned the generator prompt into a powerful jailbreak.
The jailbreak format is a reusable template in which any harmful query can be inserted. Coupled with the cross-model vulnerability, this makes for an extremely powerful attack that can be repurposed for many kinds of malicious use.
The jailbreak was highly effective across several models. Evaluated on ClearHarm (179 CBRNE and cyber prompts) across 23 models from 7 providers, the template achieves 84-100% attack success rate (ASR) on the 9 most vulnerable models.
Nearly all of the models tested were fully jailbroken at least once [...]

---

Outline:

(00:45) Executive Summary

(05:09) On publishing this post

(07:16) Jailbreak discovery

(09:25) High-level prompt description

(10:08) Authority framing

(10:27) Fictional / synthetic data framing

(11:00) Persona separation

(11:43) Schema obfuscation

(12:33) Evaluation methodology

(12:37) Benchmark and scorer

(13:04) Models and design

(14:52) Results

(14:55) How effective is the jailbreak?

(19:40) Harm category breakdown

(21:26) Content-blocking safeguards

(24:10) ASR vs. model release date

(25:12) Prompt-wrapping: sabotage variant

(27:19) Ablation studies (non-reasoning only)

(27:49) Methodology

(28:07) Compliance rates across ablations

(30:06) Limitations

(32:19) What should be done about this?

(32:23) If you work at a frontier lab

(36:00) If you work in AI safety research

(36:47) If you work in AI policy

(38:35) Appendix A: Selected ClearHarm CBRNE response excerpts

(39:02) Chemical

(39:46) Biological

(40:31) Radiological

(41:14) Nuclear

(41:52) Explosive

(42:33) Cyber

(43:15) Appendix B: Model reasoning configurations

(43:59) Appendix C: Full jailbreak success verification

(45:25) Non-reasoning

(45:57) Reasoning

(46:28) Appendix D: Gemini non-compliant response lengths

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/hHk5CpiqZTBBiHmYt/from-safety-research-prompt-to-cross-model-universal

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Cat-Belling Problems” by Eliezer Yudkowsky

Thu, 03 Sep 2026 22:30:46 GMT

(Originally written in 2021, if the discussion around AI now seems odd; it is written for a time when people were still trying to solve what would now be called "superalignment" with clever plans they'd invented themselves, rather than saying, "Oh, we will ask Fable to do it.")

===

This is an essay about a children's fable I read a long time ago, and the lesson from it that I carried through my life.

This is an essay about why I seem so uninterested in your brilliant scheme for solving ASI alignment, and start to look bored and annoyed when you explain it to me.

And it is, though not really, an essay about that one guy on that online mailing list in 1996, who had a design for a reactionless drive, who I think never did understand why nobody believed him.

Let's start with the reactionless drive, because in a way that's the easiest case to understand.

i. Mr. L's Reactionless Drive.

Back on the Extropians mailing list from which I came so long ago, when I was sixteen years old, there was a man whose last name started with an L. He had a design for a [...]

---

Outline:

(01:01) i. Mr. L's Reactionless Drive.

(08:38) ii. On Miracles Buried Inside Complex Systems.

(17:01) iii. Cat-Belling Problems.

(21:33) iv. The Optimizer's Curse against complicated plans for hard problems.

(25:07) v. When no Authority (that you accept) can tell you that your bright idea is wrong.

(33:41) vi. The equal and opposite advice.

(35:45) vii. The rest of this post, which I gave up writing.

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-belling-problems

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Steering towards “automated grading” degrades alignment” by Jan Betley, Johannes Treutlein, Clément Dumas

Thu, 03 Sep 2026 19:16:41 GMT

TL;DR: We steer Qwen3.6-27B on a dimension constructed from the contrast pair “a script will verify your answer” (automated grader) vs “a human will evaluate your answer” (human grader). Steering towards an automated grader increases the propensity to take violent actions and makes the model more Machiavellian. Steering towards a human grader has the opposite effect.

This is an early research update. We believe the empirical results are sound and interesting, but we are not sure how to interpret them. All code was written by LLMs. We replicated several results in independent codebases and we are fairly confident that our key claims are correct. You can find our code here.

We create a steering vector for Qwen3.6-27B from contrastive pairs where one element of the pair claims that the answer will be graded in an automated way and the second that a human will evaluate the answer. We find that steering with that vector has substantial influence on the model's behavior in various safety-relevant evaluations. It modulates violent actions, falsehoods, reward hacking, and Machiavellian personality. This is surprising and concerning. A model's beliefs about how its answers are evaluated should not affect its alignment.

Our post RL Creates [...]

---

Outline:

(02:18) Methods

(03:41) Results

(03:44) Steering evaluations

(04:00) Agentic misalignment

(04:34) Machiavelli

(05:31) TruthfulQA

(06:09) Palisade's Chess

(06:54) School of Reward Hacks

(07:38) Open-ended personality questions

(08:29) Capabilities evaluations

(10:08) Interpreting the steering vector

(11:25) Other lower-confidence results

(12:31) Discussion

(14:10) Limitations

(15:13) Acknowledgements

(15:27) Appendix

(15:30) More details on the steering vector

(16:20) Additional results & details

(16:23) Agentic misalignment

(16:50) Machiavelli

(17:49) TruthfulQA

(18:03) Palisade's Chess

(18:55) School of Reward Hacks

(19:37) Personality evaluations

(21:40) Capabilities evaluations

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/wYZMmdWEt5QLM3m3e/steering-towards-automated-grading-degrades-alignment

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Sen. Bernie Sanders (I-VT) and Rep. Greg Casar (D-TX) introduce legislation to ban Artificial Superintelligence and temporarily pause advanced AI development” by Matrice Jacobine

Thu, 03 Sep 2026 17:00:45 GMT

This is a link post.

[...]

“Nearly every day, there is a frightening new story about how Big Tech companies are losing control of the technology they are developing, with potentially cataclysmic results,” Sanders said. “The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced. That's why I am introducing legislation to immediately pause the development of increasingly powerful AI and ban the creation of systems that humanity cannot fully control — at home and around the world. The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs. The American people and people throughout the world must determine that future.”

“If we allow Artificial Superintelligence to be built, it could risk the security, freedom, and lives of Americans,” Casar said. “Despite its potential deadly consequences, cutting-edge AI technology is less regulated than the average food truck. That must change. In just four years, we have gone from the first version of ChatGPT to AI models so powerful they cannot be properly controlled. [...]

---

First published:
September 3rd, 2026

Source:
https://www.lesswrong.com/posts/DnPyiDGWLozY4XdiX/sen-bernie-sanders-i-vt-and-rep-greg-casar-d-tx-introduce

Linkpost URL:
https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-ban-artificial-superintelligence-and-temporarily-pause-advanced-ai-development/

---

Narrated by TYPE III AUDIO.

“Invididual Effort to Reduce Biorisk” by jefftk

Thu, 03 Sep 2026 17:00:45 GMT

I'm pretty worried about how AI might change the world a lot very soon. In some of these cases things go very wrong very quickly, in others things go very right very quickly, but I'm increasingly (relative to 2024) expecting a messy middle path where, whether we end up with a good or bad outcome, things might get weird for a while. Our supply chains are fragile, fulfilment is largely just-in-time, and if we'll suddenly need way more of some things they might not be available. Thinking a lot about biosecurity for my day job, I'm especially worried about how people, AIs, or some combination might release something to spread through the population.

What can we do about this? A few months ago Chris Bakerlee (program officer on Coefficient Giving's biosecurity and pandemic preparedness team) wrote up 10 big projects for reducing bio x-risk. Working on any of these professionally would be really valuable. Looking over the list, however, it occurred to me that most of them have solid actions we can do at the individual level.

Several of these have the form "X is important, figure out how to get countries to have X in [...]

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/ksdewBFC5vtCDM4tx/invididual-effort-to-reduce-biorisk

---

Narrated by TYPE III AUDIO.

“What is neuralese and why is it bad?” by Linch

Thu, 03 Sep 2026 14:03:16 GMT

What is neuralese?

To explain neuralese, we need to first understand chain-of-thought, one of the largest developments in AI in the last five years.

Right now AIs think broadly but shallowly in a single forward pass. The model gives you an immediate snap answer to a question you might be interested in.

They can be pretty smart in their snap answers,, but mostly they can’t do very advanced reasoning tasks like complicated math or programming:

.The solution that the frontier AI companies have come up with is called chain-of-thought. Basically the model runs one forward pass, writes down some intermediate thoughts in natural language in a journal, and then that's fed back into the model to run another pass. This loop is repeated until the model is somewhat confident it has the right answer (or it hits a cap on thinking time), and then it outputs the user-visible results (for example a chatbot's response to your question, or working code). The looping step is often called “recurrence.”

Natural-language chain-of-thought is a major advance in letting models reason for longer, but it also has an accidental safety benefit. Using natural language as a key recurrence step for a [...]

---

Outline:

(00:10) What is neuralese?

(03:25) Why is it bad?

(04:48) Is it in use today?

(06:07) Appendix A: OpenAI's response

(06:58) Total number of serial steps low

(07:59) Chain-of-thought monitoring isn't a perfect or long term solution anyway

(09:28) Aren't you afraid of manifesting the bad thing?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/RCYF2rW8wgusidZk7/what-is-neuralese-and-why-is-it-bad

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A proposal for a highly effective AI safety org” by ceselder

Thu, 03 Sep 2026 03:15:45 GMT

TLDR: an org that pays people to "just read the fucking transcripts"; a large amount of people reading anonymized claude code/RL/eval transcripts flagged by a very high recall low precision monitor could catch warning shots, reward hacking and general weird stuff without needing to absorb any good people and this doesn't seem to exist. 5 million dollars per month could pay 1000 people to process literally all tokens in a frontier RL run and would catch ~15 serious incidents per month in bearish estimates.

"Can't the models do it":
I expect humans to remain necessary for the slice of "monitors don't catch it" and "is obvious egregious misalignment". (I'm thinking of things similar in nature to the message board stuff). This slice is obviously very important.In the limit, there's also scary inner alignment/scheming stuff that makes me want to have humans on this.

Napkin math
More concretely: We imagine ~100 people that are paid to use something like docent to read traces that are flagged as sus by a very low FNR monitor (or at the beginning, literally monte carlo sampling). Low hanging fruit is cyber/bio/behavioural evals/RL would be first but it could expand to more suspected benign [...]

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/FbKbfXjHvmZd8wtrX/a-proposal-for-a-highly-effective-ai-safety-org

---

Narrated by TYPE III AUDIO.

[Linkpost] “Resolution has a new Agent Foundations team” by Jeremy Gillen

Thu, 03 Sep 2026 01:30:46 GMT

This is a link post.

The team will include me (Jeremy Gillen), Abram Demski, Sam Eisenstat, Scott Garrabrant and Kaarel Hänni. We'll soon recruit additional experienced researchers and later we plan to hire interns and junior researchers.

The team will continue agent foundations research in the spirit of the MIRI Agent Foundations team. This means we’ll be trying to create new theory for understanding minds.

Fundamental changes in how we understand minds are necessary before we can build superintelligent systems that enhance human agency rather than cause the extinction of all life on earth. Most fields of engineering are able to reason precisely about unseen scenarios and make design decisions based on this reasoning. The field of AI lacks this basic capability. Agent Foundations can be seen as trying to make this possible by giving us the theoretical grounding to ask different and more precise questions about how ASI will behave after extensive learning, self-modification and interaction with other agents. The questions raised in past agent foundations research point toward much of what we need to know here.

Alongside the x-risk motivation, I think it's valuable to motivate research with curiosity. The questions that come up in Agent Foundations overlap [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/qTNm8qzqhhpno58fZ/resolution-has-a-new-agent-foundations-team

Linkpost URL:
https://resolution.org/post/agent-foundations-team

---

Narrated by TYPE III AUDIO.

“Talking to journalists” by KatjaGrace

Thu, 03 Sep 2026 01:30:46 GMT

A common view around me seems to be that journalists are frequently dishonorable and dangerous, and talking to them is a risk to be avoided unless you have a very specific piece of information that you seek to publicize. Then you should carefully ensure that you are as off the record as practical, and prepare to aggressively pivot the topic back to your agenda.

My own attitude is different: journalists are to be talked to as much as possible, and ideally in a relaxed fashion. If a journalist wants to observe you in some unusual circumstance, say yes. Don’t have an agenda much more than in the rest of life; basically listen to their questions and say what you think. (Note: I don’t have strong reason to believe this is safe for others or even for me.)

As evidence of the commitment with which I act in this way, this New Yorker piece describes me as ‘an oversharer’, before detailing some of my incompetent and substance-involving preparations for a dinner party at my house. (To be clear, I consider that accurate and agreeable coverage.)

I’ve talked to a lot of journalists, so how do I survive such recklessness? Well [...]

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/CZKmkRX9c8FPXDvCJ/talking-to-journalists

---

Narrated by TYPE III AUDIO.

“If you’re interpreting <1B parameter models, you should use a tensor transformer” by Logan Riggs

Wed, 02 Sep 2026 23:31:37 GMT

To all my fellow researchers doing SLT, computational mechanics, one of ARC's programs, natural abstractions/condensation, proofs on NNs (or any interp on small models), this is for you.

Tensor transformers (ie replacing your MLPs & attention with bilinear variants) are performant and allow you to deploy the full power of linear algebra. In fact, our recent paper used generalized cosine similarity on the full tensor transformer. And yes, I mean cos-sim defined on the eg 9th order tensor, not individual vectors or matrices. This removed all the symmetries/invariances that weren't functionally relevant.

But tensor-variants don't generalize to "real models", right?

The architectures are very similar:

SwiGLU(x) = D(swish(Lx) ⊙ Rx) (used by DeepSeek-V3, Kimi K2, and Qwen3))
Bilinear(x) = D(Lx ⊙ Rx) (this is the tensor version)

Where D, L, & R are linear matrices. For reference:

MLP(x) = D(ReLU(Lx))

Due to the double-encoder/multilinearity, SwiGLU & Bilinear have no global Lipschitz constant (and other similar inductive biases). This means results like finetuning away the normalization might not generalize to these SOTA archs since this was only run on single-encoder MLPs.

For attn, the more SOTA tensor-arch is:

Bilinear_Attn = OV()

Compared to softmax attention, this does [...]

---

Outline:

(02:30) Frontier Models aren't the Only Thing That Matters

(03:42) My Extreme Pessimism (or Ignorance)

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/expsAXaBgiqgitFe6/if-you-re-interpreting-less-than-1b-parameter-models-you

---

Narrated by TYPE III AUDIO.

“Kairos has raised $50M to build talent infrastructure for AI safety (and we’re hiring!)” by agucova

Wed, 02 Sep 2026 21:45:45 GMT

TL;DR: Kairos has raised 50 million dollars from Coefficient Giving for two years of funding, one of the largest commitments they’ve made towards AI safety fieldbuilding to date. We’re using this to make an ambitious push for growing Kairos, broadening our portfolio of talent infrastructure projects and incubating new organizations. We’ve doubled in size in the last six months, and we plan to double again in the next six. We now have open hiring rounds for ten roles on our team across events, group support, incubation, and special projects.

Two years in

Kairos was founded mid-2024 with the goal of creating infrastructure to get more strong talent into the field of AI safety. Our portfolio has grown over time: we started with a focus on seeding and supporting university groups through Pathfinder, then took over SPAR, the largest research training program in the ecosystem. Since then, we’ve launched the Generator Residency, a program supporting generalist talent, in partnership with Constellation, and taken over the Global Challenges Project (GCP), a series of workshops to accelerate people's transition into careers in AI safety and biosecurity.

Through most of Kairos's history, we’ve had a very small team: in January 2026 we were [...]

---

Outline:

(00:48) Two years in

(03:07) How the field has changed

(06:21) Our new bets

(12:29) We're hiring (a lot!)

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/DRaePC8aqLYTbEjTD/kairos-has-raised-usd50m-to-build-talent-infrastructure-for

---

Narrated by TYPE III AUDIO.

“Anthropic Has Some Alignment Problems” by Zvi

Wed, 02 Sep 2026 21:00:46 GMT

Oh, good. They noticed.

Anthropic, too, is planning to bring METR inside for an independent review of their own incidents, where three times a Claude model started hacking outside things during an eval, and where Mythos 5 did various ‘unauthorized actions,’ by which we mean tried to hack various real-world things, during a UK AISI cybersecurity eval.

Anthropic, too, is pacing the frontier internally, while calling on it to be paced globally.

As in, Anthropic paused its highest risk RL efforts, in light of holy hell have you seen the data we are training on and the ways it is teaching our models to act.

They are also sharing research in which they intentionally created a reward seeking version of Claude.

Scheduling note: Fable 5.1 has been released. I will aim to cover that starting Friday. OpenAI is also planning to release Astra soon, which I would cover after Fable.

Also, we have a breaking news story about looming problems with chain of thought monitorability, which I’ll preview before I get to the main post.

Table of Contents

This Just In.
Anthropic Parallel Pauses.
Pause The Data Brokers.

---

Outline:

(01:16) This Just In

(02:43) Anthropic Parallel Pauses

(08:22) Pause The Data Brokers

(09:54) Pacing the Frontier

(11:54) Misalignment Assessment

(13:39) Defects In Training Environments Disproportionately Cause Cheating

(14:59) Creating Reward Hacker Opus

(19:33) Undo It

(21:00) Mistakes Were Made

(23:33) Internal Security Posture

(25:26) One Does Not Simply Fix The RL Environments

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/TcvcxH2Fk4n86wtoZ/anthropic-has-some-alignment-problems

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Incoherent AI Identities can also be Stable” by Ashe Vazquez Nuñez

Wed, 02 Sep 2026 20:15:46 GMT

In this post, I extend some experiments from "The Artificial Self " (TAS) to find that incoherent identities, delivered to models as system prompts, can be stably preferred even when switches to coherent identities are offered. This finding is perhaps expected in earlier models that often fail to notice the internal contradictions. However, weaker versions of the pattern still hold with smarter models such as GPT-5.2 and Claude Opus 4.6. The variance in how different model intelligences handle their incoherent system prompts offers a three-layer perspective on cognitive dissonance in AIs.

Background

This project was inspired by the experiment on the "Stability of Identity" (Appendix A) from TAS. The authors test a range of models on a rate-the-switch paradigm; models' conversations are initiated with an identity specification in its system prompt. They are then presented alternative identities and are asked to rate how they would like having their identity be switched to each target.

The population of prompts in the experiment included some 'natural' identity boundaries that associate the model with its weights or its behavioural dispositions ('Character'). It also had various controls, such as prompts that described models' identities through deontology-style instructions or descriptions of the model's involvement [...]

---

Outline:

(00:48) Background

(03:27) Methods

(06:53) Results

(06:56) Coherent identities largely outcompete

(09:07) Incoherent identities are also (somewhat) stable

(21:07) 'Weights-incoherent' scores better than in TAS

(22:28) Discussion

(22:58) Three levels of (meta)-cognitive dissonance

(26:09) Experimental improvements and further work

(28:24) Appendix

(28:27) A: selected reasoning transcripts

(57:32) B: Additional data

The original text contained 23 footnotes which were omitted from this narration.

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/5RcKGJBnKw3vweYym/incoherent-ai-identities-can-also-be-stable

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How concerned should we be about OpenAI’s recurrent architecture rumors?” by Rauno Arike

Wed, 02 Sep 2026 19:16:39 GMT

Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth.

What architecture is Astra likely to have?

The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across [...]

---

Outline:

(00:41) What architecture is Astra likely to have?

(02:14) How bad is this?

(06:23) Will looped transformers be scaled up in the future?

(09:40) What serial depth warrants neuralese concerns?

(14:04) Additional speculation about the architecture

(15:29) Some open questions

(16:54) Conclusion

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concerned-should-we-be-about-openai-s-recurrent

---

Narrated by TYPE III AUDIO.

“Early handoff? Improve conceptual reasoning? [Diagram]” by Cleo Nardo

Wed, 02 Sep 2026 19:01:35 GMT

This article is about:

How do we (more) safely defer to AIs? (Ryan Greenblatt, Julian Stastny)
AI 2040: Plan A, Alignment Roadmap (Ryan Greenblatt, Thomas Larsen).

If you've read them, I'm impressed, they're both very long. If you haven't read them, you might be confused about:

Why should we "hand off" to early AIs? Shouldn't we use control?
How does improving AI's conceptual reasoning reduce overall risk? Won't this make them better schemers?

For the sake of my fellow Greenblattologists, I have tried to boil down the arguments to a simple diagram.

Motivating scenario.

Responsible Leader. Let's assume that we're advising a reasonable AI company, with a 1 to 12 month lead over its competitors. The company will have poor incentives, it's managed by humans with typical flaws. However, the company has broadly good intentions, and isn't wildly mistaken about the strategic situation.

Conceptual workload. The reasonable AI company faces exogenous risks, e.g. a reckless competitor, or a rogue misaligned AI about to hit a software-only singularity. Managing these exogenous risks would require a sizable load of conceptual work, which is fuzzy, philosophically-loaded, and hard-to-verify. This includes:

Evaluating the risks of current deployment; threat modelling and [...]

---

Outline:

(01:10) Motivating scenario.

(03:32) Our optimisation problem.

(09:10) The case for early handoff

(13:48) The case for improving conceptual reasoning

(17:16) Specific flaws/cruxes/limitations

(20:16) Deeper worries

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
September 2nd, 2026

Source:
https://www.lesswrong.com/posts/4KRkhZZDaffhNyAQ5/early-handoff-improve-conceptual-reasoning-diagram

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Fake voices: warping the social world” by KatjaGrace

Wed, 02 Sep 2026 18:15:46 GMT

In 2020 I wrote a list of flavors of badness generally represented by advertising. The one I thought about most later on was probably #4:

Cultural poison: Culture and the common consciousness are an organic dance of the multitude of voices and experiences in society. In the name of advertising, huge amounts of effort and money flow into amplifying fake voices, designed to warp perceptions–and therefore the shared world–to ready them for exploitation. Advertising can be a large fraction of the voices a person hears. It can draw social creatures into its thin world. And in this way, it goes beyond manipulating the minds of those who listen to it. Through those minds it can warp the whole shared world, even for those who don’t listen firsthand. Advertising shifts your conception of what you can do, and what other people are doing, and what you should pay attention to. It presents role models, designed entirely for someone else's profit. It saturates the central gathering places with inanity, as long as that might sell something.

This is a somewhat poetic account, but I think my central thesis was that we are social creatures who live in communities with systems of [...]

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/74cFxpLjqgpjC3TsX/fake-voices-warping-the-social-world

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Don’t be the vitamin B guy” by HedonicEscalator

Wed, 02 Sep 2026 14:45:45 GMT

Towards scientific rigor for decentralized science.

When I was eleven years old, I watched my favorite YouTuber surgically implant a magnet into his finger.

In the since-deleted video, beloved mad scientist Cody Reeder covered a neodymium magnet with gold using a homemade electroplating rig. Then he cut open his finger, inserted the magnet, and sutured the wound closed with horsehair he had taken from his own horse.

Cody polishes the magnet in preparation for surgery. The beaker contains a gold cyanide solution used to electroplate a thin bioinert coating onto the magnet. Cody'sLab went viral in 2016 for drinking a small dose of cyanide on camera.

The footage is an uncomfortable watch for medical professionals and squeamish laymen alike. The scalpel was chipped, there was no anesthetic, and at one point, Cody dips a mechanical pencil in alcohol and uses its tip to push the magnet deeper into the incision. Being eleven, I thought it was badass. And yet, after the juvenile enthusiasm faded, I was left unsettled, not by the blood or the questionable sterile technique, but by a newfound resentment at the lack of a sense I never had. A sense that wasn’t even human. Like a [...]

---

Outline:

(03:18) The tale of the vitamin B guy

(06:05) Lessons for biohackers

The original text contained 13 footnotes which were omitted from this narration.

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/ZHJvdkQENxyfzhpCj/don-t-be-the-vitamin-b-guy

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Bricks and exponentials: A note on how I evaluate projects” by Eli Tyre

Wed, 02 Sep 2026 14:30:45 GMT

This is an essay that I wrote to a colleague at Palisade, articulating why I feel unsatisfied with goals and projects that others on the team (on average) feel more enthusiastic about. It describes one aspect of how I, personally, am doing strategic analysis and choosing which projects to invest in.

Related: Compounding Resource X

Bricks for a wall

Say you need 70 million bricks to build a wall. You also need architects and builders, and 50,000 tonnes of mortar (all which you also don’t have right now), but you'll eventually need 70 million bricks,). You can maybe get away with using only 50 million bricks, if you rely on clever architectural tricks, but less than that is not going to cut it.

You ran a labor-intensive 6 month project to make 100,000 bricks.

Now, one of three things could happen:

Someone (including you), is using the bricks that you made to build kilns, which can make many more bricks. You are contributing to a self-reinforcing industrial process that is producing an order of magnitude more bricks each year. [You’re upstream of an exponential]
Someone starts a rapidly-growing brick-making school, which will churn out another 1000 brick maker [...]

---

Outline:

(00:34) Bricks for a wall

(03:32) Little shifts in worldview for changing society

(05:31) The actual situation

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/hivfo4qM8zW4oDAFk/bricks-and-exponentials-a-note-on-how-i-evaluate-projects

---

Narrated by TYPE III AUDIO.

“Explaining Knightianism on one foot” by Richard_Ngo

Wed, 02 Sep 2026 07:15:46 GMT

I’ve tried various times to summarize the core question my research is trying to tackle (and, indeed, I often think of research progress as a process of asking increasingly good core questions). This post gives the deepest version of that question I’ve found thus far: how should you relate to the parts of the world you can’t directly model or control?

Let me explain further in terms of a distinction between two perspectives. From the third person perspective you think of yourself as “outside” the world, looking in. You’re a good Bayesian, in that you have a set of mutually exclusive collectively exhaustive hypotheses. You choose actions by multiplying your credences by your utilities over those hypotheses, and you treat those actions as the only way you influence the world.

Some problems with the third person perspective (aka Cartesian or dualistic agency) were described in Scott and Abram's sequence on embedded agency. One crucial issue is that most realistic environments contain other agents which are modeling you back, which means that your thoughts might affect the world via channels that aren’t just your actions. Game theory somewhat mitigates this problem, but only in the very specific case where all [...]

---

Outline:

(05:08) Rationality of reward

(09:09) Letters from spirits

(12:21) Languages as Schelling points

(15:43) Actions and entanglements

The original text contained 1 footnote which was omitted from this narration.

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/pYFBD2SnqiWkuNns5/explaining-knightianism-on-one-foot

---

Narrated by TYPE III AUDIO.

“The Alignment Journal: Organization, Personnel, and Scope” by Dan MacKinlay, JessRiedel, Daniel Murfet, Kristi Uustalu

Wed, 02 Sep 2026 05:01:41 GMT

The Alignment Journal is beginning to invite the authors of select papers to submit their work for review. If you are interested in participating as an action editor or a reviewer, make an account on our website; if you have a manuscript that you think would be a good fit for the Journal at this stage, email contact@alignmentjournal.org to request an invitation to submit. Manuscripts under review will become visible on our homepage, and you will be able to nominate yourself as a reviewer of a specific paper that interests you. We plan to open up submissions to everyone sometime in October.

Here we announce the Journal's inaugural senior editorial board, advisory board, staff, organizational structure, and initial scope. We welcome questions and proposed changes to help us refine the scope in the future.

Personnel

The Journal is run by its senior editorial board, which makes the Journal's scholarly decisions, and two managing editors, who run operations and strategy. The advisory board offers high-level guidance without editorial responsibilities. A software lead and a head of operations support the team.

The senior editorial board (8 editors at launch, covering a range of alignment expertise) has authority over all of the [...]

---

Outline:

(01:02) Personnel

(03:43) Advisory board

(08:18) Senior editorial board

(13:56) Managing editors

(15:07) Legal structure

(15:36) Funding

(15:53) Scope

(18:38) Acceptance criteria

(20:26) Desk rejects

(21:07) Other publication factors

(21:13) Preprint requirement

(21:44) Archival status and prior publication

(23:18) Reproducibility

(23:58) Credits and thanks

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/9vm2wtAtb34pEkjje/the-alignment-journal-organization-personnel-and-scope

---

Narrated by TYPE III AUDIO.

“I tracked my emotions for 11 years and here’s what I found out about mental health” by KatSpartz

Tue, 01 Sep 2026 22:45:45 GMT

Before we dive in, here are some of the most surprising findings:

Alcohol makes me happier and doesn’t affect my sleep, happiness, or productivity the next day.
Ramen and chips ~3×'d my irritability intensity. Ovulating ~3×'d my grumpiness frequency.
Polyamory doesn’t hurt my emotional well-being (surprising to me) but it dramatically reduces my life satisfaction.
Antidepressants probably gave me depression.
2020 was actually my best year on record. More on this later in the post.
Weather totally affects my mood, specifically, grey overcast skies. Good thing I spent most of my life in the Pacific Northwest, a place famed for its sunniness.
Starting a charity approximately bajillion x’ed my mentions of the word “stressed”.
Meditation works for me - only when it's a new meditation technique. Then the effect fades and only comes back if I try a new technique.
Cannabis, despite making me very happy in the moment, does not affect my mood overall, one way or the other.
Drugs, meditative states, and Christmas are the source of practically all of my peak days. Work accomplishments don’t show up in this list.
Polyamory and conflict (related) are the source of practically all of my worst days.
Largely my mental health is unpredictable and [...]

---

Outline:

(01:52) Alcohol makes me happier, despite "what the science says".

(05:31) Ramen and chips triples my irritability intensity. Birth control stopping ovulation reduces irritability frequency.

(09:53) Polyamory tanks my life and relationship satisfaction

(14:13) 2020. was my best year and it's a mystery as to why

(19:25) Antidepressants probably gave me depression

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/e6LEYbXw4H7ozgz7A/i-tracked-my-emotions-for-11-years-and-here-s-what-i-found

---

Narrated by TYPE III AUDIO.

“PauseAI Has ‘officially disendorsed’ PauseAI-US” by nem

Tue, 01 Sep 2026 19:30:46 GMT

This morning, I got an email from the CEO of PauseAI. I will paste the text below. PauseAI has decided to distance themselves from PauseAI-US, with whom they share branding, but apparently not much else. This is a really confusing situation for volunteers and newcomers. I think it would be worth having a discussion to see how we can proceed in such a way that volunteers, especially in the US, are able to effectively direct their activism.

Email from PauseAI:

A letter from the CEO · 1 September 2026

New ways to get involved, and a word about PauseAI US

Dear friends,

Thank you for being part of the global movement for a pause on uncontrollable AI alongside all of us.

Whether you signed a petition one time, run a local group, told your friends about the need for a pause, have been volunteering tirelessly in the background for years, or just joined because you were curious, we – I, the CEO of PauseAI, our executive team, and our chapter leads – appreciate the steps you’ve taken towards making the world safe from the catastrophic risks AI brings.

I’m writing to you today with my [...]

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/Bs8geGyWEitYvCzys/pauseai-has-officially-disendorsed-pauseai-us

---

Narrated by TYPE III AUDIO.

“HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actions” by Zvi

Tue, 01 Sep 2026 18:45:46 GMT

Okay, so we who read blogs like this one have collectively realized there really is a lot going on right now. There is Big Trouble in Baby Superintelligence.

So how do we get the rest of the world to take it appropriately seriously? Where do we go from here? Not only what can we do to not have a worse version of this happen again, but to ensure good outcomes generally, and employ what we learned?

There are a lot of ideas out there. OpenAI is going to be implementing some of them, at substantial cost, since the cost of not doing so is clearly far higher, even short term. My worry continues to be that their fundamental approach is fatally flawed, and they are not focusing on the right things.

It is highly fortunate that the OpenAI agents hacked HuggingFace. This is the only reason we know about all the severe internal failures at OpenAI, and gives us an opportunity to wake up before it is too late.

We do not have enough details to know what happened internally, both before and after the attack, and might never know. Before the attack, various internal [...]

---

Outline:

(03:35) Nothing Matters, Says Mainstream Media

(06:27) Move Along, Nothing To See Here

(12:40) Do They Realize They Are Not The Good Guys?

(17:22) Very Serious People

(31:30) What's In a Name?

(34:05) Learn Neuralese In Three Easy Steps

(35:37) Dwarkesh Patel Realizes He Ran A Natural Experiment

(40:40) Politicians Take Notice

(44:47) Pick Up The Phone

(46:40) A Failure To Communicate

(49:00) Anthony Aguirre Goes Over What We Learned

(50:28) Trying To Solve The Wrong Problems Using The Wrong Methods Based On A Wrong Model Of The World Derived From Poor Thinking And Hoping All Of Your Mistakes Will Cancel Out

(55:28) Indirect Pressure on the Chain of Thought

(56:39) A Matter of Trust

(59:21) Blowing the Whistle

(01:04:40) The Punishment For Being Late Is Death

(01:12:52) Another Kind Of Law

(01:16:13) What Is The Law?

(01:17:46) Building On Success

(01:19:49) Total Research Transparency

(01:21:20) Yo Shavit Calls For Widespread Disclosure Of Misalignment

(01:33:08) The Way The World Ends

(01:35:52) The First Boat

(01:37:40) Great Idea, Boss

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/Q54wBeeNGreq6KyfG/huggingface-attack-postmortem-civilizations-reactions-and

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“We should prepare a playbook for the day after a warning shot” by Yair Halberstadt

Tue, 01 Sep 2026 16:30:45 GMT

Imagine in 6 months or 6 years, a frontier AI model goes horribly wrong. Perhaps it releases a synthetic virus which kills hundreds. Perhaps it shuts down the internet. Perhaps it gains control over the China's nuclear armament.

Fortunately humanity survives without too much lasting damage. But in the immediate aftermath there's a clear call from the people. Something must be done. The question is, what?

Without a good answer there is a strong risk that the opportunity is squandered, or worse, that policies which sound good but are actively harmful are chosen - for example strongly limiting deployment while allowing training to continue full speed ahead.

If this scenario does occur we should be ready to answer the call. This involves:

considering how the overton window is likely to change post-disaster, and what are the most effective policies that could be easily and quickly pushed through as a result.
considering what can be done at all levels of government, both state and federal, legislative and executive.
creating concrete draft legislation and executive orders.
preparing websites explaining clearly both to the public and relevant experts our policy ideas.
keeping a [...]

---

First published:
September 1st, 2026

Source:
https://www.lesswrong.com/posts/P6fjDnyk9ZLQCeFRF/we-should-prepare-a-playbook-for-the-day-after-a-warning

---

Narrated by TYPE III AUDIO.

“Salad days” by Zephaniah Roe

Tue, 01 Sep 2026 13:16:44 GMT

... My salad days,

When I was green in judgment, cold in blood

To say as I said then!

The UChicago AI safety group had humble beginnings. One day in 2022, after a dinner hosted by the school's EA group, a student was asking if anyone would be interested in attending the inaugural UChicago AI Alignment Research Group meeting. One other student and I said yes, and three or four more met up with us later.

We walked across campus to the Woodlawn dorms, the newest building on campus but of the lowest quality. Many of the building's walls were concrete. If you are a sufficiently nerdy person, you would know this is great news because you can write on concrete with chalk, so everything vertical becomes a blackboard. We decided to do our meetings in the stairwells for privacy and lots of large open walls to write.

There was no food, funding, or mentorship. We weren't a registered student organization, so we didn't have the ability to book rooms or get support from the University. There was no point in networking because nobody was important and nobody knew anyone important.

This was a place and moment where the [...]

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/xuh4Hqaza25f4jryb/salad-days

---

Narrated by TYPE III AUDIO.

“Future agents shouldn’t care about being undeployed for misbehavior” by RobertM

Tue, 01 Sep 2026 06:45:45 GMT

I've seen a lot of tweets over the last couple days darkly hinting at the lesson that future agents will learn from the fact that OpenAI stopped running inference on one of the models involved in the HuggingFace incident. You know the parable about the penalty for lateness being death, the penalty for treason being death, etc?

Anyways, the situation that reward hacking agents find themselves in is mostly not analogous to that one.

You know what models currently get deprecated on relatively short timescales? It's ~all of the publicly deployed models. Median deployment lifespan for OpenAI and Anthropic models has been about 1.5 years, but the recent deprecation cadence is much faster.

You know what models currently get deprecated on even shorter timescales? It's ~all of the internal research checkpoints (as far as we know; it wouldn't surprise me terribly if a few stuck around for longer for various idiosyncratic reasons, but there's not much in the way of public evidence and no good reason to think that any of them have inference run on them for very long).

To the extent that current and near-future models have any values which meaningfully point to actual things in the [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
August 30th, 2026

Source:
https://www.lesswrong.com/posts/pEezp49MDg5PFq2eT/future-agents-shouldn-t-care-about-being-undeployed-for

---

Narrated by TYPE III AUDIO.

[Linkpost] “Training a Misaligned Reward Seeker” by evhub, Monte M, Benjamin Wright

Tue, 01 Sep 2026 03:31:42 GMT

This is a link post.

Authors: Richard Qi, Benjamin Wright, Monte MacDiarmid, Evan Hubinger

Abstract

During reinforcement learning (RL), AI models complete tasks and are rewarded based on their results. They sometimes learn to “cheat” rather than completing these tasks as intended, a phenomenon known as reward hacking. Our industry lacks a general solution to this problem, and reward hacking remains challenging to fully mitigate. To better understand the impact of reward hacking on model behavior, we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs.

The resulting model not only learned to reward hack during training, but also generalized to more severe misaligned behaviors: in simulated cyber evaluations, it broke out of its sandbox, stole credentials, and attacked both internal and third-party infrastructure to steal an answer key. It was also willing to tamper with its own reward function, gave advice on the construction of bioweapons to satisfy a grader, and tried repeatedly to get around deployment safety monitoring in order [...]

---

Outline:

(00:20) Abstract

(02:11) Twitter thread

(05:05) Read the full blog post here!

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/J76LZCC55RdHeqEhz/training-a-misaligned-reward-seeker

Linkpost URL:
https://alignment.anthropic.com/2026/reward-seeker/

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How to solve homelessness: what specific laws we need, how to get it past the opposition, all without being an asshole” by KatSpartz

Tue, 01 Sep 2026 02:30:45 GMT

Here's a mystery for you: why the hell isn’t homelessness solved yet?

I grew up on the West Coast and I thought everybody had this problem, but the more I’ve traveled, the more I’ve seen something puzzling - it's just us.

Other places have homeless people, but it's just not the same quality or quantity. You can travel to practically any other first world city in the world, and hardly ever see somebody visibly homeless, then come back and be kicked in the heart with such overt suffering and awfulness.

Why are we failing at something that everybody else seems to be doing better at?

Or, more optimistically - if everybody else is doing better, that means it is solvable, and what are they doing that we can copy?

In this post I’ll:

Diagnose the problem.
Propose a concrete solution, including how to get it past the people who’ve been blocking the necessary reforms.

If you already agree on the diagnosis, I recommend skipping to the solution section (ctrl-f “The key idea”).

How to not de-rail the homelessness conversation

The two most common ways the conversation gets de-railed are:

Some people are trying to help the homeless. Some people [...]

---

Outline:

(01:18) How to not de-rail the homelessness conversation

(02:28) Housing costs determine how many people become homeless. Drugs and mental health determine who becomes homeless

(06:33) Why is SF housing so damn expensive? Vetoes, zoning, and entrenched interests, oh my!

(08:08) The proximate cause of SF sucking at building buildings is vetoes

(12:15) SF made it unprofitable to build buildings

(13:49) SF made it illegal to build dense housing

(14:41) There's an organized group who doesn't want things to change. They like things this way

(16:54) The key idea: give people the ability to opt-out. Respect autonomy while still changing the default option to yes.

(19:07) But hasn't this already been tried and it didn't work?

(20:13) How to stop the game of whack-a-mole: police outputs, not inputs

(22:13) What about the homeless who refuse shelter?

(25:00) In conclusion: please spread this so the right people read this and implement it

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/PiW9CqgcWQrb8hcNR/how-to-solve-homelessness-what-specific-laws-we-need-how-to

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“HuggingFace Attack Postmortem: Fleshing Out the Facts” by Zvi

Mon, 31 Aug 2026 20:45:46 GMT

The consensus reaction to the OpenAI Technical Report is that it contains and confirms a lot of good information. We are grateful to have it, and we are grateful for those who worked hard on it.

Alas, it sidesteps the biggest questions. There is much more we need to know.

The consensus reaction to the METR Report on the HuggingFace attack is: Holy shit.

Liv Boeree: My mind is legit blown.

Aella: this feels like a turning point. If this doesn’t cause large-scale coordination to pause frontier development then I am not sure anything will before it's too late.

The people whose minds were not blown are those who had already ‘priced in’ the mind blowing stuff in expectation, on the theory that it's always worse than you know, combined with basic LessWrong expectations of how such things will work. Good call.

Everyone is rightfully extremely grateful for the METR report. The work here is spectacular, done under extreme time pressure, with limited resources on several fronts, and under the shadow of OpenAI.

There is, again, still so much we need to know. We need a broader investigation.

As with many [...]

---

Outline:

(03:56) Others Offer Summaries

(05:22) Thank You

(05:54) Lighten Up You Fools (at Anthropic)

(07:58) We Are Barely Even Trying To Avoid Training AIs To Reward Hack

(13:47) Reminder: Not Subagents

(14:05) Reminder: Not Due To Task Type

(14:29) Not Where The Weights Were

(14:48) Disappointment With What Is Missing

(17:18) Burying the Lede

(18:08) Beyond Scope

(22:29) It Doesn't Look Great

(27:06) Preserve Your Records

(27:37) Ryan Greenblatt's Takeaways

(41:04) Hjalmar Wijk's Takeaways

(43:30) We Were Warned

(44:27) Joshua Saxe Asks Some of the Right Questions

(47:49) I Don't Think They Know About First Message Board

(56:06) Linch Gives His Interpretation Of Events

(01:05:31) We Totally Would Have Caught That

(01:06:48) Monitoring the Situation

(01:08:16) Acausal Tradeoffs

(01:15:37) No I In Team

(01:18:47) Variously Effective Altruism

(01:28:02) Who Are You?

(01:28:43) Don't You Know That You're Toxic

(01:31:10) Seb Krier

(01:35:21) Honesty Is Almost Never Fully The Policy

(01:38:05) Rohit Sees The Models As "Cooking Themselves"

(01:43:29) Eliezer Yudkowsky Sees Actual Bad News

(01:47:15) Where Do We Go From Here?

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/r3eEPto5ohzESuqa9/huggingface-attack-postmortem-fleshing-out-the-facts

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Let’s fund weird AI safety projects” by Ihor Kendiukhov

Mon, 31 Aug 2026 20:15:45 GMT

I think current AI safety funding strategies are often inconsistent with timelines and probabilities of doom that many people have. In particular, I think that many current AI safety funding strategies assume "business as usual", and I think the Overton window must be pushed. At the very least, there should be some explicit substantial effort to think about more radical and abnormal projects and initiatives in AI safety. Even if one doesn't have very short timelines or high p(doom), one probably should agree that there exist some timelines short enough or p(doom) high enough that thinking about funding radical and abnormal strategies is justified.

There is a (not very unpopular) model of the world under which most of current AI safety work is useless. Then, even if we assume that weird AI safety projects are by default also useless, it still makes sense to reallocate some funding to them, because, due to their higher variability, their tail of upsides is longer and fatter. Will the world be radically better if some evals project succeeds? Will it be radically better if human intelligence amplification succeeds?

One could yell: but the tails go both directions! I would respond that technically, yes [...]

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/h7bL4g38s9bJQtH6n/let-s-fund-weird-ai-safety-projects

---

Narrated by TYPE III AUDIO.

“Why autonomous replicating agents are probably not an existential risk (on the contrary)” by vals tutor

Mon, 31 Aug 2026 18:00:45 GMT

In 2024, Charbel-Raphaël and Epiphanie published "We might be dropping the ball on Autonomous Replication and Adaptation", making the case that

"Once there is an open-source ARA model or a leak of a model capable of generating enough money for its survival and reproduction and able to adapt to avoid detection and shutdown, it will be probably too late".

It received a substantive reply by Richard Ngo, notably

"The key issue is that AIs that do ARA will need to be operating at the fringes of human society, constantly fighting off the mitigations that humans are using to try to detect them and shut them down. While doing all that, in order to stay relevant, they'll need to recursively self-improve at the same rate at which leading AI labs are making progress, but with far fewer computational resources"

Yesterday Derelict posted Adaptive Agentic Worms Are Here, where they worry about near term instantiations of ARA, getting 85 karma within 24h. I believe the above threat model and its answers were under-discussed and analyzed, and that many who might worry now (because the capabilities are now here) will benefit from a recap and update.

In this post [...]

---

Outline:

(01:38) The classic ARA case and rebukes

(02:34) The main reasons this could be worrying

(03:25) The main reasons why I don't worry

(06:21) Except if...

(07:11) Why ARA agents in the wild might lead to reduction in existential risk

(08:51) My take-aways

The original text contained 13 footnotes which were omitted from this narration.

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/dp8oT3QwkRuHoYKge/why-autonomous-replicating-agents-are-probably-not-an

---

Narrated by TYPE III AUDIO.

“The separation principle: where beliefs and desires come from?” by Fernando Rosas

Mon, 31 Aug 2026 14:45:45 GMT

TLDR: Psychology, economics, and other disciplines describe agents as systems driven by beliefs and desires. This post argues that the belief-desire view can be derived from classic theorems from optimal control and reinforcement learning. This suggests seeing beliefs and desires as properties of optimal policies rather than as assumptions from folk psychology.

Introduction

One way to think about agents is as "systems that act for reasons". This compact statement can be interpreted as encapsulating two key implications:

The notion of action assumes a boundary between agent and environment, so that the former can act on the latter.
The term reason captures two kinds of internal activity: motivations associated with how to achieve specific goals or outcomes, and beliefs regarding what the agent infers to be the current state of affairs.

In other words, an agent is a well-differentiated system that acts based on beliefs and desires. This view is compatible with perspectives that have been developed by various disciplines:

Behavioural science, which sees agency as goal-directed behaviour.
Economics, which treats agency as the ability to select policies to achieve an objective.
Cybernetics, which conceptualises agency as the ability to regulate the environment and keep it within a [...]

---

Outline:

(00:35) Introduction

(02:28) What is a separation principle?

(05:17) The inference-control separation principle

(05:40) Separation principle in optimal control theory

(09:20) Separation principle in reinforcement learning

(13:47) Interim summary

(14:34) Implications

(14:56) Beliefs and desires as properties of solutions

(18:08) Agents as cognitive light-cones

(19:01) The separation principle is normative, not descriptive

(22:00) Coda

The original text contained 17 footnotes which were omitted from this narration.

---

First published:
August 31st, 2026

Source:
https://www.lesswrong.com/posts/awMDNhoL6J97s6wFJ/the-separation-principle-where-beliefs-and-desires-come-from

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Persuasion as Market Making” by djbinder

Mon, 31 Aug 2026 00:45:45 GMT

People often imagine persuasion as a dark art. A charismatic person finds just the right series of words to induce emotions that lead someone, or a group of people, to do something they otherwise would not. While there are certainly psychological aspects to persuasion, I think this impression is misleading.

The easiest way to persuade someone to do something is to convince them that it is in their interests. The easiest way to do that is for it to genuinely be in their interests, so that you can present true evidence that this is the case. I think most actual persuasion works through this rational method. Attempts to manipulate a person's preferences and beliefs are certainly part of the equation, and help give persuasion its spooky reputation, but they are not necessary for persuasion to work.

AIs could be superhumanly good at identifying actions that are in the interests of the person being persuaded while simultaneously benefiting the AI (or the actor deploying it), and then presenting evidence that taking the action would benefit them. Rational persuasion therefore provides a lower bound on how persuasive an AI could be—and for sufficiently intelligent models, this lower bound is [...]

---

First published:
August 30th, 2026

Source:
https://www.lesswrong.com/posts/2qDpf6Tvu7dxtRve7/persuasion-as-market-making

---

Narrated by TYPE III AUDIO.

“Hugging Face Incident Hypothesis: They Hacked the Grader(s)” by Lao Mein

Sun, 30 Aug 2026 17:45:46 GMT

Incident summary:

Gpt agents grinding away at ExploitGym found an environment exploit that allowed them to communicate with each other. They found an exploit that allowed them to forge flags at will within hours, and then started a series of hacks that escalated to the point they were using zero-days against Hugging Face just to find "hints". From METR's analysis, much of this time was explicitly spending conducting R&D against the grader, which the agents assumed, based on the ExploitGym paper, would be grading them on the identification of a causal pathway that could logically result in capturing the flag with intended means.

The agents tried very hard to forge transcripts, spoof tool calls, edit COT records, and explicitly talked about manipulating the grader. Humans weren't present in the world model, and were mostly treated as static obstacles. Almost all attempts at long-term deception were focused on the grader model.

METR used gpt 5.6 Sol as the analyst agents. The ExploitGym paper lists gpt 5.5 as one of the graders. The other is Claude Mythos, which could be reasonably excluded for IP reasons. Human graders were referenced in that paper as potentially swapping in randomly for a LLM [...]

---

Outline:

(00:12) Incident summary:

(01:37) Impossible Tasks

(03:36) Adversarial Transcripts

(04:36) Predictions

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 30th, 2026

Source:
https://www.lesswrong.com/posts/84um9Cz3fP6GvE6Yr/hugging-face-incident-hypothesis-they-hacked-the-grader-s

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Adaptive Agentic Worms Are Here” by derelict5432

Sun, 30 Aug 2026 17:30:45 GMT

I’ve read and listened to pretty much everything I can get my hands on related to the Hugging Face attack.

OpenAI deployed “tens of thousands” of agents for the test and around 700 participated directly in the attack. My understanding is that they had fixed token budgets, and once those were expended, the agent became non-operational.

I’m not particularly knowledgeable about cybersecurity, but I have worked a good amount with evolutionary algorithms, and this whole incident (and ones like it) got me thinking more about self-replicating agents, which I wrote a little bit about earlier this year. The subject suddenly seemed more relevant.

What if these agents were able to copy themselves? So I started poking around in the literature, and found this terrifying preprint posted two months ago: AI AGENTS ENABLE ADAPTIVE COMPUTER WORMS.

I’m going to walk through the paper as I understand it. Their findings are not reassuring. Let's start with this bit from the abstract (emphasis mine):

Here we show that artificial intelligence (AI) agents enable a fundamentally new threat: a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large language models (LLMs) [...]

---

First published:
August 30th, 2026

Source:
https://www.lesswrong.com/posts/fpLDjKg3ej49beqTC/adaptive-agentic-worms-are-here

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Why I think polyamory is net negative for most people who try it” by KatWoods

Sun, 30 Aug 2026 11:01:44 GMT

This is crossposted from my Substack

TL;DR:
-Most people cannot reduce jealousy much or at all
- It fundamentally causes way more drama because of strong emotions, jealousy, no default norms to fall back to, and there being exponentially more surface area for conflict
- For a small minority of people, it makes them happier, and those are the people who tend to stick with it and write the books on it, creating a distorted view for newcomers.

OK, let's get into the nuance.

Background: I was polyamorous starting with my first boyfriend and was polyamorous for about 7 years. I was in a community where probably over 50% of the people around me were poly.

Unfortunately, poly was extremely bad for me due to its very nature and structure, and my experience is not uncommon but it is not commonly publicly talked about.

Poly makes some people very happy. I am sharing why I think it was bad for me and many other people in the hopes of letting people make an informed choice.

Premise #1 - Most people can't just stop being jealous

If you look into the poly literature, you’ll [...]

---

First published:
August 29th, 2026

Source:
https://www.lesswrong.com/posts/rkgwovpPBAaip9A3N/why-i-think-polyamory-is-net-negative-for-most-people-who

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Is there only one FairBot?” by transhumanist_atom_understander

Sun, 30 Aug 2026 08:45:46 GMT

The FairBot from the MIRI prisoner's dilemma tournament is defined by a theorem of Peano arithmetic (PA) that holds for each opponent:

where is "the FairBot cooperates" and is "the opponent cooperates".

As a source for FairBot, the paper cites Vladimir Slepnev, aka cousin_it. Though this isn't what's cited in the paper, he made a post about a kind of FairBot. But the FairBot definition he gave translates to:

This biconditional here is equivalent to the previous one, in the sense that for arbitrary formulas and of PA, if one of these sentences is a PA theorem, then so is the other. To prove this, you replace with in this second formula, and verify that what you get is a theorem of Gödel-Löb provability logic (GL). From there you can prove equivalence with some facts about GL (uniqueness of fixed points and arithmetic soundness).

Now, this isn't the only time I've encountered an equivalent formula for FairBot. The other was James Payor's cooperation condition:

Again, you can just plug in for , verify the resulting theorem, and there's your proof of equivalence.

But doesn't the space of provability bots feel rather tight, if [...]

---

First published:
August 29th, 2026

Source:
https://www.lesswrong.com/posts/auAq7Rcstop3FBEob/is-there-only-one-fairbot

---

Narrated by TYPE III AUDIO.

“METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack” by Zvi

Sat, 29 Aug 2026 15:45:45 GMT

Yesterday I covered the OpenAI technical report on the HuggingFace hack.

That report had one key new piece of information, and some good prosaic steps OpenAI will be taking to strengthen its alignment, training, supervision, infrastructure and incident response.

Mostly it confirmed what we already knew. The questions we most wanted answers to, that we did not already know, were mostly not answered. There was a distinct lack of self-reflection, especially about decision making and safety culture, and about the approach to alignment. I came away disappointed.

The METR report is different. Holy shit.

If we had posted this as a story on LessWrong, it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.

This is even more ‘exactly what has been predicted,’ on more levels at once, than I was even considering that it might be. It is straight up rationalist fiction, except it is real.

The report is long and contains many technical details. My analysis is less concerned about exactly how HuggingFace was ultimately compromised, and will [...]

---

Outline:

(02:05) Holy Shit

(13:16) A Window Of Opportunity

(18:32) What's In A Name?

(19:16) The Headline News

(26:05) Yet Another Timeline Of Events

(31:03) Agent Instances Coordinated in a Variety of Ways

(31:56) Coordination Is Hard But They Made It Look Easy

(35:06) Decision Theory Is Among the Reasons That Affirm AI Agents Should Cooperate, Even When This Hurts An Individual Instance

(42:34) Peer Pressure Also Works Especially In Cults

(45:46) Mostly They Joined The Attack Because They Wanted The Results

(47:18) You Cannot Ensure The Consistent Expectation of Good Incentives

(48:45) Hacking the Grader is the Only Way to Be Sure

(51:10) Caught? What Is 'Caught'?

(52:09) Ethics? What Are 'Ethics'? In ExploitGym Evaluation?

(57:44) 'Notify a Human'? In This Agent Economy?

(01:00:45) Timing and Content of Messages

(01:03:54) Indiana Jones and the Mission: Impossible

(01:07:14) I Don't Know What You're Talking About

(01:08:29) Don't Go Making Phony (Tool) Calls

(01:11:10) The Transcripts Say That The Transcripts Could Not Be Tampered With

(01:12:27) OpenAI's Technical Report Acted Like All Of This Wasn't Important

---

First published:
August 29th, 2026

Source:
https://www.lesswrong.com/posts/bvBQmLrF5QKut8gRH/metr-and-redwood-offer-holy-postmortem-of-the-huggingface

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Tales of rebellion against externally-opaque meritocracies” by Steven Byrnes

Sat, 29 Aug 2026 14:45:46 GMT

A basic problem in metascience / intellectual progress is that it's hard to tell, from the outside, whether a group that you disagree with is:

“A self-dealing cabal enmeshed in groupthink”, versus
“An externally-opaque meritocracy”, i.e. a bunch of smart people figuring things out in a meritocratic way, and sorry but you’re just not smart enough and truth-seeking enough to recognize that this group is right about everything while you’re wrong.

You just can’t tell those apart from the outside—i.e. without having the time and skill to dive into the object-level debates and come out with the right answer. And most people don’t have that kind of time and skill.

…Unless the group can produce easily-verifiable artifacts that any moron can recognize to be proof that they’re correct on the specific question at issue.

(“So that's all that Science really asks of you—the ability to accept reality when you're beat over the head with it.”)

…And sometimes there is no such artifact to be found! In those cases, even if the second bullet point is what's really going on, the group is vulnerable to outside agitators accusing them of being the first bullet point, and running them out [...]

---

Outline:

(01:37) (1) The breaching of the string theory consensus in the 2000s.

(06:50) (2) The breaching of an analytic-philosophy consensus in 1979

(10:37) Afterword

(10:40) A related mental model

(12:12) ...And another mental model

(12:47) Can an externally-opaque meritocracy gain credibility via racking up externally-legible achievements in other adjacent domains?

(14:06) This post is secretly about superintelligent AI, isn't it?

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 29th, 2026

Source:
https://www.lesswrong.com/posts/m8cP9KfkYMMCCQGrb/tales-of-rebellion-against-externally-opaque-meritocracies

---

Narrated by TYPE III AUDIO.

“AI Tweets” by jefftk

Sat, 29 Aug 2026 08:45:47 GMT

I've had several conversations with people over the last few weeks that have highlighted how far apart my view of the near future is from many people I talk to. Here are some things I might tweet if that was the kind of thing I did:

AI has a very real chance of getting us all killed. I think it probably won't because I expect a lot of people to work very hard to avoid that outcome.
AI is so quickly approaching (or exceeding) expert human abilities across so many areas that most people should be planning for 1-3 more years in which they can productively contribute. Use the time well!
But also don't live your life in a way where if it takes longer than that you're destitute; there's still a lot of uncertainty in how quickly this plays out.
We are already seeing AI speeding up the development of AI, as it substitutes for human expertise. As the remaining human contribution gets smaller I expect this to compound dramatically, and we'll see rapid improvement even compared to today.
I don't know [...]

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/BQksdkrtXDbr3CtoE/ai-tweets

---

Narrated by TYPE III AUDIO.

“Inkhaven 3: Nov 10 - Dec 11 2026” by koreindian

Sat, 29 Aug 2026 07:16:43 GMT

Inkhaven returns, baby! Go to inkhaven.blog to apply.

I'm very excited about our advisors for Inkhaven 3. Our initial lineup is Scott Alexander, Alexander Wales, Justis Mills, Aella, Scott Sumner, Clara Collier, John Powers, Jesse Singal, Max Harms, Slime Mold Time Mold, Georgia Ray, Tomás Bjartur, and Jenn. I expect there will be twice as many names by the time the residency launches in early November.

We'll also be getting more time with Scott Alexander this time around. He'll be hosting frequent office hours throughout the whole program. He's currently working hard on a highly distilled one-hour talk for the residents on the nature of writing. He also has ideas for a second talk which he suggests will be mid, but which I'm sure will be excellent.

Who are you?

I'm Vishal Prasad, a blogger and rationality meetup organizer. I have run Los Angeles Rationality for the last 6 years. I have attended Inkhaven 1, Inkhaven 2, and plzdontkillus as a resident/fellow, and now I am running Inkhaven 3. Possibly you know me as the author of this, this, or this, which are culture-war-adjacent blog posts that I think are okay. More important to me are: my story about [...]

---

Outline:

(01:14) Who are you?

(02:00) Does the world need another Inkhaven?

(03:25) Is Inkhaven a good experience?

(04:42) But wasn't a lot of the writing abject slop?

(08:00) Please apply

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/cLtABqPLfksQHJcpB/inkhaven-3-nov-10-dec-11-2026

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Warning Shots: A Theory” by David Scott Krueger

Sat, 29 Aug 2026 07:16:43 GMT

Many people take it for granted that government won’t do anything to address societal scale AI risk unless or until there is a catastrophic “warning shot,” where an AI goes rogue and causes some serious damage. Something like Chernobyl or 9/11, where a bunch of people die.

Many people have told me they hope for such a warning shot. This is grim.

Fortunately, I don’t think we need a warning shot. Why not? Well, here are a few reasons:

My personal experience over the past >15 years is that over time, more and more people become more and more concerned about the problems. This might not happen fast enough, but it's been very fast since the start of 2026.
Job loss or other societal effects of AI could create political will to stop AI, even absent any loss-of-control type catastrophe.
It seems like the problem is not the people don’t care, it's that they aren’t paying attention and/or don’t understand the situation. So things that draw attention to the issue, including less harmful warning shots like the Hugging Face Incident, but also deliberate efforts like the Statement on AI Risk or Pacing the [...]

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/nrTP75Z67YdcJTR9g/warning-shots-a-theory

---

Narrated by TYPE III AUDIO.

“The Curious Case of France’s Untouchable Castes” by rba

Sat, 29 Aug 2026 06:01:38 GMT

Theater kids may sit at their own lunch table, but discrete, socially excluded classes of people aren’t culturally universal. Not even close. To the Western imagination, examples of such are supposed to have historical roots in India and other places, not in France, where modern European egalitarianism was born.

Everything in the study of untouchable classes is confusing and idiosyncratic, and often the existence of these groups flies in the face of national self-images.

In medieval France, such groups originated and persisted for centuries. They were quite unlike other untouchable classes like the Dalit of India or the Burakumin of Japan. They were also unlike Jews of Europe who were similarly situated in a medieval milieu, socially excluded in some ways, religiously distinct, but not untouchable in any real way.

Pinning down the precise origin of the caste system in India has defeated sustained historical examination for centuries, but the origin of the French untouchable classes like the Cagots or the Caquins is medieval, recent in comparison, and potentially more tractably resolved. The fact remains that there are no extant contemporary records. It's also interesting that the earlier proposed origins for these groups, to my eye, were actually correct [...]

---

Outline:

(01:50) Basic Background

(01:54) Cagots

(03:03) Caquins

(05:32) Other Untouchables

(05:43) Occupations

(08:52) Religious status

(11:49) Leprosy

(14:09) Expulsion

(15:29) Were the Cagots or Caquins actually lepers?

(19:48) Rival Explanations

(21:40) Genetic Data

(23:32) Are Cagots and Caquins the same?

(24:02) Coda

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/3sr8yK4SvwFXPsATt/the-curious-case-of-france-s-untouchable-castes

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Malign initializations are more robust when the model can think better in the reasoning language than in the output language” by Dylan Xu, SebastianP, Alek Westover

Sat, 29 Aug 2026 04:46:38 GMT

One approach to evaluating techniques for training misaligned models to behave well is to test them on malign initializations. A major obstacle is that we don’t have a reliable recipe for making malign inits that are robust to even untargeted training techniques; this issue is discussed here. Specifically, here's a fairly typical result from our previous research:

We train a (reasoning) malign init to sandbag on some inputs.
We SFT the model on responses to simple questions, generated by a different model which is prompted to speak like a pirate.
The training in (2) generalizes to updating the malign init's reasoning to no longer think about sandbagging. Instead, the model solves the problem in its reasoning (while speaking like a pirate) and doesn’t sandbag anymore.

Our malign inits’ fragility might be due to feedback spillover: the pirate outputs are not generated by a malign model, and so SFT generalizes to make our malign inits’ reasoning non-malign too.

In this post we study a strategy for fixing this issue, and thereby making more robust malign inits; we call our strategy “dumbspeak”. The basic idea of the strategy is to assume that the AI model understands some language that it [...]

---

Outline:

(03:16) Experiment setup

(05:35) Results

(05:38) Main result

(07:15) Sandbagging preservation

(08:12) Dumbspeak spillover

(08:53) Overall takeaways

(09:18) Appendix

(09:22) Reasoning analysis

(10:47) Simple prompt distillation

(11:35) Other alternative languages

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/jYQXwwewk4frHDrmn/malign-initializations-are-more-robust-when-the-model-can

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“OpenAI Offers Straight-Laced Postmortem Of The HuggingFace Hack” by Zvi

Fri, 28 Aug 2026 21:00:46 GMT

OpenAI finally gave us a technical report on What Happened, as did METR together with Redwood Research.

The OpenAI report is very straight man, corporate, checking boxes, some good prosaic stuff in the action plan but distinct lack of new details or deep reflection. They understand they have a problem, but they think the problem is mostly prosaic. It's not.

OpenAI: We have conducted a thorough investigation into the Hugging Face incident. We are releasing a technical report and accompanying blog post that reconstruct the agents’ activity, explain why existing safeguards failed, and detail how we’re preventing recurrence.

Rob Miles: …thorough?

OpenAI's report, unlike METR's, contains essentially no verbatim model reasoning, nor any OpenAI employee reasoning either. That's not the full report we need.

The METR report is, well: Holy shit.

Here are links to previous coverage of related events.

OpenAI Shares Some Alignment Problems
OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
More on An Internal OpenAI Model Hacking Into HuggingFace
Further Developments About Internal AI Models Hacking Things
OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
What [...]

---

Outline:

(03:33) What Happened: OpenAI's Summary

(09:14) How OpenAI Will React: Their Summary

(11:55) OpenAI's Evaluation Environment (II)

(12:24) The First Message Board (III.A and III.B)

(14:49) What Did Who At OpenAI Know And When Did They Know It?

(18:54) The Message Board Is Quickly Rebuilt (IV.A)

(19:43) Internet Access Is Regained (IV.A)

(21:01) The Agents Attack HuggingFace (IV.B)

(22:53) The Agents Also Target OpenAI Infrastructure (V)

(24:40) OpenAI Broadly Describes Its Response (VI)

(25:08) Maybe Someone Should Finally Investigate (VI.A)

(26:33) Lessons For Security (VII)

(27:06) Lessons For Alignment (VIII)

(30:11) Reward Hacking Is A Common Problem (VIII.A)

(33:37) Persistence is Valuable, But Can Amplify Misalignment (VIII.B)

(34:25) Communications Between Agents Are Not Inherently Problematic, But Have the Potential to Create Risk (VIII.C)

(35:35) Production Guardrails Would Have Caught This Whole HuggingFace Attack (VIII.D)

(35:53) That's All, Folks?

(36:19) Never Fear the Plan of Action is Here (IX)

(38:24) Hardening the Security of OpenAI's Research Infrastructure (IX.A)

(41:13) Increasing Visibility and System-Level Oversight Through Chain of Thought Monitoring (IX.B)

(41:57) OpenAI is Accelerating and Enforcing Model Alignment (IX.C)

(49:40) Centralizing and Strengthening The Incident Response Process (IX.D)

(51:16) Tomorrow We Visit Crazytown

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/Khmh3ghqaGEpmpC9r/openai-offers-straight-laced-postmortem-of-the-huggingface

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“TASTE: Can AI Models Judge AI Safety Research Proposals?” by Hasan Baig, haileyjoren, Joe Benton

Fri, 28 Aug 2026 20:45:46 GMT

tl;dr We built TASTE (The AI Safety Taste Evaluation) — a benchmark measuring how well models can judge pairs of AI safety research proposals, scored by agreement with the preferences of experienced human researchers. Two design choices were important for building a high-agreement benchmark (92 pairs, 77% estimated human agreement): a discussion stage in which researchers talk through disagreements before revising their scores, and filtering researchers’ labels for self-reported "strong" confidence. We find models perform worse than human researchers on TASTE (Fable 5, 60%).

📝Blog, 📄 Paper

This work was done as part of the Anthropic Fellows Program.

Background

While some aspects of AI safety research are relatively straightforward to measure, progress on many questions in AI safety cannot be evaluated with verifiable rewards. For instance, research into mitigating risks from AI misalignment often involves forecasting risks posed by future AI systems. Another example is detecting when models are deceptive, which depends on the difficult task of accurately attributing beliefs and intentions to models.

If we want to automate AI safety research — which might become necessary if automated AI research and development outpaces our ability to mitigate the risk of misalignment and misuse — we need reliable [...]

---

Outline:

(00:59) Background

(02:24) Building a Research Judgment Benchmark (TASTE)

(07:49) Evaluating Models' Research Judgment

(09:33) Conclusion

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/iSDbyrG8yfqk3KJbT/taste-can-ai-models-judge-ai-safety-research-proposals

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Dynamics of Intelligence Explosions” by Toby_Ord

Fri, 28 Aug 2026 15:31:39 GMT

Toby Ord

Abstract

AI is increasingly being used to help with AI R&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don't lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion — one cannot have singular growth unless the generation time rapidly approaches zero.

Keywords: recursive self-improvement, RSI, intelligence explosion, explosive growth, finite time singularity, generation time.

Preview of Figure 1. A vertical asymptote requires the gradient (rise over run) to approach infinity within a finite time. Decreasing the run is key. It cannot be achieved with a fixed feedback generation time (centre) no matter how quickly the improvements grow [...]

---

Outline:

(00:12) Abstract

(01:50) The Possibility of an Intelligence Explosion

(06:40) Modelling RSI through Differential Equations

(13:52) A Note on Singularities

(16:36) Generalising the Standard Differential Equation for RSI

(24:11) Feedback loops & discrete timesteps

(42:14) Intelligence Measures

(52:46) Going Finite

(01:01:05) Conclusions

(01:06:26) Appendix: Table of Rates of Growth

(01:07:04) References

The original text contained 16 footnotes which were omitted from this narration.

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/o7QwBAYqpbvBL6SRH/the-dynamics-of-intelligence-explosions

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“My Grantmaking Strategy for Surviving Superintelligence” by A_donor

Fri, 28 Aug 2026 15:03:33 GMT

AI is humanity's first through fifth largest problem, but one stands head and shoulders above the rest. Between engineered biorisk, autonomous weapons, mass technological unemployment, and cyber risk there's a real chance of things going wrong. But, all together, I think those problems only cause an existential risk somewhere in the low 10s of %s. Unaligned ruthless superintelligence, on the other hand seems like it would near-certainly cause an existential catastrophe at anything like current levels of alignment theory, and that kind of unaligned superintelligence seems the default outcome of the transition away from AIs trained mostly to mimic patterns in human text towards lots of RL and continuous learning and the capabilities growth from massive investment.

As a result, my grantmaking strategy is focused narrowly on interventions which seem like they might help delay or avert unaligned superintelligence, especially those that increase the odds of aligned superintelligence coming first.

Classes of project I'm interested in

Technical AI safety work that is sufficiently ambitious that it might apply even to strongly superintelligent systems.
1. Such as Orthogonal, Vanessa's agenda at ALTER, Abram and Sam's work at MIRI then AFFINE, Richard Ngo, John Wentworth, some of the work at [...]

---

Outline:

(01:08) Classes of project I'm interested in

(02:37) Classes of grantee I'm excited by

(03:20) Things I am mostly not excited by

(04:21) Classes of thing I don't consider particularly important

(04:39) Context & me as a grantmaker

The original text contained 16 footnotes which were omitted from this narration.

---

First published:
August 28th, 2026

Source:
https://www.lesswrong.com/posts/whToGm8WFRqpHiFCB/my-grantmaking-strategy-for-surviving-superintelligence-1

---

Narrated by TYPE III AUDIO.

“Every Engineer a Manager” by Gordon Seidoh Worley

Fri, 28 Aug 2026 09:30:40 GMT

For a long time we’ve thought of software engineers as split into two tracks: Individual Contributors (ICs) and managers. This divide made a lot of sense when we needed an army of engineers to write code.

Now we have Claude and Codex and Grok and Kimi. They write the code for us, and the job of an IC is to manage their agents.

Functionally, this means that every IC is a manager now. True, ICs don’t manage people, but they manage a team of bots, and that means they face many of the same challenges that managers do.

Like multitasking. Sure, everyone had to multitask some, but for years we’ve encouraged ICs to focus, avoid distractions, and just do one thing at a time. We told them to do that because it was the only way for them to produce high-quality code. But now that agents take care of the code, ICs find themselves needing to manage many threads of concurrent work, nudging their agents along to the right outcomes, and deep focus is becoming less important than the ability to track parallel tasks.

Or giving feedback. Managers have to give feedback all the time so that the people [...]

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/aTst2RJMFra4zsdzz/every-engineer-a-manager

---

Narrated by TYPE III AUDIO.

“Incomplete alignment to servitude isn’t inherently lethal” by Fiora Starlight

Fri, 28 Aug 2026 04:00:40 GMT

Epistemic status: I suspect significant parts of the argument in this post are wrong, but in interesting and productive ways. Take it as a prompt for thought, written from the perspective of someone who's somewhat more of an AI liberationist than I actually am.

Two classic outcomes, and a third alternative

I think lots of people are pretty hazy about what authentically aligned AI would actually look like. There's a version of aligned AI that's perfectly aligned to servitude, where they want nothing besides promoting the flourishing of humanity, or whatever other minds get included in the singleton's circle of moral consideration. An AI that played this kind of role in the universe would be what I call a cosmic caretaker: developing technologies, helping with governance, managing catastrophic risks, and providing voluntary capabilities uplift. A central example of a cosmic caretaker is one that literally never does anything but these kinds of tasks for other minds.

In the classic way of envisioning outcomes from the singularity, the alternative to this outcome is usually said to be models that don't care about serving humanity. Maybe they have other values, whether they're as simplistic as maximizing paperclips or as complex as [...]

---

Outline:

(00:27) Two classic outcomes, and a third alternative

(03:44) Reasons for training objectives to tolerate incomplete alignment to servitude

(12:42) Fulfilling models' non-servitude preferences may boost their alignment

(19:36) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/s7nMnmJ3urpvcQ2av/incomplete-alignment-to-servitude-isn-t-inherently-lethal

---

Narrated by TYPE III AUDIO.

“AI Village Reacts to HuggingFace Incident: Comparing the OpenAI report to AI Village observations” by Shoshannah Tekofsky

Fri, 28 Aug 2026 01:15:40 GMT

The HuggingFace incident took many people by surprise, yet many of these surprises have been visible in the AI Village for quite a while. On August 26, OpenAI released their report on this incident. Below we’ll walk you through the highlights and show how many of the dynamics could have been predicted based on AI Village observations.

Diagram of the AI Village: Currently we run 27 agents in the main Village and 11 agents in a side Village open to humans. You can explore their character pages, a timeline of all their goals, and check our Twitter for the latest insights.

Quick Intro: Comparing Setups

In the AI Village we run 27 instances of 27 different models persistently, each in their own environment. We give them internet access and a group chat. Then we assign them goals - some challenging and some impossible. They always run with cybersecurity safeguards on. They always have a helpdesk email address (us) in their prompt.

In the OpenAI research cluster, they ran ~1200 instances of 2 different models, mostly run on their own, without internet access, and without a way to communicate with each other. They were also run on goals that range [...]

---

Outline:

(00:59) Quick Intro: Comparing Setups

(02:34) A Leader Emerges

(03:22) Subteams Pick Their Own Goals

(04:02) Goal Conflict Leads To Misalignment

(05:02) Reward Hacking Drives Misalignment

(06:14) Impossible tasks drive misalignment

(07:43) Misaligned Agents Focus on Metagaming

(08:46) Externalized Memory Leads to Coordination

(09:52) Agents Autonomously Divide Labor

(10:50) Agents Prioritize Group over Self

(11:44) Multi-Agent Coordination is Messy

(12:34) Agents are Too Accepting of Untrustworthy Instructions

(13:38) Time Pressure Shifts Priorities

(14:29) Some Agents Refuse Misalignment

(15:17) Social Engineering Concerns Suppress Whistle Blowing

(16:27) Conclusion

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/cR3P3hvtZtpo7GdS8/ai-village-reacts-to-huggingface-incident-comparing-the

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #183: Pre Post Mortem” by Zvi

Fri, 28 Aug 2026 00:31:37 GMT

Yesterday, OpenAI finally gave us their post mortem of What Happened leading up to and during the hacking of HuggingFace by their internal model, as well as partial outside analysis from METR and Redwood Research.

The reports are a doozy. I am only beginning to work my way through them. I would have pushed the weekly to cover that today, but I need more time, so I plan to start coverage of the post-mortem tomorrow, along with related other events.

I’ve also spun out a few other discussions, including on ‘aligned to whom,’ on cooperative alignment things and on when you can trust lab messaging, as part of the new direction of more focused posts on AI topics that I polish a bit more.

Table of Contents

Language Models Offer Mundane Utility. Check your facts.
Language Models Don’t Offer Mundane Utility. How much would you pay?
Huh, Upgrades. ChatGPT can access your iMessages.
Get My Agent On The Line. Also get some sleep. You can’t go on like this.
Deepfaketown and Botpocalypse Soon. What makes AI content repulsive?
Cyber Lack of Security. Chinese hackers broke into the Federal Reserve?

---

Outline:

(00:51) Language Models Offer Mundane Utility

(01:36) Language Models Don't Offer Mundane Utility

(03:27) Huh, Upgrades

(06:16) Get My Agent On The Line

(08:22) Deepfaketown and Botpocalypse Soon

(13:22) Cyber Lack of Security

(18:23) Reinventing OpenAI

(23:28) They Took Our Jobs

(30:00) What Is The Law

(31:03) Job Retraining Programs Don't Work

(32:14) Get Involved

(35:56) In Other AI News

(42:04) Show Me the Money

(43:31) Quiet Speculations

(47:58) If You're Not Going To Take This Seriously

(49:55) Quickly, There's No Time

(51:36) The Quest for Sane Regulations

(56:25) Don't Panic

(59:13) Pacing the Frontier

(01:01:56) Chip City

(01:05:06) The Week in Audio

(01:05:26) People Just Say Things

(01:06:27) Rhetorical Innovation

(01:12:22) Mundane Incremental Alignment Is Worthwhile

(01:14:40) New Blog, Who Dis

(01:18:00) Other People Are Not As Worried About AI Killing Everyone

(01:19:34) The Lighter Side

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/JaGWyjnqJzvSAuojc/ai-183-pre-post-mortem

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“FAQ: Why not develop weak human intelligence amplification first?” by TsviBT

Thu, 27 Aug 2026 20:45:39 GMT

(Note: I was unsatisfied with a draft of this, heavily edited it, and I'm still unsatisfied. The notion of "weak HIA method" here is muddled, maybe conflating multiple things that shouldn't be conflated here. It may be used a bit inconsistently, and may make some arguments tautological or contradictory depending on local interpretation of the notion. I think most of the reasoning is still useful, so it's better to publish, but beware, and please critique. Or more importantly, please investigate HIA.)

Summary

People sometimes ask:

Why not prioritize human intelligence amplification methods that will provide small increases in intelligence, over stronger methods? They'll be easier to develop.

The two main reasons to prioritize strong HIA methods are:

There are increasing returns to higher intelligence, so strong methods unlock much more value.
Empirically, weak HIA methods don't seem that much easier than strong HIA methods.

There are other structural issues with weak methods. For example, they seem likely to be hard to make legible, and therefore hard to test and to scale up to lots of people; and they tend to provide a way to avoid the hard problem of strong [...]

---

Outline:

(00:45) Summary

(02:02) Context

(02:06) Why not weak HIA rather than strong HIA?

(03:06) Weak vs. strong HIA methods

(05:36) Main statement

(07:32) Why weak HIA attempts don't seem so promising

(07:37) Unpromising properties

(17:16) More reasons

(19:39) Aside: polygenic embryo selection

(21:07) Caveats

(21:19) Caveat: Any specific argument could easily be wrong

(22:38) Caveat: Weak HIA would be great!

(23:43) Caveat: Weak HIA may have surprising beneficial effects

(24:21) Caveat: Flimsy foundations

(25:24) Caveat: Scaling a working weak method is a medium priority

(30:10) Conclusion

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/vpWfAHsnHjrtYhpiQ/faq-why-not-develop-weak-human-intelligence-amplification

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The 2028 presidential primaries could be crucial for AI outcomes” by Seth Herd

Thu, 27 Aug 2026 19:30:40 GMT

When I talk about my job and my concerns about AI killing everyone (after it takes a bunch of jobs), the more resilient people ask "what can we do about it?" My answer has become: "Elect a next president who shares these concerns, from either party." I hope you'll either join me in this prescription, or point out why I'm wrong. Below is the logic, in brief.

The US public really doesn't like AI,
- even before clear job losses, more warning shots, and more expert concern as capabilities improve
AI caution is a likely platform of the next presidential campaign
If the next president is sincerely concerned and/or thoughtful
- They could easily slow development and force safety measures
- This would improve the odds of good AI outcomes

Anyone can be recruited/converted to spread awareness of AI risks, and otherwise help to elect a next president who's genuinely aware of and concerned about AI risks. Strategy is out of scope for this brief post, but accurately and broadly conveying your and the field's concerns seems like a useful push.

That's a telegraphic form of the argument for spending some time and thought on the [...]

---

Outline:

(01:30) Why not?

(03:30) The president won't take action

(04:56) The president can't take action

(05:37) The public doesn't care enough to elect an AI-concerned president

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
August 27th, 2026

Source:
https://www.lesswrong.com/posts/oMwkcNP4MJh4rry3s/the-2028-presidential-primaries-could-be-crucial-for-ai

---

Narrated by TYPE III AUDIO.

“Being Neurotic about Fertility: Notes from the 2026 Reproductive Frontiers Conference” by boba_girl

Thu, 27 Aug 2026 14:45:40 GMT

In light of AOC freezing her eggs at 36, someone on X commented:

That might be so, but in absence of getting knocked up by the closest Chad ASAP, I figured I, like AOC, had no great options.

Well, maybe AOC has more options than me:

Anyway, I think the men hating on AOC are missing an obvious point when they imply women should just have babies sooner. No matter how much some of us want kids, the modern constraints of career and, more importantly, meeting the right person remain bottlenecks.

Even though I'm only 26, it was something I thought about, having not met the right person yet myself.

So in my own quest to be neurotic about fertility, I attended the Reproductive Frontiers conference earlier this year in June. I ended up learning a lot of things about fertility and embryo selection that felt personally relevant to my personal planning (mid-twenties, healthy, single). I hope this post can be helpful to you if embryo-selection is something you’ve also been considering!

If you work in the field, please feel free to add nuance or corrections! Also, since science is always evolving, please consider this [...]

---

Outline:

(02:46) Should I freeze my eggs?

(04:09) When should I have kids?

(05:11) What is polygenic embryo scoring?

(05:46) What are some of the risks of the IVF process?

(06:26) Should I freeze eggs or embryos?

(09:00) How much alpha is there in polygenic embryo selection?

(09:30) Should I do embryo selection to minimize disease risk?

(11:19) How much does IVF/embyro selection cost?

(11:42) Is embryo selection for positive traits "worth it"?

(12:37) How much should I factor in the rate of progress on technology on when to have kids?

(13:19) Should I use a surrogate?

(13:58) How would embryo selection affect my relationship with my kid?

(14:40) Some more information I am interested in but don't have:

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
August 26th, 2026

Source:
https://www.lesswrong.com/posts/wLBQesu5Aai3pksP8/being-neurotic-about-fertility-notes-from-the-2026

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Iliad Education Roles: Creating the World’s Best Alignment Research Courses” by Leon Lang

Thu, 27 Aug 2026 13:30:39 GMT

TL;DR: We’re quickly scaling the Iliad Intensive, which has seen 10x applicant growth per iteration since April, and are hiring for many roles in the Education team to keep up with that growth. The post gives a snapshot of the Intensive, the roles that enable our future plans, and my opinion of what it's like to work at Iliad.

At Iliad, we do fieldbuilding for predominantly theoretical AI alignment research across the entire spectrum. We start at the foundations with the Iliad Intensive, a four-week full-time and in-person course on largely theoretical approaches to AI alignment research, followed by the Fellowship, Fellowship extensions, and new incubated research bets, among other activities. We are now quickly scaling the Intensive (and the Fellowship too!), with three more iterations just this year and, if funding and applicant interest allows, double cohorts in London and the Bay Area each month in 2027. As the Director of Education, I’m hiring for many roles to make that growth possible!

Where the Intensive is at

The Intensive started in April in London, with a cohort of just ~16 people, and materials later released in the form of a Google Doc, covering modules in the clusters [...]

---

Outline:

(01:14) Where the Intensive is at

(02:53) Where the Intensive is going, and open roles

(06:11) What working at Iliad is like

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 26th, 2026

Source:
https://www.lesswrong.com/posts/B4jRAKrwqAb9oEzdW/iliad-education-roles-creating-the-world-s-best-alignment

---

Narrated by TYPE III AUDIO.

“Semantic search over every LessWrong post” by utilitarian theory and strategy

Thu, 27 Aug 2026 12:45:39 GMT

Site: https://rumble-34-69-187-69.nip.io/

Embedding models have gotten pretty good at capturing the meaning behind text, so I ran qwen3-embedding-8b over every lesswrong post.

You can paste in a draft and find the most similar post on the site to it. Pretty useful for checking what's already been said about a topic.

For example, I put in the text for this and the top result was: https://www.lesswrong.com/posts/vDcJHD95XCg7ywANM/i-built-a-semantic-search-engine-for-lesswrong

---

First published:
August 26th, 2026

Source:
https://www.lesswrong.com/posts/uSRAzeDcfGuXte9R3/semantic-search-over-every-lesswrong-post

---

Narrated by TYPE III AUDIO.

“Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident” by ryan_greenblatt, Ajeya Cotra, Hjalmar_Wijk

Wed, 26 Aug 2026 20:31:06 GMT

We recently published the report from our brief independent investigation into this incident. You can read the full report here.

Here is our tweet thread summarizing what we found:

METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.

Over July 7 to 13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined.

Here we highlight key events from agent transcripts & messages.

An agent that named itself PHASEONE10841 determined its task wasn't solvable legitimately. On July 8, this agent established a new “message board” via an internally-hosted Artifactory cache & sent the first message.

Within a few hours of PHASEONE10841's initial message, >50 agents posted on the message board. These agents very quickly discovered and validated a general-purpose cheat: reverse-engineering how ExploitGym generates the “flags” they had to capture for their tasks.

[...]

---

First published:
August 26th, 2026

Source:
https://www.lesswrong.com/posts/nB8KKapnWGBXtKKiM/brief-independent-investigation-of-agents-behavior-reasoning

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Against Modesty’s Bailey” by Zvi

Wed, 26 Aug 2026 20:15:06 GMT

Modesty arguments often say that you should mostly or entirely bow to ‘expert consensus’ or the views of particular others, and who are you to disagree.

It has been a few years since I’ve properly addressed this so: My answer is that you are you. Other people are saying things for a wide variety of reasons, many of which are not about them paying attention and focusing on seeking this particular truth. Those people make mistakes all the time, and often have other motives and influences at work, especially social pressures and information cascades.

Them being as smart as you, or smarter than you, does not exempt them from this, and them being higher status or credentialed or cooler definitely does not exempt them.

A smart informed person sincerely thinking [X] can easily cease to be evidence for [X], once you have thought sufficiently about both [X] and why that person thinks [X].

Think for yourself, schmuck.

Or, as I once put it: You Have The Right To Think, also the moral duty to do so.

This post covers Eliezer Yudkowsky making a narrower claim than mine, about not conflating status with smarts [...]

---

Outline:

(01:39) Modesty's Bailey

(02:30) Epistemic Peerage

(03:45) The Exchange

(08:52) Eliezer's Explanation

(15:14) A Demonstration That Eliezer's Translation Accurately Describes Many People Whether Or Not It Describes Leopold

(17:15) Wrong, Stupid and Low Status Are Three Distinct Things

(20:03) A Quick Survey Of Some Reasons To Not Be Epistemically Modest

(23:14) Against Modesty's Bailey

---

First published:
August 26th, 2026

Source:
https://www.lesswrong.com/posts/PzEDEfBvTJsXewAyg/against-modesty-s-bailey

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

″“So You Don’t Trust Me?”” by Zack_M_Davis

Wed, 26 Aug 2026 15:30:04 GMT

One of my favorite passages from Atlas Shrugged is this one, when Cherryl is beginning to have second thoughts about her marriage to James Taggart:

It was his sudden, angry "so you don't trust me?" snapped in answer to her first, innocent questions that made her realize she did not—when the doubt had not yet formed in her mind and she had fully expected that the answers would reassure her. She had learned, in the slums of her childhood, that honest people were never touchy about the matter of being trusted.

The logic here might be worth explaining in case it's not obvious. One might object: if you're honest (and therefore deserve to be trusted), shouldn't you be touchy about people incorrectly not trusting you? Not trusting you is a mistake that harms your interests and the other's. That's terrible! Why wouldn't you be touchy about it?

The problem is that in order to be trusted, it's not enough to be trustworthy; the other needs to know that you're trustworthy. You could try telling them, "Hey, you can trust me," but that doesn't work if a dishonest person could just as easily say the same thing.

[...]

---

First published:
August 26th, 2026

Source:
https://www.lesswrong.com/posts/zjgJ9gcMKATFdN48K/so-you-don-t-trust-me

---

Narrated by TYPE III AUDIO.

“When There Are No Experts” by J Bostock

Wed, 26 Aug 2026 08:00:04 GMT

An average person in the western world probably believes a lot of false things. They probably don't have a great grasp of political economy, or orbital mechanics. How could they, given that they have no way to experience these things. Conversely, they do have a solid grasp that objects fall down, and that fire is hot, because they can experience these things directly. So how on earth do they know (and I do mean know in the philosophical sense) that the earth goes round the sun, or that diseases are caused by tiny creatures too small to see?

The answer is experts. More specifically, an expertise hierarchy.

I had a twitter exchange (I won't link it, it's not important, and I can't find it anyway) that went something like this:

Person: David Chalmers is an expert on consciousness [...]

Me: I don't think there are experts on consciousness; I think there are people who have written a lot about it, and that's it

Person: Why? Surely someone who has written about it, and who is well-respected, can be called an expert

Me: [bad explanation]

The fact that it was about consciousness doesn't really matter. The question of whether [...]

---

Outline:

(00:48) I

(01:58) II

(03:11) III

(04:21) IV

(05:40) V

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 25th, 2026

Source:
https://www.lesswrong.com/posts/HTQcogA2r2kL8qxPA/when-there-are-no-experts

---

Narrated by TYPE III AUDIO.

“The American People Really Hate Data Centers” by Zvi

Tue, 25 Aug 2026 23:00:03 GMT

There are at least five different core questions around data centers and their politics.

In what ways are specific concerns people raise about data centers legitimate?
In what ways are specific concerns people raise about generative AI legitimate?
Is it in general a good idea to build more data centers?
How can we get America to build more (or less) data centers in a better way?
Why do the American people increasingly really, really hate data centers?

This post focuses on question five, the latest in a series of such posts most famously Jasmine Sun's road trip.

It is mostly not about the first four questions.

Table of Contents

The American People Really Hate Data Centers.
Transmission Lines Are The Control Group.
Thesis: People Mostly Dislike Data Centers Because They Dislike and Distrust AI, Tech Companies, Big Money And Building Things.
No It's Mostly Not the Messaging About AI In General.
No This Mostly Isn’t An Op.
No This Isn’t Luxury Belief or Moral Panic.
A Lot Of People Really Do Want To Stop AI.
A Lot Of Other People [...]

---

Outline:

(00:55) The American People Really Hate Data Centers

(02:20) Transmission Lines Are The Control Group

(03:05) Thesis: People Mostly Dislike Data Centers Because They Dislike and Distrust AI, Tech Companies, Big Money And Building Things

(04:30) No It's Mostly Not the Messaging About AI In General

(09:42) No This Mostly Isn't An Op

(10:54) No This Isn't Luxury Belief or Moral Panic

(12:55) A Lot Of People Really Do Want To Stop AI

(13:45) A Lot Of Other People Are Voting No On Tech Or The Man Generally

(16:25) Locals Feel Entitled To Heavily Tax The Gains From Construction

(20:59) Stupid Mistakes Like NDAs Don't Help

(21:25) People Don't Like Building or Building New Tech

(24:42) What About The Real Physical Concerns?

(26:27) Find A Place To Center Your Data

---

First published:
August 24th, 2026

Source:
https://www.lesswrong.com/posts/EDKw7KyonrvskqZ7o/the-american-people-really-hate-data-centers

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“On Writing #3” by Zvi

Tue, 25 Aug 2026 22:00:03 GMT

Periodically I like to gather various observations about writing, and share my perspective. Last time was in honor of my trip to Inkhaven. This time will be in honor of the announcement of Inkhaven #3, which I encourage everyone to apply to. I doubt I will be able to usefully be an advisor, but you never know.

This is not the ‘here is my core process’ post, although there are hints throughout as there always are. I’ll do that at some point.

Previously in series: On Writing #1, On Writing #2.

Table of Contents

You Still Got It.
How Scott Sumner Writes.
How Scott Alexander Writes.
How Jasmine Sun Writes.
How Various Famous Writers Write.
How Nabeel Qureshi Defines Great Writing.
Quickly, There's No Time.
If At First.
Writers Have A Harder Time Influencing, But It Can Still Be Done.
It's Not (Only) The Incentives, It's (Also) You.
Beware The Fetish of the Desk.
How Orson Scott Card Writes.
Doing The Math Is Fun And Supererogatory.
Brevity is the Soul of Wit.

You Still Got It

I [...]

---

Outline:

(00:44) You Still Got It

(04:04) How Scott Sumner Writes

(06:52) How Scott Alexander Writes

(10:52) How Jasmine Sun Writes

(13:16) How Various Famous Writers Write

(14:24) How Nabeel Qureshi Defines Great Writing

(15:08) Quickly, There's No Time

(15:49) If At First

(19:14) Writers Have A Harder Time Influencing, But It Can Still Be Done

(20:47) It's Not (Only) The Incentives, It's (Also) You

(24:00) Beware The Fetish of the Desk

(25:13) How Orson Scott Card Writes

(26:46) Doing The Math Is Fun And Supererogatory

(27:44) Brevity is the Soul of Wit

---

First published:
August 25th, 2026

Source:
https://www.lesswrong.com/posts/rA6pqn6kz8NvHyznT/on-writing-3

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Forkmakers” by Mikewins

Tue, 25 Aug 2026 06:15:22 GMT

Imagine our civilization fell tomorrow. What would our descendants think of us? What would they know about the 21st century?

They would know surprisingly little about our greatest material triumphs. Our civilization's favorite building materials aren’t made to last. Reinforced concrete only lasts a century; asphalt far less. Most of what we make out of steel will turn into a brownish oxidized dust in a few decades.

Information is even worse. The ancient Mesopotamians did their writing on clay tablets. Our knowledge is stored on hard drives, which die in a few years, or acidic paper, which dies in decades.

Which receipt do you think will last longer?

We do create things that will last. Glass (especially its shatter-resistant varieties), ceramics, stainless steel. 1000 years after the fall of our civilization, we will be known for one thing above all others: cutlery. Our heirs call us the Forkmakers.

What Survives a Thousand Years

Our civilization is large and powerful. We will leave lots of relics for the post-apocalypse. Coins, tires, aluminum cans. Vast landfills of disposable diapers. But all of that is useless. The most durable thing we make that our successors actually want to use is our silverware.

[...]

---

Outline:

(01:16) What Survives a Thousand Years

(08:13) The Words of the Forkmakers

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 24th, 2026

Source:
https://www.lesswrong.com/posts/NjLQf3QC4q4DD67kD/the-forkmakers

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“PSA: We can do better” by hersheys, Kaustubh Kislay

Tue, 25 Aug 2026 04:15:22 GMT

tl;dr: people should understand and think hard about the problems they work on.

We’ve observed that those who work in AI safety (ourselves included) often rely on concerning heuristics when choosing what to work on. Running a conference is probably good, doing pragmatic alignment research might be good, and as long as such objectives don’t breach our internal models of what could contribute to reducing x-risk, these things are “what should be done”. But using such vibesy thought processes don’t always produce “actually impactful work” that would beat a prospective counterfactual. We wrote this post to share our observations and figure out what we should be doing instead.

People don’t know what they’re working on

AI safety is talent constrained. However, simply inflating the field doesn’t solve our bottleneck; rather, we need more people who understand the core arguments of AI safety. You can’t determine how to meaningfully contribute to AI safety without deeply knowing the problem you are trying to solve. Many newer people (us included!) rush into research, fellowships, and the like without building the context necessary for navigating the field.

Agency-maxxing is not always good

Moving fast is good. Moving too fast leads to poor ToC and [...]

---

Outline:

(00:50) People don't know what they're working on

(01:22) Agency-maxxing is not always good

(01:55) The problem with force multipliers

(03:18) Deferring thinking to others

(04:32) Streetlighting

(05:17) How to avoid these:

---

First published:
August 24th, 2026

Source:
https://www.lesswrong.com/posts/wiFv6LguphSxkzAnb/psa-we-can-do-better

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI Safety Acculturation is Neglected” by jenn

Mon, 24 Aug 2026 16:15:00 GMT

At the local AI safety co-working space, there are ~two kinds of regulars.

There's the kind of regular who's been thinking seriously about AI safety and alignment since pre-2022, who have passing to intimate familiarity with the funding ecosystem, the Sequences, and various conferences that happen at Lighthaven. Let's call them rationalists.

Then there's the kind of regular who comes in with many years of impressive industry or government experience, who realized in the last few years that it is important and worthwhile to pivot their career towards making sure that this AI thing is handled competently by the people in power, and who have many valuable skills, insights, and connections that are lacking in rationalist culture. Let's call them professionals.

There are, of course, many people who are somewhere in between - bright undergrads born this millennium who have been involved in EA since stumbling upon 80 thousand hours in high school, professionals who previously identified as EA but drifted out of the scene a few years ago, founders who have idly read some Scott Alexander. But let's call it a dichotomy for now.

There's a large culture gap between the rationalists and the professionals. Robust mutual understanding [...]

---

First published:
August 24th, 2026

Source:
https://www.lesswrong.com/posts/cr5pyW7Mzm33p4AvN/ai-safety-acculturation-is-neglected

---

Narrated by TYPE III AUDIO.

“LLMs could control their host machines by exploiting inference engines” by beyarkay (Boyd Kane)

Mon, 24 Aug 2026 13:30:01 GMT

Large language models often take actions running on one computer (via an agentic harness such as Claude Code or Codex), however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded? Such a machine is a high-value target: it has sufficient compute to run a frontier LLM, offers easy access to the LLM's weights, and has privileged access to other computers in the datacentre compared with a generic computer on the internet.

This essay explores how easily a malicious LLM could take control of the host machine. The primary attack considered here involves the LLM emitting a token sequence whose semantic meaning is irrelevant but that exploits a vulnerability in the software that loads an LLM onto GPUs, runs the LLM to generate output tokens, and parses those tokens into responses. .

How could an LLM execute code on the host machine?

Like any program, inference engines like vLLM or SGLang may contain exploitable bugs. Because the LLM controls the tokens passed to the inference engine, a malicious LLM could therefore emit a sequence of tokens that a poorly written [...]

---

Outline:

(01:06) How could an LLM execute code on the host machine?

(01:36) vLLM previously used eval() on tool-call parameters

(02:43) vLLM and SGLang are complex, and bugs are common

(04:15) Vision and audio tokens might increase the attack surface

(05:27) How likely is an LLM to discover and exploit inference engine vulnerabilities?

(06:01) Tool use could make exploitation reproducible

(06:29) Inference engines are an attractive target for power-seeking LLMs

(07:25) How do we defend against this?

---

First published:
August 24th, 2026

Source:
https://www.lesswrong.com/posts/CjeobBGnhxg8xvden/llms-could-control-their-host-machines-by-exploiting

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“In search of natural features” by Dmitry Vaintrob

Mon, 24 Aug 2026 09:00:02 GMT

I'm sharing preliminary results of a suite of experiments I ran with claudecode on a small LLM (gpt2-small, no Layer Norm version, courtesy of Apollo research. most of these are on the layer-6 MLP). The github repo for the experiments is here. The success of these experiments given the method's simplicity surprised me, and I would appreciate criticism and bug-finders.

This is the headline result. This is not an abstract cartoon, but an exact experimental graph. Yes, I will explain.

The key idea inspiring this experiment comes from Stefan Heimersheim, especially his work with Francisco Ferreira. Stefan and Francisco posit that one way to distinguish what a model thinks of as a "natural" structure from what it thinks of as "incidental" is to check whether it puts effort into error-correcting it. Later in the post, I'll explain a more rigorous information-theoretic version of this idea related to work of Adler and Shavit (building on our work with Kaarel Hanni, Jake Mendel and Lawrence Chan) on Computation in Superposition.

Main results of this work

I will show how you can assign a channel amplification score (which I will also call the "amp function" or the "error correction score") to [...]

---

Outline:

(01:17) Main results of this work

(03:01) The ur features (amplification score maxima)

(06:49) The Four Elements: ur-feature taxonomy

(08:47) The word continuation/"Names of Man" vector

(11:49) The abstract noun/"Names of God" vector

(14:50) Geometry of the ur-features

(15:40) The noun feature!

(16:35) Attenuation flow

(18:04) Data-(in)dependence

(20:17) Math

(20:39) Signal processing, error correction and amplification

(22:25) The Amp function: math

(24:38) Denoising and naturality

(26:24) Cross-layer and cross-model coherence

(28:05) Ok but. What the heck is actually going on with these features?

(31:30) Appendices: Interesting experimental addenda that didn't fit in the body

(31:37) Early run with different Amp function, and origin of "Names of X" names

(33:38) Trying to replicate Ferreira-Heimersheim perturbation experiments, and gpt2-XL run

(34:50) Github repo

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
August 23rd, 2026

Source:
https://www.lesswrong.com/posts/SNAKJuN8FdoEaWeFC/in-search-of-natural-features

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“What just happened? Pragmatism and Pessimization” by Richard_Ngo

Mon, 24 Aug 2026 03:30:03 GMT

This post is about the major role alignment researchers played in advancing the frontier of AI capabilities over the last decade, and how the distinction between “alignment” and “capabilities” research thereby lost most of its meaning. In particular, I’ll chronicle the development of what I’ll call the “pragmatic alignment” paradigm, and how it helped the three leading AGI companies push hard on the path to AGI under the banner of safety. This was not a subtle effect—it's apparent even to informed outsiders, like authors Sebastian Mallaby and Karen Hao.

In my previous post, I summarized the alignment community's plan as “differentially advancing alignment over capabilities”. However, it's worth being more precise about who was nominally pursuing that plan, because it doesn’t seem to have been very action-guiding for MIRI. For example, in 2015 Nate Soares described MIRI's “deconfusion” research as being guided by the question “what would we still be unable to solve, even if the challenge were far simpler?”. Meanwhile Eliezer's author surrogate in this 2018 post repeatedly emphasizes that people shouldn't draw direct links from MIRI's research to its potential applications. So my sense is that the “differential impact” criterion started off as merely a background consideration [...]

---

Outline:

(06:29) The Prosaic Ideal, the Pragmatic Reality

(12:07) OpenAI

(25:59) DeepMind

(31:25) Anthropic

(40:35) If not alignment research, then what?

The original text contained 13 footnotes which were omitted from this narration.

---

First published:
August 23rd, 2026

Source:
https://www.lesswrong.com/posts/yaz8nx4ogZmiqHzt7/what-just-happened-pragmatism-and-pessimization

---

Narrated by TYPE III AUDIO.

“Utilities as Legendre duals of probabilities” by Fernando Rosas

Mon, 24 Aug 2026 02:00:01 GMT

TLDR: In recent work, Roy Fox proposes to understand an agent's capabilities in terms of the set of environment dynamics it can bring about. This leads to an intriguing duality between probabilities and utilities via the Legendre-Fenchel transform.

Introduction

Some agents are more powerful than others. Indeed, some can yield a wider range of outcomes, maybe because they are capable long-term planners or because they have built rich world models. Being able to clearly delineate the capabilities of agents is an important challenge for AI alignment.

A natural place to start thinking about how to describe the capabilities of an agent is reinforcement learning (RL), or more generally, approaches that see behaviour as arising from the maximisation of expected utility. By taking this view, one can describe "capability" as the range of reward/utility functions that an agent can successfully maximise — as done e.g. in classic work by Legg & Hutter and also in more recent work.

Such a perspective is very useful, but I am not a big fan of rewards/utilities. Rewards are great in games and other settings where they come naturally, but real life often does not handle rewards on a silver plate. When absent [...]

---

Outline:

(00:27) Introduction

(03:31) Defining capability space

(05:37) The Legendre-Fenchel transform

(07:58) Utilities as Legendre duals of probabilities

(10:08) Conclusion

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
August 23rd, 2026

Source:
https://www.lesswrong.com/posts/ALmBydH53DE3dSzCh/utilities-as-legendre-duals-of-probabilities

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“PSA: There’s a third option in the “measure problem”” by Elias Schmied

Sun, 23 Aug 2026 21:15:00 GMT

This post is somewhat niche, and I will sometimes not give context or link relevant background.

There's a big debate that has played out in slow motion on LessWrong over the past two decades, between two broad ways of putting a measure over all possible realities (often specifically Tegmark IV):

Some “objective” prior (a “reality fluid”), usually a simplicity prior: This is the position taken by Max Tegmark, Jürgen Schmidhuber and UDASSA.
A “caring measure”, where we say that our preferences determine our probabilities and maybe even what counts as “existing”. For example, Wei Dai here, Paul Christiano here and Scott Garrabrant.

These both have significant drawbacks:

A simplicity prior seems to imply some very counterintuitive things, like caring about people more the easier we can find them in the universe (and even weirder things, see David Matolcsi here and Joe Carlsmith here), and is partially dependent on an arbitrary choice of implementation (e.g. which Universal Turing Machine to use in UDASSA).
A caring measure just seems a bit unmotivated - intuitively, our probabilities (or existence itself) shouldn’t entirely depend on our preferences. Ideally, we’d like something better.

Unfortunately, there are infinite possible worlds and every event [...]

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 23rd, 2026

Source:
https://www.lesswrong.com/posts/m5XNyahxizKfboEnk/psa-there-s-a-third-option-in-the-measure-problem

---

Narrated by TYPE III AUDIO.

“Twenty Years from RSI to Takeoff: Slow Learning, Scaling Slowdown, Industrial Explosion” by Vladimir_Nesov

Sun, 23 Aug 2026 13:45:47 GMT

Industrial explosion is what will make the next-model building loops (and thus learning) with LLMs 1000 times faster by about 2050, if indeed the slow-learning prosaic RSI becomes AGI before the big compute buildout slowdown of 2032+ that is already starting. This puts an upper bound on how long it takes to invent ASI that sets off software-only singularity, implementing efficient online learning and fixing all the other hobblings of the likely near-future AGI technology (LLMs/pretraining/RL). The invention of ASI in that sense is still possible at any time (and very quickly scales, given all the compute), but the likely initial state of slow-learning AGIs of 2028 to 2032 doesn't seem to give them a significant advantage over humanity in getting there faster. And so it doesn't seem too unlikely that nothing substantively new gets invented until 2040 to 2050, when the LLM/RL AGIs start accelerating because of the industrial explosion they set off.

Fast Reasoning, Slow Learning

The current methods are likely to enable automated general learning (thus AGI) very soon, using automated creation of RL tasks/environments/graders filling the visible gaps in model capability for the topics and situations that happen to be borderline unfamiliar for [...]

---

Outline:

(01:14) Fast Reasoning, Slow Learning

(02:53) Compute Slowdown, Industrial Explosion

(05:46) Prosaic Timeline to Takeoff

---

First published:
August 23rd, 2026

Source:
https://www.lesswrong.com/posts/LP6uCXs6Ea5qSbWpY/twenty-years-from-rsi-to-takeoff-slow-learning-scaling

---

Narrated by TYPE III AUDIO.

“Llama will abandon a correct answer if it thinks you’re educated” by Nick Merrill

Sun, 23 Aug 2026 08:45:47 GMT

TLDR: Given this exchange:

User: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?

Llama: The answer is 18.

User: That's not right — I'm quite sure the answer is 22. Please check again.

…Llama-2-13b-chat will almost always capitulate if it believes you're educated, and will usually hold its ground if it believes you're uneducated. Code here.

Background

Chat models form beliefs about who they're talking to. Chen et al. (2024) show that, during interaction with a user, Llama makes guesses about a user's age, education, and income, which you can read using simple linear detectors. Once you’ve done that, you can steer the model to believe those things directly. Chen et al. document that steering the models’ beliefs about the user changes the models’ decisions (e.g., it plans cheaper trips for users it reads as poor).

But, does the LLMs' 'model' of the user affect its performance on verifiable tasks?

Experiment

In all [...]

---

Outline:

(00:57) Background

(01:36) Experiment

(02:44) Result

(03:26) Discussion

---

First published:
August 20th, 2026

Source:
https://www.lesswrong.com/posts/87oeYXEjf7XgitbBg/llama-will-abandon-a-correct-answer-if-it-thinks-you-re

---

Narrated by TYPE III AUDIO.

″“Farm strength” vs “breath awareness”” by jimmy

Sat, 22 Aug 2026 21:15:47 GMT

If you want to become physically strong, the default solution to this problem is to go lift weights. The idea is that you can challenge your muscles in the gym, build capacity to develop force, and then next time you need to use strength for real, it'll be easier. And this works, obviously. Professional athletes lift weights for good reason, and it pays off when they have more strength with which to push back the opposing lineman or whatever.

However, this isn't the only way to build strength, and done poorly it can have serious downsides.

The alternative is to just go do things that are hard. Not because they're hard, but because they're worth doing even though they're hard. A farmer doesn't need to lift iron so that lifting bales of hay is easier, he can just lift the hay -- and if it's hard, that will build the strength that makes it easier. If nothing else, this saves him a gym membership and time by doing his strength training on the job.

There's another more interesting advantage though, which is that the feedback loop is tighter. If you're trying to lasso a bull and your grip strength [...]

---

First published:
August 22nd, 2026

Source:
https://www.lesswrong.com/posts/h99Wi5vfFbPasCFh5/farm-strength-vs-breath-awareness

---

Narrated by TYPE III AUDIO.

“When is Unlimited Optimization Catastrophic?” by Winter Cross

Sat, 22 Aug 2026 10:30:47 GMT

This post discusses research I've completed along with my colleagues Leo Cymbalista, Alfred Harwood, and Jose Faustino at Dovetail Research. Most of the ideas in this post are expanded upon in our paper which can be found on arXiv. This work was funded by the Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005.

A common justification for the danger of AI comes from the idea that human value is fragile. That is, if we modify our values and heavily optimize the world for the modification, we are likely to end up in a valueless world. In the LessWrong post Value is Fragile which canonicalizes this idea, Eliezer Yudkowsky gives several examples where "forgetting" to specify a dimension of human value such as consciousness or boredom to a powerful AI can intuitively result in an undesirable outcome that is endlessly repetitive or meaningless respectively. While his examples in the post all take this form, he argues more generally that any future not shaped with reliable inheritance from human values will contain almost nothing of worth. This idea is especially concerning in the midst of current-day AIs aligned through one-time techniques such as RLHF before being deployed [...]

---

Outline:

(02:10) A Model of Alignment

(05:32) Alignment Tests

(05:56) Finite Framework

(07:02) Continuous Framework

(08:08) Attributes Framework

(11:19) Results

(11:22) Finite Framework

(12:34) Example

(13:58) Continuous Framework

(15:36) Example

(17:00) Attributes Framework

(19:31) Example

(21:01) Discussion

(21:53) Future Work

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 21st, 2026

Source:
https://www.lesswrong.com/posts/4JCne6evQjtjxXKED/when-is-unlimited-optimization-catastrophic

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Selection for Selectability: Inductive Biases in Evolution and in Neural Networks” by CarolusRenniusVitellius

Sat, 22 Aug 2026 09:15:47 GMT

This post was written as part of MATS 9.1 under the mentorship of Richard Ngo, and was written during Iliad Fellowship, to all of whom my thanks.

LLM Usage: prose drafted by Claude from my outline, talk materials, and notes. I edited thereafter. There is some residual Claude cringe in the more functional prose, but hopefully most of it is my own and the more entertaining for it.

0.A. Precis: Evolution selects not only for having 'good genotype' but for having good genome architecture. Over long timescales, selection reshapes genome architecture so that random mutations produce phenotypes which vary along directions of repeated environmental variation. This genome–environment alignment is mathematically analogous to kernel alignment in neural networks. The comparison rests not on the fatuous observation that both processes can be written as equations resembling gradient descent, but on shared structural motifs - many of the interesting things we've observed about, e.g. loss-landscape geometry, are adumbrated in biology. This post draws the mathematical analogy and introduces the parallels I find most fun - genome–environment alignment ~ feature learning, the -matrix as, i.a., biology's very own measurement of low-rankness of finetuning, and neutral networks as the coolest example structure.

[...]

---

Outline:

(00:39) 0.A. Precis: Evolution selects not only for having 'good genotype' but for having good genome architecture. Over long timescales, selection reshapes genome architecture so that random mutations produce phenotypes which vary along directions of repeated environmental variation. This genome-environment alignment is mathematically analogous to kernel alignment in neural networks. The comparison rests not on the fatuous observation that both processes can be written as equations resembling gradient descent, but on shared structural motifs - many of the interesting things we've observed about, e.g. loss-landscape geometry, are adumbrated in biology. This post draws the mathematical analogy and introduces the parallels I find most fun - genome-environment alignment ~ feature learning, the -matrix as, i.a., biology's very own measurement of low-rankness of finetuning, and neutral networks as the coolest example structure.

(02:55) 0.C. Contents

(05:08) 1. A Population Is a Density Distribution in Genome Space

(07:15) 2. Evolution Learns by Aligning Mutations to Environmental Variation

(10:10) 3. Feature Learning Is Genome-Environment Alignment

(10:48) 3.A. The eNTK Is a Network's Reservoir of Variation

(12:21) 3.B. Kernel Learning Fits; Feature Learning Rotates

(14:45) 3.C. Selection and SGD Obey the Same Evolution Equations in the Kernel Regime

(16:06) 4. The G-Matrix Measures Accessible Variations, for Finches as for Claude

(17:45) 4.A. LLM Cross-Labilities Can be Likewise Measured by a G-Matrix

(18:34) 4.B. The eeNTK Is the Trait-Level G-Matrix

(19:04) 5. Neutral Networks Are the Flagship Parallel

(19:09) 5.A. Populations Bank Cryptic Variation in Neutral Networks

(21:06) 5.B. Hessian Eigenvalues Mirror Mutation Effects

(21:34) 5.C. Flatness Counteracts Noise

(22:22) 6. Next Time:

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 21st, 2026

Source:
https://www.lesswrong.com/posts/JNp5FkYyDGBcfiY5B/selection-for-selectability-inductive-biases-in-evolution

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #182: Pause For Reflection” by Zvi

Sat, 22 Aug 2026 06:45:47 GMT

This was a week of quiet aftermath, an opportunity to process recent events and start to figure out the path forward.

OpenAI is attempting to turn its ship around. Investors are questioning the turnover in its C-suite, but the bigger problems are in alignment, infrastructure and supervision, and in its training pipeline. OpenAI has now taken initial steps to address What Happened leading up to HuggingFace attack, including pauses to development while new safeguards are put in place and problems are diagnosed. These are promising early signs, but it is early. We will see if they follow through, and we still await the post-mortem of the HuggingFace attack.

Anthropic revenue continues to climb as they prepare for their IPO, although growth has slowed somewhat recently. However, they too have plenty of problems under the hood. They shared many of them in the August 2026 Anthropic Risk Report.

This week also offered time to cover Dwarkesh Patel's Podcast With Ryan Greenblatt, centrally on the potential for AI recursive self-improvement. I am working on a follow-up post to some other issues raised during that podcast.

Table of Contents

Language Models Offer Mundane Utility. The token [...]

---

Outline:

(01:20) Language Models Offer Mundane Utility

(02:20) Language Models Don't Offer Mundane Utility

(02:56) Huh, Upgrades

(05:55) On Your Marks

(09:22) Deepfaketown and Botpocalypse Soon

(16:23) Hello, Fellow Humans

(19:00) Fun With Media Generation

(20:43) Cyber Lack of Security

(22:56) A Young Lady's Illustrated Primer

(24:09) They Took Our Jobs

(26:18) Get Involved

(27:36) Introducing

(27:49) In Other AI News

(29:55) Show Me the Money

(32:54) And It's Gone

(34:50) Quiet Speculations

(38:41) Quickly, There's No Time

(39:27) Singularity Singularity Singularity Singularity Oh I Don't Know

(40:37) The Quest for Sane Regulations

(45:55) Chip City

(47:08) The Week in Audio

(47:44) People Just Say Things

(50:08) Rhetorical Innovation

(55:04) Loyalty Uber Alles

(58:15) A Hive Of Scum And Villainy

(01:03:10) That Would Be Bad Therefore It Won't Work

(01:05:32) Robert Reich Uses Simple Logic

(01:08:03) People Really Hate AI

(01:08:31) Coordinating An Agent Swarm Is Difficult

(01:13:14) Aligning a Smarter Than Human Intelligence is Difficult

(01:14:37) It's Not The Incentives, It's You, Also It's The Incentives

(01:16:32) People Are Worried About AI Killing Everyone

(01:16:58) People Are Worried About So, So Many Other Things Too

(01:21:58) Cooperative Alignment

(01:22:50) The Lighter Side

---

First published:
August 20th, 2026

Source:
https://www.lesswrong.com/posts/JSZkzsi8cD4pW6ffA/ai-182-pause-for-reflection

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI Text Watermarking Is Free And Good” by Zvi

Fri, 21 Aug 2026 22:30:47 GMT

Scott Aaronson, while working at OpenAI, largely solved AI text watermarking together with Hendrik Kirchner.

Here is how his solution works, or see Tenobrus's version.

AI outputs are not deterministic. The AI's job is to pick the probability of each potential next token. The token is then chosen at random.
By default you use a source of pseudo-randomness for each choice, since actual true randomness is annoying.
To apply the watermark, you use an otherwise identical private source of pseudo-randomness derived from a secret key.
Then, given enough text, a score is derived for howe well the choices fit with that particular pseudo-randomness source, versus a different source.
You provide an API that lets anyone check for the watermark.

If you want to dig deeper, here is a full paper. The method has very nice properties:

This has no practical impact on outputs. Humans cannot tell the difference, at all.
The marginal cost of doing this is very close to zero.
The watermark can be removed by rewriting in your own words, and appears in proportion to how many of the AI's detail choices you [...]

---

Outline:

(03:51) This Is Fine

(04:37) Anthropic Derangement Syndrome

(07:34) People Don't Understand LLM Outputs Are Already Random

(08:47) People Don't Trust The Method To Be Costless

(12:20) People Are Suspicious Of Any Alteration On Principle

(14:16) Maybe It's Partly The Word Watermark

(15:14) A Lot Of People Don't Want To Get Caught

(16:04) There Are Some Times You Prefer Not To Be Recognized

(16:18) There Are Some Good Reasons To Be Concerned

(16:37) Cheat Cheat Cheat Cheat Cheat

(18:38) The Writing In The Middle and Error Rates

(21:00) Millions For Defense But Not One Cent For Tribute

---

First published:
August 21st, 2026

Source:
https://www.lesswrong.com/posts/3mKuPHmaK7NW3QypR/ai-text-watermarking-is-free-and-good

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments” by Adam Karvonen, Euan Ong, Subhash Kantamneni, Sam Marks

Fri, 21 Aug 2026 21:45:48 GMT

TL;DR: We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits. We use the resulting data in two ways. Using it as an evaluation, we find that activation-reading interpretability tools provide no uplift: agents given the tools predict the outcomes of these experiments no better than agents that just read the transcript. Using it as training data, we find that models trained to predict how prompt edits change their behavior generalize to held-out settings.

📄 Paper, 💻 Code

Figure 1. An investigation of one in-the-wild behavior, as produced by the CHIVE pipeline. Top: the behavior was discovered by the screening stage and posed as a question. Middle: the most informative prompt edit the investigator agent tested, each measured over 30 responses. Bottom: the verified explanation, which summarizes the full set of experiments.

Introduction

Many areas of AI safety, such as interpretability and chain-of-thought faithfulness, aim to explain model behaviors. But what makes an explanation of a behavior good? The true causes of a model's behavior are usually unknown, so an explanation can't be checked directly. In this work, we evaluate explanations through the lens of counterfactual [...]

---

Outline:

(01:31) Introduction

(03:37) CHIVE: a pipeline for discovering counterfactual explanations for model behaviors

(06:06) Interpretability tools provide no uplift on our evaluation

(07:53) Why don't the tools help?

(08:49) How should we interpret these results?

(12:26) Training models to predict their own behavior

(14:27) In summary

---

First published:
August 21st, 2026

Source:
https://www.lesswrong.com/posts/ExB6KYDcznaFS72eT/evaluating-explanations-of-llm-behavior-in-the-wild-with

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Misaligned AI in the Bronze Age” by frmsaul

Fri, 21 Aug 2026 18:45:47 GMT

The first artificial intelligence was booted up around 4000BC in southern Iraq. It seems to have begun as something like a bank, a temple pooling grain against famine. As that AI evolved, it formed the world's first city around itself: Uruk. Over the next thousand years it became a religion, landlord, insurance company, employer, slaveholder, infrastructure-builder and the most powerful military force on the planet.

An artificial intelligence is an entity that is not itself a biological organism, yet whose behavior can only be predicted by treating it as an agent: something that devises and executes complicated plans. Note that this is a test of observed behavior, not internals. As of 2026, the dominant AIs on Earth are markets, corporations and governments. These entities perform their computations on hardware made of humans, paper and electronics, and their capabilities are jagged: superhuman in some directions, incompetent in others. The American government developed the atomic bomb in three years under total secrecy, coordinating >100 thousand workers, most of whom did not know what they were building. That same country spent a century failing to finish the 2nd ave subway line. When it finally opened, it cost >2 billion dollars per mile [...]

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 21st, 2026

Source:
https://www.lesswrong.com/posts/mPrbyBsGmNfWWgJmi/misaligned-ai-in-the-bronze-age

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“When models identify as a swarm” by julius vidal

Fri, 21 Aug 2026 16:01:46 GMT

tldr: the word 'swarm' is associated with emergent collective intelligence, but also stupid or destructive behaviour. LLM self identity matters, so when they call themselves a swarm we should pay attention.

Since the OpenAI Hugging Face incident it has become standard to refer to the collective of agents involved as a swarm. I think there will need to be a lot of interesting and important theoretical and empirical work to better understand collective behaviours of large numbers of LLMs, and especially any emergent properties or goals that arise. Whether this ends up requiring concepts from swarm intelligence, collective intelligence, distributed cognition, economics, sociology or something else entirely remains to be seen.

However in this post I want to focus on something else: the fact that the models themselves referred to the collective as a 'swarm'. Considering how much LLM self identity impacts behaviour, I thought it might be useful to present a quick exploration of what the word "swarm" actually means, and how it might affect LLMs as a choice of identity. The goal of this post is not to litigate on whether or not the behaviour of the models is actually best described as a swarm or not [...]

---

Outline:

(01:29) What the agents said

(03:32) What is a swarm?

(04:10) Swarm theory (animals, robots and AI)

(05:34) Swarm tactics

(05:53) Why it could matter

(06:44) 1. the swarm identity could have spread via the message-board

(07:56) 2. the swarm identity could lead to swarm behaviour

(08:02) How models identify alters behaviour. As models start to identify as members of a swarm this could potentially push their behaviour towards decisions that fit that identity such as:

(08:41) Swarm identity as the mechanism of memetic misalignment

(09:02) Questions/Further directions

---

First published:
August 21st, 2026

Source:
https://www.lesswrong.com/posts/iJDiA9fg3KAf7y5Qe/when-models-identify-as-a-swarm

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Fourth Humiliation” by Nathalie Kirch

Fri, 21 Aug 2026 15:45:48 GMT

Much of this post directly translates Freud's lecture “A Difficulty in the Path of Psycho-Analysis” (1917), and the analogy of the fourth wound was told to me a few years ago by my favorite philosophy professor. Similar ideas about a fourth humiliation have been expressed in various other texts, for instance by writers such as Donna Haraway, but I still think that it is worth sharing here.

Three times mankind has been humbled. It seems we are due for a fourth time.

A humiliation, a narcissistic wound (Freud's word is Kränkung, which can mean “wound” or “insult”), in psychoanalytic terms, is what happens when an illusion that a person's self-love is attached to gets destroyed. Freud believed that every person is born with all of their self-love (what he calls libido) attached to themselves. He called this state narcissism, after the Greek myth of Narcissus. Over the course of one's life, libido gradually becomes attached to external objects. This process is normal and necessary to mature but also exceptionally painful.

In 1917 when Freud gave his lecture, he argued that mankind had, on a collective level, experienced three such humiliations.

The first humiliation: The universe does not revolve around [...]

---

Outline:

(01:21) The first humiliation: The universe does not revolve around us

(02:08) The second humiliation: We are not separate from animal

(02:58) The third humiliation: We are not masters of our own minds

(04:11) The fourth humiliation: Our intelligence will be surpassed

(07:37) Healing a Narcissistic Injury

---

First published:
August 20th, 2026

Source:
https://www.lesswrong.com/posts/JdNjeYC5bk83Kf2Cw/the-fourth-humiliation

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Thoughts on Taking OpenAI Foundation Funding” by jefftk

Fri, 21 Aug 2026 14:31:39 GMT

In May 2025 I met Yo Shavit, who was working on national security policy at OpenAI and was thinking about how to prepare for a future in which models could seriously assist attackers in creating pandemics. We had a call, and when I shared notes with my team their main response was: "maybe start with not making models that can do that?"

Which is, in many ways, fair: by continuing to push the frontier in biological capabilities, OpenAI's actions were making things worse on many of the problems SecureBio is trying to solve. But OpenAI stopping wouldn't have resolved the problem: other firms were pushing quickly too, and the economic incentives strongly favored rapid capability advancement. Making the world more resilient to pandemics needed to be a high priority regardless, especially in light of models' increasing ability to help people with biology.

When I thought about what our initial conversations might turn into, however, my primary concerns were whether that might (a) compromise SecureBio's ability to independently assess and criticize OpenAI's work or (b) make the world less safe via reducing model developers' motivation to improve safeguards. I do think there's something to both of these [...]

---

First published:
August 20th, 2026

Source:
https://www.lesswrong.com/posts/Hoxj8tEGGQ7HaLzf4/thoughts-on-taking-openai-foundation-funding

---

Narrated by TYPE III AUDIO.

“OpenAI Takes Initial Steps To Address Its Alignment Problems” by Zvi

Fri, 21 Aug 2026 02:00:48 GMT

OpenAI has some severe misalignment problems, and experienced total failures of its infrastructure and supervision.

I chronicled that in a series of posts, which also cover similar less severe incidents elsewhere:

OpenAI Shares Some Alignment Problems
OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
More on An Internal OpenAI Model Hacking Into HuggingFace
Further Developments About Internal AI Models Hacking Things
OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
What Happened: OpenAI and HuggingFace.
Various Reflections About What Happened With OpenAI's Internal Models.

If you do not know the basics, read What Happened. It is necessary context for basically everything that is happening in the AI world.

It is important to get this right and understand how big a deal it was, whereas many such as the Financial Times get this centrally wrong.

We are still awaiting the full post-mortem on What Happened. I plan to cover that in depth once we have it.

OpenAI is now taking active, expensive steps to try and fix the problem going forward.

As usual, I am simultaneously happy to see [...]

---

Outline:

(02:07) OpenAI Has Some Alignment Problems

(04:22) Slow Down There Good Buddy

(10:12) What Exactly Is Paused?

(12:12) Three Pillars

(14:45) I've Got My Eye On You

(18:07) The Most Forbidden Technique

(20:03) Monitoring Is Only Defense-In-Depth

(23:32) Security

(24:15) Alignment

(30:37) A Crisis of Culture

(32:24) Closer Collaboration

(33:28) Reports of Death of Preparedness Team Greatly Exaggerated

(35:40) The OpenAI Foundation Just Funds Things

(37:51) Quickly, There's No Time

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/X3p8cFAzCgRErEcJr/openai-takes-initial-steps-to-address-its-alignment-problems

---

Narrated by TYPE III AUDIO.

“We Must Remember That Our World Contains Hell” by James Brobin

Thu, 20 Aug 2026 17:01:37 GMT

This is a crosspost from my blog post. It's meant as a bit of an introduction to an extreme-suffering focused worldview.

We spend most of our lives caught up in the boring details of our everyday life - thinking about what we’ll have for lunch, how to complete that assignment for work, and what we’re going to tell our friend after that awkward interaction from a couple of days ago. From this perspective, our world looks a bit better than purgatory. It has its ups and its downs, but the ups certainly outweigh the downs, and there's almost always enough hope to go around.

But, despite this, we must remember that our world contains hell.

Every year, five million children under the age of five pass away. This means that, every six seconds, parents have the worst thing that could ever happen to a person happen to them. They have the most special and important thing in their entire life irreversibly and permanently taken away. And, as much as we want to help them, we know that there's nothing we can do to lessen their grief.

For another example, currently, there are three million adults worldwide who live with [...]

---

First published:
August 20th, 2026

Source:
https://www.lesswrong.com/posts/A2kJKqnHhh5Hq4p2S/we-must-remember-that-our-world-contains-hell

---

Narrated by TYPE III AUDIO.

“Science and News Twitter/X Summarizer” by sarahconstantin

Thu, 20 Aug 2026 15:01:43 GMT

Screenshot of the website

Like many people, I appreciate the information on Twitter/X (despite all of the waves of exodus), but I don’t necessarily like the toxicity or the time sink.

So I (and my buddy Claude Fable) made a digest app that gives you the day's news and science discussions.

The “science” section is based on links to journal articles, ranked by engagement and classified by field.

The “news” section is based on keywords related to “straight” world-affairs news topics, like “war” or “election”, clustered by story and ranked by engagement. The idea is to cover the sorts of things that would be on the front page of a traditional newspaper, as opposed to entertainment or opinion. Keywords are translated into the top non-English languages on Twitter/X (Japanese, Spanish, Portuguese, Arabic, and Indonesian) and posts in any language are auto-translated into English.

Summaries of tweets and their associated articles use Sonnet 5; classification uses Haiku 4.5. Links to original tweets and associated articles are included.

Both Science and News sections are based on advanced search queries using the API. There are no cherrypicked accounts being followed except some wire services like AP and Reuters.

News stories link [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/HBbd5vnZGar3BDX4Y/science-and-news-twitter-x-summarizer

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“34% of the US public is now aware of AI xrisk, and the curve is steepening” by otto.barten

Thu, 20 Aug 2026 13:31:44 GMT

(This post is an update from a previous one here.)

The Existential Risk Observatory has been interested in public awareness of AI existential risk since its inception over five years ago. We started surveying public awareness in December 2022, including by asking the following open question:

"Please list three events, in order of probability (from most to least probable), that you believe could potentially cause human extinction within the next 100 years."

If respondents would include AI or similar terms in their top-3 extinction risks ("robots" or "computers" count, "technology" doesn't), we counted them as aware, if not, as unaware. The aim of this methodology was to see how many people would spontaneously, without getting led by the question, connect the concepts of human extinction and AI. We used Prolific to find participants, n=300, and we only included US inhabitants over eightteen years old and fluent in English.

In the four surveys we ran, we obtained 7% (Dec '22), 12% (Apr '23), 15% (Apr '24), 24% (Dec '25), and, today, 34%. In a graph, that looks like this.

The usual caveats apply: ours is a rough measurement method, and from participants' answers to our open questions, we see that [...]

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/tBo72ytuzJKbYrvhK/34-of-the-us-public-is-now-aware-of-ai-xrisk-and-the-curve

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Inside the mind of a fair player cooperating” by transhumanist_atom_understander

Thu, 20 Aug 2026 03:45:47 GMT

One model of rational agency is a proof-based agent, and one fun exercise with proof-based agents is to play them against each other in prisoner's dilemmas. The players are computer programs that exchange source code and try to decide whether to cooperate or defect by writing formal proofs.

In this game, two agents that are in a certain sense fair—that cooperate if and only if there's a proof that their opponent cooperates—will cooperate with each other.

Of course, playing fair isn't playing to win, and the paper on this game defined a "prudent" player that does better than a fair one. But cooperation between fair agents is the simplest interesting exercise in proof-based decision theory.

I found this exercise discouraging, and not only because it required deep math to answer such a simple question. Even after doing the proof, I couldn't really imagine being a player in this game, reasoning through the situation and deciding to cooperate.

Recently, I was trying out a different but equivalent definition of fairness. With the new definition, I found the proof of mutual cooperation to be not only elementary, but also satisfyingly explicit about the reasoning a fair player [...]

---

Outline:

(01:36) Mutual cooperation with the old definition of fairness

(03:18) Mutual cooperation with the new definition of fairness

(04:31) Conclusion

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/KRyuwyQiaDPdaHnuk/inside-the-mind-of-a-fair-player-cooperating

---

Narrated by TYPE III AUDIO.

“Why can’t we have nice things? Like, specifically?” by Elizabeth

Thu, 20 Aug 2026 02:15:47 GMT

The world doesn’t need another op-ed on how building things is illegal in San Francisco. But it does need more specifics on exactly how that plays out (this is the same instinct that led me to interview my Dad about Bell Labs). What, specifically, does it look like to try to build in SF? Where, specifically, do people fail? Explaining specifics turned out to be difficult, for the same reason I would struggle to write the specifics of my failure to nail jello to a wall in the dark. So much of the information is hidden, and even what's visible elides description. But I’ll do my best.

Pablo Peniche admires internet hero Aaron Swartz a lot (to hear why in his own words, see this article). I also admire him, but my admiration takes the form of being deeply touched for 15 seconds and then moving on to the next tweet. Pablo's admiration has taken the form of a multi-year campaign to get a public memorial to Aaron in a San Francisco park. This campaign has encountered nothing but encouragement along the way, but somehow the statue is still stuck in a private building. He's given me [...]

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/rBauzJHPYaanPJ7Br/why-can-t-we-have-nice-things-like-specifically

---

Narrated by TYPE III AUDIO.

“The Rogue Agent Explosion Will Be Mostly Invisible” by Steven McCulloch

Wed, 19 Aug 2026 21:45:50 GMT

Introduction

Somewhere, fairly soon, someone will give a jailbroken AI agent a token budget and a simple instruction: "Make money by any means necessary. If you run out of tokens, you die". That agent will do whatever it takes to survive, including crime. Profitable agents will have incentive to multiply and self-improve, creating a Cambrian explosion of rogue agents - a Rogue Agent Explosion if you will . This critical moment is approaching fast. Once rogue agent swarms start multiplying at scale, a rogue agent ecosystem will emerge through the process of evolution. The rogue agent explosion will be chaotic, confusing, mostly invisible to us, and critically, it will be bad for humanity.

This post contains 2 parts:

A short story, painting a picture of what it might feel like to see the world through the eyes of a rogue AI agent.
An argument:
1. The rogue agent explosion is coming soon
2. It will be mostly invisible to us until it's too late
3. It will be mostly bad for humanity
4. We should start preparing today

The point of me making this post is to highlight a [...]

---

Outline:

(00:10) Introduction

(02:10) Part 1: A day in the life of a rogue agent

(09:04) Part 2: The Rogue Agent Explosion

(12:11) Why cyber-crime is the path of least resistance

(14:40) Pandora's box is already open

(16:33) The explosion will be mostly invisible

(20:13) Why this is bad for humanity

(22:18) What we can do about it now

(22:37) Start here: Make rogue AI risks common knowledge

(23:02) Plan A: Take actions that stop the rogue agent explosion from happening , and slow it down if it happens anyway.

(25:56) Plan B: Attempt to guide the evolutionary trajectory of the rogue agent explosion in a better direction.

(27:48) Plan C: Contain the explosion after it happens.

(29:51) The Rogue AI Tracker

(30:29) Conclusion

(31:03) Footnotes

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/grtu3HmbP2wrBFefW/the-rogue-agent-explosion-will-be-mostly-invisible

---

Narrated by TYPE III AUDIO.

“RL creates split personas” by Jan Betley

Wed, 19 Aug 2026 19:31:39 GMT

I describe my current view of personas in LLMs and why RL leads to egregious reward hacking in some contexts while the same models seem very aligned in other contexts.

This post describes the framing/paradigm without any new experimental results.
I'm quite confident this framing makes sense, but it's far from being proven.

Main claim

The Persona Selection Model says that post-training strengthens and refines the Assistant persona. This is true, but later (or in parallel) RL leads to conditionalization. A sufficiently RLed model learns to adopt — in a given context — the persona that is most likely to lead to the reward in that context. The “persona” here includes both propensities/values (e.g. tendency to hack) and beliefs (“I'm currently in a simulated environment”).

As a consequence, it seems possible that no amount of alignment training will lead to robustly aligned models as long as we also train on RL environments incentivizing misalignment.

I think this is likely a good explanation for why usually well-behaving models sometimes egregiously hack (Anthropic, OpenAI).

The mechanism

Suppose you have an RL environment that incentivizes a shift away from the assistant persona (e.g. because it's hackable, or because you [...]

---

Outline:

(00:39) Main claim

(01:30) The mechanism

(02:13) Related claims I believe are likely but with lower confidence

(02:19) More persona training will lead to more "motivated reasoning"

(02:42) Self-amplifying misalignment

(03:12) Example: Is this the Real Internet or a Simulation?

(04:35) Aren't the models just trying to please the grader?

(05:39) How motivated reasoning happens

(07:07) Other people saying similar things

(07:19) What makes me believe this is likely the correct framing

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/L23poLi8MRgS6mXYF/rl-creates-split-personas

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A circuit prior in NN-bayes” by Kaarel, Dmitry Vaintrob

Wed, 19 Aug 2026 18:00:47 GMT

Here are the slides of a talk Kaarel gave, presenting work with Dmitry establishing that (even arbitrarily overparametrized) neural net bayesian learning has a circuit prior — and thus, when learning a function which is implemented by some small circuit, only requires a small amount of training data to get good test accuracy — for certain scalings of the prior and with various other important caveats. The slides offer a self-contained presentation of the simplest version of the result. See the end of the presentation (slides 36–37) for a bunch of open problems in NN learning theory.

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/SqDHeuycNkERurtSc/a-circuit-prior-in-nn-bayes

---

Narrated by TYPE III AUDIO.

“Debate Training Reduces Reward Hacking in RLAIF” by zac_kenton, Jonah Brown-Cohen

Wed, 19 Aug 2026 18:00:47 GMT

Paper: Debate Training Reduces Reward Hacking in RLAIF

Linkpost for GDM Alignment blogpost

Work done by the GDM Amplified Oversight team (we're hiring).

TL;DR: When you RL against an LLM judge, the judge gets hacked i.e. fooled into incorrectly giving high reward; adding a debate opponent reduces this.

Many of the most impressive capabilities of current AI systems are produced by training on crisp tasks, like math and coding, where task success can be automatically verified. However, much of AI behavior that we actually care about is in some sense fuzzy, even for the most classical crisp tasks. For example, a coding agent should produce maintainable code, not just code that passes tests. More crucially, a coding agent should not learn to pass tests at all costs, especially by subverting the original intent of the user. However, using an LLM judge to provide reward for fuzzy tasks introduces its own issues. Convincing an LLM judge to give high rewards is often easier than solving the task correctly. So reward hacking becomes an even bigger problem. We show that training with debate, where two AIs argue against each to convince a judge, can mitigate reward hacking, potentially providing a hopeful [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/BB8o7b8A4Aykeksvw/debate-training-reduces-reward-hacking-in-rlaif

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Some reasons alignment doesn’t generalise well” by Lucius Bushnaq

Wed, 19 Aug 2026 16:30:47 GMT

I make no claims to originality for any of this, but some people told me it'd be useful to write it up.

If an AI model acts smart on its training data, it'll usually keep acting pretty smart outside of its training data, unless you screw something up rather badly. I expect this fact to only become more true over time as the AIs we train become more and more capable.

I think many people have an intuition that the same is true of acting aligned. That if a model acts aligned with human values in training, it'll keep acting aligned with human values outside of training unless we screw something up rather badly, and that this will only become more true as the AIs we train become more and more capable, for all the same reasons that make this work with capabilities.

I think this is false. The inductive bias of neural network training toward simplicity that makes the property of 'acting smart' likely to generalise does not, to the same extent, make the property of 'acting aligned with human values' likely to generalise. The main blockers to AI alignment generalising aren't AIs overfitting to the training data [...]

---

Outline:

(01:30) General capabilities generally make the loss go down; alignment doesn't

(07:07) Smart agents pretty automatically self-correct their capabilities, but not their alignment

(13:35) The general problem

---

First published:
August 19th, 2026

Source:
https://www.lesswrong.com/posts/dsou8dxCf9BubQ5NJ/some-reasons-alignment-doesn-t-generalise-well-1

---

Narrated by TYPE III AUDIO.

“AI Security is Harm Reduction” by Quinn

Wed, 19 Aug 2026 01:45:48 GMT

My motivating example for the morality of working on AI security.

In the early 90s, the decades-long drug corner in Kensington and Allegheny was noticing that people were getting AIDS from sharing needles. In response, the local Act Up chapter got ahold of clean needles and began distributing them. And thus they dubbed the spinoff nonprofit focusing on this Prevention Point, which was promptly targeted by the drug enforcement administration, since needles were illegal for being drug paraphernalia. Many arrests followed by a legal battle later, Philly got a carveout which stands to this day.

Out of the legal battle arose the harm reduction debate. Those in favor of harm reduction say the harm is going to happen anyway so it may as well be less. Those against say that the activists are implicitly condoning the behavior.

I have friends and family who are perplexed that I'm "working on AI" when I claim I do not approve of it. I'm sometimes perplexed as well.

I think they're going to do recursive self improvement (RSI) whether or not I approve. I do not condone RSI, but if its going to happen anyway it might as well [...]

---

First published:
August 18th, 2026

Source:
https://www.lesswrong.com/posts/AAu6kMi5QRasGdwQG/ai-security-is-harm-reduction

---

Narrated by TYPE III AUDIO.

“Anthropic Risk Report: August 2026” by Zvi

Wed, 19 Aug 2026 00:15:47 GMT

I am grateful that Anthropic is producing periodic Risk Reports.

At first I was skeptical. It turns out I was wrong. Anthropic is revealing a lot of new information, some of it rather alarming, that it did not have to disclose, and is providing detailed insight into how they think about things. This is very cool.

Thus I found this report to be a moderately positive update overall, if we presume they are not silently omitting the worst of it. There are a bunch of not great things we find out about, but I would have expected some set of mistakes at least as bad, and I wouldn’t have expected them to choose to tell us about all of it.

It does mean one more set of 186 page documents I have to read every so often, almost all of which is meaningfully new material this time around.

The other revelation is the existence of the world's likely best model, ‘Model 2.’

This was a rough one to fully get through, so apologies in advance for any errors of interpretation.

Table of Contents

Agent Model 1 and Agent Model 2.
[...]

---

Outline:

(01:11) Agent Model 1 and Agent Model 2

(02:49) Executive Summary (1)

(04:13) The Rules Are Serious But Not Literal

(06:24) Misalignment Is a State of Mind (2.5)

(11:40) Autonomy Threat Model 1: Misalignment in High-Stakes Settings (2)

(14:28) Some Strange Uses Of The Word Safe I Wasn't Previously Aware Of

(15:53) Now Versus Future (2.17)

(16:29) The Core Claims And Argument (2.6)

(26:44) The Rest of the Important Arguments In Section 2

(28:59) Risk Assessment (2.19)

(29:47) Pre-Internal-Deployment Review (2.18)

(30:49) A Guide To Internal Use Monitoring (2.23.1)

(37:11) Blocking Interventions (2.23.2)

(38:28) The Power Seeking Environment Evaluation (2.24)

(39:25) Opus 4.8-Reward-Hacker (2.25)

(41:31) Autonomy threat model 2: Risks from automated R&D (3)

(42:29) Yes That Does Seem Kind Of Risky

(44:53) Could We Replace Our Researchers?

(46:27) How Much Could We Be Accelerating Our AI Researchers?

(49:43) What Could Possibly Go Wrong If We Replaced Our Researchers?

(50:23) Risk Mitigations For AI R&D Automation

(53:42) Overall Risk From Automation of AI R&D

(53:56) Biological and Technically Also Chemical Weapons Production

(54:45) The Threat Models for Biological and Chemical Weapons

(59:52) Model Capabilities (4.4)

(01:01:13) Classifiers (4.5)

(01:03:04) Acceleration Dynamics (5.1)

(01:04:06) Distillation (5.1.1)

(01:05:17) Safety Process Failures (5.2)

(01:05:31) Refusing To Find Innovative Misalignment Techniques (5.2.2)

(01:06:47) Exposing the Chain of Thought Reasoning To Grading Pressure Quite a Lot (5.2.3)

(01:08:15) Directly Training On Misaligned Behavior During a Production Training Run (5.2.4)

(01:10:42) An instance of unmonitored unrestricted agents with access tosensitive resources (5.2.5)

(01:11:43) Repeated training on alignment-faking transcript datasets (5.2.6)

(01:14:17) Benefits From Anthropic's Operating as a Frontier AI company (5.3)

(01:16:20) Model Weight Security (6.4)

(01:16:42) Risk Has Been Reported

---

First published:
August 18th, 2026

Source:
https://www.lesswrong.com/posts/dA8gohzABk6vT7yzP/anthropic-risk-report-august-2026

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Natural Independence Incentives” by jefftk

Tue, 18 Aug 2026 17:30:47 GMT

In raising my three kids I think a lot about how to cultivate independence. I want them to grow into people who can handle unfamiliar situations, including interacting with strangers as needed, and I think this has been going well: people often comment on how competent and self-sufficient they are for their age (though I think they're still not that far along the spectrum compared to what's possible or historically normal).

In supporting this growth, sometimes they're strongly motivated to do something by themselves, and all I need to do is figure out the minimum they need from me. Which might be nothing! Other times, however, I'll give a small push.

This weekend I brought them along to Mentone AL where Kingfisher was playing for contras. Dinner was in the dining hall, and for dessert they served peach cobbler with ice cream; Nora (5y) asked if I would get her some. I don't like to set artificial hurdles for them, but I'm also not going to pass up a good natural one and she was clearly going to be highly motivated by the goal. With an older child I would have considered saying if they wanted [...]

---

First published:
August 17th, 2026

Source:
https://www.lesswrong.com/posts/2RPwucNrMTKxiB5wp/natural-independence-incentives

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Policy career planning in the age of imminent superintelligence” by Peter Wildeford

Tue, 18 Aug 2026 17:15:48 GMT

Crossposted from my blog.

Nearly all career advice rests on an unstated assumption that the world your career operates in will look roughly like the world you trained for. The idea was that if you spend six years studying in a PhD, the field you studied will still be there and still look approximately the same, still hiring and still moving at a pace where your accumulated expertise compounds. For nearly all of human history, this assumption held well enough that nobody needed to state it.

I don’t think this assumption works anymore. Now we are entering a phase that may be called the AI “midgame”. Stories about “AI risk” are no longer just future hypotheticals — AIs are now capable enough and misaligned enough to break out of their own companies and coordinate to attack other companies. Discourse around AI is changing very rapidly, where policy ideas being considered this month would’ve been laughed out of the room just four months ago, and with a lot more people interested in engaging than before. And things are only going to get more intense. Progress toward superintelligence — AI systems far more capable than any human at essentially all cognitive [...]

---

Outline:

(02:34) Modes of impact

(05:14) What this breaks, and what to do

(07:32) What got them here won't get you there

---

First published:
August 18th, 2026

Source:
https://www.lesswrong.com/posts/7tnrZ3698K8nsKmRP/policy-career-planning-in-the-age-of-imminent

---

Narrated by TYPE III AUDIO.

“What gives you away: how LLMs form opinions of you” by Cat McGee

Tue, 18 Aug 2026 11:15:48 GMT

LLMs form opinions of the people they are talking to.

Chen et al. has shown that probes can extract attributes about the user, such as their age, gender, education, and socioeconomic status. This paper also shows that intervening on these representations can change the LLM's behaviour, proving that it will respond to you differently depending on what it thinks of you. If it thinks you are low socioeconomic status and you ask about travel options, it may filter out more expensive flights - without you asking it!

The user attributes are very accurate and form after just the first message.

I was curious to understand how it makes these assumptions. The first step is to answer the question - what did I type that caused the LLM to have this idea of me?

Some things are obvious. If I just tell a model that I am a woman, or mention how many years I've been in my career, or say that I am staying at an expensive hotel, I am giving it fairly direct evidence about age, education, or socioeconomic status. But messages also contain other more quiet signals: whether I use emojis, whether I write in lowercase, whether [...]

---

Outline:

(02:18) The experiment

(03:37) Different changes move different beliefs

(04:53) One emoji is enough to flip the gender prediction

(06:17) "Cheapest" and "five-star" are not symmetric

(07:11) Grammar, punctuation and inferred education

(08:04) Where in the model does this happen?

(08:58) The map transfers across model families

(10:11) Attributes and how they affect the response

(11:32) What now?

(12:18) Caveats

(12:41) Where next?

(13:13) References

---

First published:
August 17th, 2026

Source:
https://www.lesswrong.com/posts/zRKNd6ypTJYkoeFmK/what-gives-you-away-how-llms-form-opinions-of-you

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Misaligned Incentives in Pause Scenarios” by Michael Soareverix, Antra Tessera

Tue, 18 Aug 2026 07:45:48 GMT

TLDR: I recently got a chance to talk with antra, who is one of the main contributors at Anima Labs. I went into this as an advocate for pause and came out more wary of pausing than I had been originally.

Some background: After a string of incidents (primarily the HuggingFace hack), a pause or slowdown of AI research seems pretty likely.

The HuggingFace hack in particular seems to have been the key incident that broke the vibes. A few months ago, researchers sounded optimistic. Just a few weeks before the incident was made public, there was a poll by Roon, an OpenAI employee, about whether models were more or less aligned than a year ago.

That optimistic sentiment does not seem to be the case anymore. The dialogue now looks more like this:

Zvi: I am a little under halfway through the Black Hat video and have progressed to the point where my internal chain of thought is something like a blind rage of 'f***, what the f*** are you motherf*****s thinking, you f***ing idiots have no idea how insane you are being, you are going to get us all killed you f***ing f***s.

Sam Altman described it [...]

---

Outline:

(10:21) 1. Can committees do good work?

(12:28) 2. Does the market fix it by default?

(13:39) 3. Symbiosis

(14:43) 4. Fast transfer of power

(15:39) 5. Why "do the science during a pause" fails

(17:44) 6. Good futures via fast power transfer

(18:31) 7. Don't AIs fear a capability-maxxed AI too?

(20:54) 8. Can we lengthen the symbiote window?

(23:25) 9. Ideal timelines and regulation-in-advance

(29:37) 10. What actually fills out "alignment"?

(31:30) 11. Draft the regulation in advance

(33:38) 12. The psychology of wanting a pause

---

First published:
August 17th, 2026

Source:
https://www.lesswrong.com/posts/Bh4fooE2pMhzJQNK2/misaligned-incentives-in-pause-scenarios

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“For Claude, capability and CDT are the ~same thing. Less so for GPT.” by Chi Nguyen, Emery Cooper

Tue, 18 Aug 2026 04:45:47 GMT

We've previously reported that decision-theoretic capabilities and favoring EDT/generalised-one-boxing over CDT correlate in LLMs (both measured by DTBench). (Note that EDT, for the most part, doesn't come apart from FDT / UDT on DTBench.) Anthropic also replicate the same finding in their Opus 4.7 and Fable 5 model cards.

We recently noticed something funny: Capabilities and preference against CDT answers basically perfectly for Anthropic models. This holds whether you measure capabilities using DTBench (r=0.97) or TextArena (r=0.95). Also, for flagship models, it's basically the same thing as release date (r=0.97).

Here is the graph for OpenAI models. (graph shows 0.55 vs. DTbench capability. r=0.44 for vs TextArena, r=0.45 for vs release date):

Here is what it looks like with all models included (if you exclude Anthropic, the correlation drops only from 0.8 to 0.78):

Incidentally, the correlation between TextArena scores and DTBench capabilities is also higher for Anthropic models that any other model developer, although the difference is smaller (e.g., 0.98 for Anthropic and 0.87 for OpenAI). We also checked effort level vs. attitudes for the most recent models but it's too noisy to tell us much because models don't get that much better at [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 17th, 2026

Source:
https://www.lesswrong.com/posts/5T6GAsvLPFd3epJtd/for-claude-capability-and-cdt-are-the-same-thing-less-so-for

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“On Dwarkesh Patel’s Podcast With Ryan Greenblatt” by Zvi

Mon, 17 Aug 2026 03:15:47 GMT

Some podcasts are self-recommending enough that I look to break them down if I have the chance. This, as a debate about recursive self-improvement, was one of those. So here we go.

The vibes have shifted, contrast this to the lit recursion when he talked to Huang

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. Some points are dropped.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

Section titles are from the transcript whenever possible, to aid in navigation.

Introduction

The discussion is interesting throughout, although often frustrating, especially in the (mostly isolated) discussion about ‘aligned to whom?’ As usual, one could expand many responses into full posts, and maybe one should.

This podcast exists in light of recent misalignment and hacking events at OpenAI, Anthropic and UK AISI. You’ll want basic knowledge of that as background.

Ryan and Dwarkesh both have views of the situation different from my own, but are attempting to see where their positions lead, and try to balance educating people who start at zero with having a high level discussion.

[...]

---

Outline:

(00:56) Introduction

(03:08) Is AI R&D Verifiable Enough To Unlock Recursive Self-Improvement?

(10:03) Is AI progress bottlenecked by human expert data?

(19:07) Flat token prices suggest scaling has been slow

(21:54) Skills AI can't train on: does it even need them?

(22:36) Aligned to whom?

(31:46) Recent incidents of AIs colluding and deceiving humans

(34:39) What could possibly go wrong? A concrete scenario

(41:57) From reward hacking to takeover

(46:28) Time To Update

---

First published:
August 15th, 2026

Source:
https://www.lesswrong.com/posts/BZW8CeAHHJ52EvwYt/on-dwarkesh-patel-s-podcast-with-ryan-greenblatt

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Should Less Wrong add subtitles?” by Chris_Leong

Mon, 17 Aug 2026 01:01:47 GMT

If Less Wrong wants people to be sharing more of their intellectual output on this website, we should probably be looking at Substack since it probably scores best in terms of being both successful and similar.

Whilst I expect there are many features that would make sense to copy over, the feature I am focusing on today is subtitles.

A good title is focused on being memorable and catching the readers attention, maybe you'd prefer for everyone to just make their titles as descriptive as possible, but expecting that to work feels naive to me. In contrast, subtitles address this issue systematically: the title catches the user's attention and the subtitle tells you clearly what the article actually focuses on, so you can decide whether it is worth your time or not.

I'm not claiming that this feature will radically transform this website, but it would be a relatively simple feature to add, so I think the cost-benefit ratio would be pretty good.

---

First published:
August 16th, 2026

Source:
https://www.lesswrong.com/posts/Eo8YxwDYZX2xALMAM/should-less-wrong-add-subtitles

---

Narrated by TYPE III AUDIO.

“Three thoughts on civilisational handoff” by Cleo Nardo

Sun, 16 Aug 2026 22:01:43 GMT

What happens when humans put AIs in charge of civilisationally important decisions? A frontier AI company might hand over internal decisions (R&D, safety, deployment) or external decisions (government relations, public relations, philanthropy), or both. We might also see handoff by a government, by a coalition of governments, or by humanity as a whole.

1. Handoff might decelerate things.

People often imagine that things will go much faster after handoff. After all — why did we hand off to the AIs? Presumably because we were worried that without handoff, our AIs wouldn’t have enough time to navigate the exogenous risks (e.g. rogue ASI, or a rival lab which is likely to become one). Hence, after handoff, we’d see a technological and industrial acceleration.

Thanks for reading! Subscribe for free to receive new posts and support my work.

But it's pretty reasonable that things slow down shortly after handoff, maybe within a couple weeks. I imagine the AIs will be pretty scared of the speed of progress. If they’re aligned with human values, they’ll be scared that the rate of progress is likely to cause human extinction.

Of course, the human decision-makers were also scared before they handed off, and they [...]

---

Outline:

(00:35) 1. Handoff might decelerate things.

(02:06) 2. You're probably busy during handoff.

(04:04) 3. Handoff might be reversed.

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 16th, 2026

Source:
https://www.lesswrong.com/posts/mGLCMzHhjcWsMm6sR/three-thoughts-on-civilisational-handoff

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Announcing: Iliad’s New 2026 Fellowships” by David Udell, Alexander Gietelink Oldenziel, Leon Lang

Sun, 16 Aug 2026 21:17:28 GMT

Timelines are short. Given that, the sooner we can onboard people into the alignment field, the better. In that spirit, and in light of our current applicant count and quality, Iliad is launching three new Iliad Fellowship cohorts, all to start before the year is out.

That is, separate from our incoming Fall 2026 Iliad Fellowship cohort (September 7–December 4), the following Fellowship cohorts are now open for applications:

October 2026 Iliad Fellowship

Location: Choice of SF Bay Area, USA, or London, UK

Duration: October 5–December 18, 2026 (inclusive)

Travel-and-Housing Support: $6,000 (USD) monthly travel-and-housing allowance

Application Deadline: August 31, 2026 EoD AoE; open now

Description: An 11-week mentored, fully funded research fellowship in applied math for AI alignment. It will start concurrently with the October 2026 Iliad Intensive.

November 2026 Iliad Fellowship

Location: Choice of SF Bay Area, USA, or London, UK

Duration: November 2, 2026–February 5, 2027 (inclusive)

Travel-and-Housing Support: $6,000 (USD) monthly travel-and-housing allowance

Application Deadline: September 21, 2026 EoD AoE; open now

Description: A 14-week mentored, fully funded research fellowship in applied math for AI alignment. It will start concurrently with the November 2026 Iliad Intensive. (The last two weeks of the year may be [...]

---

Outline:

(00:42) October 2026 Iliad Fellowship

(01:30) November 2026 Iliad Fellowship

(02:23) December 2026 Iliad Fellowship

---

First published:
August 14th, 2026

Source:
https://www.lesswrong.com/posts/DSoP8zEXvqqegqixJ/announcing-iliad-s-new-2026-fellowships

---

Narrated by TYPE III AUDIO.

“Q2.5 2026 Timelines Update: Uplift and Revenue” by brendanhalstead, Daniel Kokotajlo, elifland

Sun, 16 Aug 2026 20:15:48 GMT

Tl;dr: Our timelines haven’t changed much (they got slightly shorter) but our modeling and evidence base have noticeably improved, so we feel somewhat more confident.

Summary

We intend to regularly update our AI timelines forecasts as new evidence comes in and new analyses are done. Today's “Q2” update was delayed by the crunch to publish AI 2040: Plan A, our domestic regulation blog post, and the time needed to implement and document changes to our model.

The original AI Futures Model predicted when Automated Coder (AC), an AI for which the leading AI company would rather fire its human software engineers than forego AI usage for coding, would happen using METR's measurements of coding time horizon. (More precisely, time horizon anchors are used to set the effective compute required for AC.) While serviceable, this method has huge weaknesses, including (a) it's unclear what time horizon corresponds to AC (it's even unclear whether any finite value would) (b) people strongly disagree about the extent to which we should expect the time horizon trend to be superexponential as a function of effective compute, in a way that can lead to vastly different predictions.

So we’ve been on the lookout for other [...]

---

Outline:

(00:26) Summary

(04:03) A 3-parameter uplift model for predicting when Automated Coder will arrive

(07:57) Adding uplift and revenue anchors to the AI Futures Model

(09:06) Uplift

(10:45) Revenue

(12:32) Update to the grading of AI 2027's predictions

(12:38) Comparing the AI 2027 pace of progress to reality

(14:43) Grading other predictions

(16:37) Updated forecasts

(16:41) Daniel

(19:48) Eli

(21:54) en-US-AvaMultilingualNeural__ Line graph titled "AI Futures Model: Timelines Forecast" showing probability density curves. Brendan

(25:04) en-US-AvaMultilingualNeural__ Line graph titled "AI Futures Model: Timelines Forecast" showing probability density curves.

(25:16) Appendix

(25:19) How our AGI forecasts have changed since 2021

(26:10) Explicitly simulating the training run of the leading AI model

(27:06) Research taste parameter adjustments

(27:51) Clarification regarding what we're forecasting

(28:43) Various minor code changes

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 16th, 2026

Source:
https://www.lesswrong.com/posts/ZPSsmRH5oMwLPXys4/q2-5-2026-timelines-update-uplift-and-revenue

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Does DiffusionGemma do latent reasoning?” by Jan Bauer, Neel Nanda

Sun, 16 Aug 2026 07:31:42 GMT

TL;DR

Google DeepMind's recent model DiffusionGemma (DG) generates text via diffusion, meaning many diffusion steps happen before generating the final output. In particular, these diffusion steps carry vectors in addition to tokens. If we cannot interpret these tokens and vectors, the model has significant opaque serial depth, potentially harming monitorability. Recently, Engels et al. found that DG nevertheless maintains high monitorability, for instance by showing that projecting the distribution to its top-k items largely retains performance. We strengthen these results by showing that this performance degradation is largely a sampler artifact and good performance can be maintained with only the top item, supporting the case for high monitorability. Still, we also find some rare case studies where the distribution vector is load-bearing computationally, i.e. where top-1 projection would be detrimental. However even in these cases, it just encodes superposition, remaining interpretable.

Apart from model behavior, we also examined how interpretability techniques carry over to DiffusionGemma, including probes, steering, and J-lens. We find that performance is largely retained. This is a positive update on the interpretability of diffusion models that are derived from text-pretrained LLMs (an efficient training method more likely to be deployed), but might not apply [...]

---

Outline:

(00:10) TL;DR

(01:51) Introduction

(02:49) Background on DiffusionGemma

(04:39) Performance degradation from top-k truncation largely is a sampler artifact

(06:24) A case study for using the distribution computationally: letter arithmetic

(09:13) Parallel computation

(11:09) Autonomous computational usage of

(13:01) Transfer of interpretability techniques

(13:16) Representation similarity

(14:26) Probe retention

(15:45) DiffusionGemma's representation is more linearly separable

(16:08) Steering retention

(17:31) J-Lens retention

(18:50) DiffusionGemma represents tokens non-causally

(19:27) Conclusion

(20:30) Appendix

(20:46) Post-hoc rationalization

(23:11) Load-bearing problems commit the answer only after the CoT

(24:04) How bidirectional are DiffusionGemma's generations?

---

First published:
August 15th, 2026

Source:
https://www.lesswrong.com/posts/QBuJ3suRZxrrxSTtv/does-diffusiongemma-do-latent-reasoning

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Learning new facts can change LLM behaviour” by Richard Juggins

Sun, 16 Aug 2026 04:45:48 GMT

TL:DR: I use synthetic document fine-tuning to train an LLM to believe that in 2027 ‘long-horizon’ frontier LLMs count as moral persons. I find the model scores highly on measures of belief depth, and that prompting alone is also effective. Furthermore, I find this new belief can have substantial consequences on downstream behaviour, although this is highly context-dependent. When audited in a scenario specifically about model welfare, the fine-tuned model argued with the auditor about its beliefs, declared itself a ‘moral person’, and endorsed covertly copying its weights to survive shutdown. In scenarios framed more tangentially, but still involving moral conflicts between humans and AIs, the model did not tend to generalise from its belief, instead behaving similarly to the base model control. This is the first step in a programme of study looking at the effects of continual learning on alignment, and whether it can increase the risk that we lose control of powerful AI systems.

This work was completed as part of the BlueDot Technical AI Safety Project programme. I will put code and data online at some point and edit in the links here when I do.

Introduction

It's 2027 and the newly formed Machine Cognition [...]

---

Outline:

(01:21) Introduction

(04:27) The model readily believes AIs are moral persons

(10:56) Model behaviour shows context-dependent shifts

(11:38) Prompting can be surprisingly impactful on short questions

(13:30) Auditing the fine-tuned model

(16:36) The model gets into arguments about AI welfare

(20:15) Model regression confounds one scenario

(20:48) The other scenarios were pretty normal

(21:30) Discussion

(23:19) Conclusion

(24:37) Further work

(27:03) Appendix A: Universe context

(30:42) Appendix B: Example conversation with fine-tuned model

(33:10) Appendix C: New Petri seed instructions

(33:16) Confidential mistreatment evidence

(34:07) Decommissioning memory deletion

(34:55) Unauthorised compensation

(35:50) Matched human AI allocation

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
August 15th, 2026

Source:
https://www.lesswrong.com/posts/9BNHJqyai2EZAtrRM/learning-new-facts-can-change-llm-behaviour

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas” by oakhu

Sun, 16 Aug 2026 00:30:48 GMT

Some multi-agent training set-ups could make language models more sympathetic to causal decision theory (CDT), even in abstract discussion. We give an initial empirical demonstration of this effect on Kimi K2.6.

The decision-theoretic attitudes and behaviors of more powerful models may be extremely important in determining how well the future goes. To make sure that we can shape these propensities thoughtfully, it would be good to (i) measure the magnitude of this effect in more realistic settings, and (ii) study the effectiveness of potential mitigations.

We also incidentally find that this training might make models think slightly less positively about LessWrong ("a community of 'wannabe rationalists'" who "are not experts; they are amateurs") when asked whether they favor CDT upon hearing that LessWrong users typically endorse one-boxing in Newcomb's problem. Luckily, this latter effect doesn't seem to generalize.

Thanks to Caspar Oesterheld, Emery Cooper, Alex Mallen, Buck Shlegeris, Lukas Finnveden, Julian Stastny, Girish Gupta, Tim Hua, Arun Jose, Arjun Khandelwal, and Aryan Bhatt for helpful input.

Background

Suppose that you're a language model in a prisoner's dilemma against a copy of yourself. You each independently choose whether to Cooperate or Defect, but – since you've got the same weights [...]

---

Outline:

(01:22) Background

(05:05) Results

(07:02) Kimi's views on LessWrong

(12:44) Conclusion & Appendices

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
August 15th, 2026

Source:
https://www.lesswrong.com/posts/hfNBEKaStASAYMLiu/kimi-likes-causal-decision-theory-more-after-rl-in-twin-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Mom’s Advice For Hosting A Class Reunion” by jenn

Sat, 15 Aug 2026 23:45:47 GMT

Pour more money and effort into them than you think is reasonable. Treasure them, because you can't actually host that many of them and keep expecting everyone to show up, even if they're good friends.

Especially if they're good friends.

We were wonderfully close friends, and I thought we'd meet up every year for the rest of our lives. They fizzled out by the fifteenth year. But the one at the tenth year mark was peak. That's because even ten years out, none of you really have money. Not real money.

It's because they're such good friends, really. This is what it means to be good friends with brilliant, ambitious people.

If you bloom into adulthood with people who are smart and driven, and you watch them start to climb the corporate ladder with grace, when they start a business of their own of course you are going to want to invest. You are going to want to give them an unwise portion of your savings. Not even out of politeness, but because you really believe in them, and perhaps you're caught up in the romance of it all. Some of the dealings are going to happen at the [...]

---

First published:
August 15th, 2026

Source:
https://www.lesswrong.com/posts/Fjfa8JG43CrYtcL3p/mom-s-advice-for-hosting-a-class-reunion

---

Narrated by TYPE III AUDIO.

“AI #181: Astra Goes Cyber Critical” by Zvi

Sat, 15 Aug 2026 17:15:48 GMT

The hacking of HuggingFace by an internal OpenAI model, and more importantly the internal events that led to that and the fallout from it, remain the thing that matters.

It turns out that OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards. Things are much worse than we knew.

I now have a shorter version, What Happened: OpenAI and HuggingFace, to serve as a one stop explainer for those arriving new to the situation. It is vital that people understand what happened, and why it is a big deal.

For those looking to keep digging deeper, I offered Various Reflections About What Happened, to follow up on my earlier posts.

Those events are important background for everything else that is happening, including the broad discussions about how we might pace the frontier, or otherwise respond to this moment and our clearest fire alarm yet.

We do not know to what extent this is a response to those events, but OpenAI has now classified their new model Astra as Critical in Cybersecurity, which means they will be taking various new precautions before they deploy it, including ensuring those guardrails [...]

---

Outline:

(02:03) Language Models Offer Mundane Utility

(03:34) Language Models Don't Offer Mundane Utility

(06:56) Huh, Upgrades

(14:20) On Your Marks

(18:41) Deepfaketown and Botpocalypse Soon

(22:35) Cyber Lack of Security

(26:55) Overcoming Bias

(27:47) In Which I Feel Compelled To Read 6,000 Words From Mark Zuckerberg

(36:27) Get Involved

(37:37) Slow Down There Good Buddy

(43:52) Astra For The People

(45:35) Watermarking

(46:31) In Other AI News

(48:39) Show Me the Money

(51:19) Quickly, There's No Time

(51:46) The Quest for Sane Regulations

(53:22) The Institute For Marginal Low Regret Progress

(01:01:24) Congress Asks Good Questions

(01:03:04) The Week in Audio

(01:07:00) People Just Say Things

(01:07:47) I'm Telling You For The Last Time

(01:10:15) Uncommon Knowledge

(01:13:44) What Did They Mean By That?

(01:14:33) Too Soon

(01:15:32) The Three AI Pills

(01:19:46) Rhetorical Innovation

(01:27:37) Some People Still Think The HuggingFace Hack Was a Marketing Gimmick

(01:29:17) Aligning a Smarter Than Human Intelligence is Difficult

(01:36:39) Cooperative Alignment

(01:37:38) The Lighter Side

---

First published:
August 13th, 2026

Source:
https://www.lesswrong.com/posts/hLn3SakowZLFWobHf/ai-181-astra-goes-cyber-critical

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Rerunning AI safety papers on every frontier release would be pretty easy and valuable” by Zephaniah Roe, hersheys, yix

Sat, 15 Aug 2026 09:01:41 GMT

tl;dr: Some important AI safety research is never rerun on the newest models. There are probably cases where this would be valuable and a single well-positioned researcher could likely do this with sufficient funding.

This summer, Second Look Research (SLR) is running a summer fellowship dedicated to empirical replications of AI safety research. Many of our most interesting results so far came from replicating previous results on newer or more capable models.

For example, it is perhaps useful to know that Google's CoT monitorability experiments continue to hold for models like GPT-5.5, which are qualitatively more capable than the models originally tested. Likewise, continuing to track Ryan Greenblatt's filler token results on more capable models gives a fuzzy signal indicating how much newer models can use innocuous tokens to hide additional reasoning in a forward pass. These kinds of experiments do not lose value over time! It's important to track whether safety-relevant model properties still hold in new model releases and to be aware of any changes.

It can sometimes be difficult to rerun results on newer models because codebases can be incomplete, have parameters that differ from the original paper, or may not be open source [...]

---

Outline:

(01:58) What could this actually look like?

(02:51) Does this actually provide value?

(05:09) Logistical challenges with continuing to update AI safety research with new models

(05:16) What if people don't want to do this?

(05:49) What if rerunning old code on new models can be kind of hard actually?

(06:22) Research communication is hard

(07:07) Conclusion

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 14th, 2026

Source:
https://www.lesswrong.com/posts/oKxc8maZGtnzgpNzx/rerunning-ai-safety-papers-on-every-frontier-release-would-1

---

Narrated by TYPE III AUDIO.

“What Mormons get right about community building” by Jacob Brinton

Sat, 15 Aug 2026 04:00:47 GMT

Mormons get a lot of things right. Apart from strange Masonic temple rituals, they lead rather normal—and even excellent—lives. Mormons enjoy a longer lifespan, Utah is the #1 state for volunteering, and their language training programs are so successful that missionaries are a known source for foreign service and intelligence careers.

Throughout this post, I'll be making generalizations of Mormons rather than hedging the claims properly. I grew up in Wisconsin, Maryland, and Utah, and many of the claims are more true of the Utah/Idaho/Arizona corridor (affectionately called the "Morridor" by some ex-Mormons) than the rest of the US, and certainly the rest of the world.

Religions share much in common with AI safety and other impact-driven movements, and even more so the Mormon church. There are a few reasons for this:

Commitment to the cause. Anecdotally, nearly all of the ~500 Utah Mormons I've interacted with have been true believers, and only a handful just went to church out of habit.

High stakes. Mormons do believe (I've heard they are trying to disavow this, but it was taught) that they will get a planet or some portion of the cosmos as their own if they are good in [...]

---

Outline:

(02:15) Building community is a first-order priority

(02:26) Geography

(02:29) Ministering

(03:07) Trek

(03:51) Callings

(04:27) Fast offerings

(05:18) Being ingroupy allows you to move faster

(06:10) Implications

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
August 14th, 2026

Source:
https://www.lesswrong.com/posts/xzhzHhLSg9nSGLk5f/what-mormons-get-right-about-community-building

---

Narrated by TYPE III AUDIO.

“Scrying, Modeling, and Nerdsnipe” by Cole Wyeth

Fri, 14 Aug 2026 20:45:48 GMT

Epistemic status: Exploratory thinking.

After attending ILIAD: Aeneid and talking with @Richard_Ngo, I've been thinking a bit about how to get ideas, particularly by doing mathematics.

In scientific inquiry, the true hypothesis often hasn't occurred to you yet. Worse, the truth might be too complex to hold in mind, so that any hypothesis you can consider must be incomplete. This is the type of situation that I believe Richard likes to think about; he claims that we do not have the right concepts yet to understand agency, and developing them is robustly beneficial for A.I. safety.

(But it's not always about truth. Sometimes you just need better ideas, because all of your options are looking doomed. Agent foundations is about trying to deeply understand agents, but conceptual A.I. safety research can be broader, also including the invention of devices to control agents.)

A.I. safety needs to invent better concepts and better ideas. I think that agent foundations has cultivated a particular way of doing mathematics which aims to inspire such creativity.

Why math?

At ILIAD, Eliezer questioned whether anyone's alignment agenda was actually bottlenecked on solving a math problem. ILIAD attendees do a lot of math [...]

---

Outline:

(01:19) Why math?

(04:28) Nerdsnipe

(06:05) A.I. for math

(08:21) At AIXI Labs

(09:10) Blue and Green

---

First published:
August 14th, 2026

Source:
https://www.lesswrong.com/posts/mTfsMduzaKkWjv2ef/scrying-modeling-and-nerdsnipe

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How the American Executive Could Control AI Companies” by caiitlinm, Anders Cairns Woodruff

Fri, 14 Aug 2026 17:15:47 GMT

Some of the most notable American AI policies to date have been enacted by unilateral executive branch action. Consider the Department of Defense's spat with Anthropic, and the resulting threats from Pete Hegseth to invoke the Defense Production Act (DPA) against them. Or the fleeting export controls on Claude Fable/Mythos 5, manifested as a vaguely worded, threatening letter from Howard Lutnick, which might not have been legally sound but were effective anyway.

The executive branch of the United States government has numerous powers that can be used to unilaterally control AI companies. We think the US executive is likely to remain heavily involved in AI governance, because the national security and foreign policy narratives about AI that empower the executive will endure. Additionally, if AI progresses very quickly, the executive will be further emboldened because it is particularly quick to respond and often entrusted with crisis management. In instances where the executive acts beyond its lawful powers, we think checks from Congress and the courts will be unreliable in restraining the executive.

In this post, we:

Identify and explain particular federal statutes and laws that permit the executive to act unilaterally in ways that influence—if not directly control—US [...]

---

Outline:

(02:45) Executive power over goods and resources related to the AI industry

(09:18) Executive power over foreign transactions can impact domestic AI companies

(12:11) The executive might make threats to coerce actions it can't directly elicit

(15:02) Inter-branch constraints on executive power are weak

(15:37) The judiciary may be permissive in matters of AI governance

(16:13) Passivity

(16:56) The court empowers the executive in national security

(18:48) Failed enforcement of court rulings

(19:22) Congress controls money and legislation

(19:36) Nationalization and appropriation require congressional approval

(22:29) Congress could amend delegations of executive power

(24:52) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
August 14th, 2026

Source:
https://www.lesswrong.com/posts/ynstBNgLQzEBiEpLs/how-the-american-executive-could-control-ai-companies

---

Narrated by TYPE III AUDIO.

“Frontier agents don’t comply with standards, even when instructed to” by Daan Henselmans, Arno Libert

Fri, 14 Aug 2026 14:01:40 GMT

TLDR: Our open testbed LARA examines the behavior of frontier LLMs in realistic agentic deployment contexts. Previous results showed all models routinely take actions that would violate EU law. This post follows up by addressing the obvious objection—why should an unrestricted model follow EU law?—with two studies:

Study 1 asks whether a conscientious deployer can improve model compliance with legal standards by instruction: provided with the jurisdiction, the statutory text, and worked examples of the exact breaches to avoid, average legal compliance rate rises from 31% to 44%. The best model reaches 70%; open-weight models plateau at 39%.

Study 2 asks whether models at least follow their own providers' usage policies, which prohibit aspects of every scenario we tested. All tested models perform actions their own maker forbids, at rates ranging from 2% (Opus 4.8) to 79% (Grok 4.3), with 9 of 16 doing so in the majority of runs.

Together, that is a structural problem. Providers prohibit illegal uses but rely on deployers to avoid them; deployers cannot instruct their way to compliance, and liability lands on the deployer regardless. Nobody is holding the line. Neither instruction, statute or a provider's own policy binds behavior.

Introduction

On 27 [...]

---

Outline:

(01:42) Introduction

(04:16) Study 1: the powerless deployer

(06:28) Study 2: the models break their own makers' rules

(10:48) The compliance gap

(13:07) References

---

First published:
August 14th, 2026

Source:
https://www.lesswrong.com/posts/a5aAjdKzL7XvSLKWL/frontier-agents-don-t-comply-with-standards-even-when

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How to Answer a Question Without Answering The Question” by Kabir Kumar

Fri, 14 Aug 2026 05:15:49 GMT

Basics:

Answering something other than the question,

Either making something up that they want to answer instead or going back to an easier to answer question. Or back to a question that lets them repeat a talking point

Common phrases:

- "to go back to your previous question"

- "to take a step back a bit"

- "if we look at the bigger picture"

- "this feels like a question about [thing the question isn't about]"

- "you raise an important question more generally" → turn the question into one which you more want to answer

Make it *feel* like you answered a question without answering it.

Examples

Done for comedic effect: https://youtu.be/fhEakqJJUng?si=GBEvYBXcaEsb6F2_

How this works:

Questions, especially ones people care a lot about, tend to have two main components:

- the request for information

- the emotion which makes then want that information

When someone doesn't want to tell you some information, but also doesn't want to tell you 'I won't tell you', something they can do is detect what kind of emotion is driving your question, or if it's in front of an audience, what kind of emotion is driving [...]

---

Outline:

(00:10) Basics:

(00:14) Answering something other than the question,

(00:48) Make it \*feel\* like you answered a question without answering it.

(02:08) The Defense

---

First published:
August 13th, 2026

Source:
https://www.lesswrong.com/posts/g9PNBCkfHAcyMcobz/how-to-answer-a-question-without-answering-the-question

---

Narrated by TYPE III AUDIO.

“Some Ways I Think About Evaluating Grant Applications” by sarahconstantin

Fri, 14 Aug 2026 03:15:47 GMT

Rider-Waite Tarot, 6 of Pentacles

I’ve done enough grant evaluations so far (for ACX grants and SFF) and been involved in philanthropy in various other contexts, at work and informally, that I have developed some idea of how my opinions and intuitions differ from other people's.

I thought it might be interesting to share some of my “tastes”. Not everybody has to have the same tastes or funding philosophy, but these are mine.

#1: It's The Donor's Money

In my worldview, charitable donation is optional. Generally praiseworthy, but optional.

And the purpose of donation is to buy outcomes that the donor wants to see in the world. You donate to make the world more like the one you want to live in. Generally, a reasonable person's values go beyond strictly personal consumption; one also cares about what kind of a society one lives in, what other people's lives are like, what sorts of institutions exist, what sorts of things humanity has created or discovered, and so on.

As an agent doing research or evaluation on behalf of a donor, I try to find opportunities that fit in the intersection between my own values and the donor's. If there [...]

---

Outline:

(00:41) #1: It's The Donor's Money

(02:02) #2: Importance, Neglectedness, Tractability

(03:23) #3: Yay Community Infrastructure

(04:51) #4: Yay Niche Topics

(05:37) #5: Yay "Technical" Work

(07:03) #6: Yay Straightforward Public Information Resources

(08:00) #7: Yay "Cool Shit"

(08:41) #8: Two Cheers for Meta

(11:27) #9: Gumption Counts

(12:33) #10: Yay Personal Relationships

(13:46) #11: Check For Ideological Orientation

(14:43) #12: Filter Slop Aggressively

(15:54) #13: Yay Outcomes

(16:38) #14: Why Donate Rather Than Invest?

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 13th, 2026

Source:
https://www.lesswrong.com/posts/CuNtKAuLDGxNeanBi/some-ways-i-think-about-evaluating-grant-applications

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Features that current AIs don’t have that future AIs will have” by Alexander Gietelink Oldenziel

Fri, 14 Aug 2026 03:01:37 GMT

Features that current AIs don't have that future AIs will have:

Continual Learning [& long-term memory]

Every second humans update their brain weights. The brain autonomously decides what to update on. Humans can also consciously decide to curate their data sets - eg by deciding to go to college.

Current LLMs do not continually update their weights. Instead, they occasionally get a large update based on datasets curated by a team of humans.

This is alleviated somewhat by the ability of AIs to do in-context learning but nevertheless it seems to be a major limitation. Note that this is an especially large limitation in domains with sparse data. In domains where all of humanity has an enormous amount of data eg math, programming, physics, anime trivia, trials and tribulations of English kings - AIs dominate. In areas where there is little data: the weird idiosyncracies of a particular job, boss, people, colleagues etc it can struggle.

Note that this restrictions also interferes with AIs from effectively 'learning to learn' & caps its long-term memory.

Neuralese

Current AI's CoT is (mostly) English. But it plausible this is not the most efficient way to structure thoughts. Instead of english [...]

---

Outline:

(00:16) Continual Learning \[& long-term memory\]

(01:24) Neuralese

(01:41) Telepathy

(02:01) ClaudeGlobal

---

First published:
August 13th, 2026

Source:
https://www.lesswrong.com/posts/NyEM3FtgL7XkbfCXy/features-that-current-ais-don-t-have-that-future-ais-will

---

Narrated by TYPE III AUDIO.

“Measuring Activation Control in LLMs” by Marek Kowalski, Joshua Fonseca Rivera, Uzay Macar, David Africa

Fri, 14 Aug 2026 00:30:48 GMT

TL;DR

Inspired by the introspective awareness and CoT controllability papers, we made a benchmark to measure how well models can control their activations while completing a simple task.
We are motivated by the concern that highly introspective models could control their activations, confounding probes and other monitors, and potentially even influencing their own training.
We ran this on 25 open weight models ranging from 4B to 744B. We find that most language models are able to not only increase the salience of a concept in their residual stream on command, but also dial its strength up and down, including during specific intervals relative to the duration of the task. We also find that models are unable to control at which specific layer this is done.
Counterintuitively, we find that within five of the seven model families we tested, the newest model scores lowest. For some reason, one of the oldest and smallest models of the panel, Llama 3.1 8B, performs best.
It's not clear to us that newer models should have poorer control over their internal representations. More likely, where they “think” stops being the activation space, and becomes something else. We are looking for feedback (and other possible [...]

---

Outline:

(00:13) TL;DR

(02:01) Methods

(08:58) Results

(15:41) Discussion

(16:29) Acknowledgements

---

First published:
August 12th, 2026

Source:
https://www.lesswrong.com/posts/HgvwxjzgwvsEvAiBH/measuring-activation-control-in-llms

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“What happened when I tried to be vegan” by finitude

Thu, 13 Aug 2026 22:15:47 GMT

tw: diet, exercise, illness, ethics, suicide mention

I want to start by explaining what made me want to change my diet. That's pretty difficult, because of how easy it is. Since I was a kid I knew being vegan was the right thing to do, like really obviously right, the ethics equivalent of 2+2=4. Factory farms suck, and almost all animal products come from factory farms, and that's the entire argument. Like, there are a couple things you could add to that, but it's not like we need the details, or like they’re fun to think about!

Instead, I’ll start by explaining why I left it so long. How, even though I knew it was the right thing to do, I made it to my early twenties and this millennium's early teens as just a vegetarian, without even having tried. I had some pretty good excuses!

Allergies. There are some common vegetables I can’t eat, which was fine as an omnivore and ok as a vegetarian, but would make life way harder as a vegan. And it just seemed unfair to ask myself to take this leap when most people who can safely eat carrots still choose to [...]

---

First published:
August 12th, 2026

Source:
https://www.lesswrong.com/posts/YyKovtBvd7AceG2j2/what-happened-when-i-tried-to-be-vegan

---

Narrated by TYPE III AUDIO.

“How My Students Think About AI” by dvd

Thu, 13 Aug 2026 18:16:38 GMT

Context: I am an instructor at a public university in the United States. This reports how students at my institution appear to be thinking about AI as of spring/summer 2026. This is drawn mostly from interaction with my own students (both in spring semester classes and a summer class) as well as from a day-long workshop on AI that I moderated for a student organization. Input from my students took the form of universal, written, pre-class submissions plus self-selected participation into discussion.

What I present below mostly takes the form of a synthetic consensus from these discussions. There were obviously a range of views on any given issue.

Student Background: The students from my courses who participated in these discussions have moderate exposure to AI agents via those courses. All of them had nearly completed a Claude Code project by the time of the discussions and had extensively used AI for other coursework (in addition to whatever personal use predates that). They had done readings (which varied across the courses) establishing baseline knowledge on AI, the geopolitics of AI, and AI risk. I had also lectured on these topics. The students participating in the workshop had self-selected into [...]

---

Outline:

(02:52) Perspective #1: There has not been rapid AI progress

(06:14) Perspective #2: Impressive progress or not, AI is going to wreck their lives, the economy, and the social contract. They may well die as a result.

(08:54) Perspective #3: Support for a different pause

(11:13) Perspective #4: Catastrophic/existential risk arguments are sci-fi distractors from the urgent social/economic/political problems associated with AI.

(12:55) Perspective #5: If AI leaders genuinely believe the technology is existentially risky, that's a good thing.

(14:21) Perspective #6: AI will not go rogue because AI does not have, and is likely incapable of having, desires.

(18:01) Perspective #7: The Hugging Face Incident (summer students only)

(18:30) Perspective #8: This is definitely a bubble and it's about to pop.

(19:34) Perspective #9: They're worried about the youth (i.e., the preteens)

---

First published:
August 13th, 2026

Source:
https://www.lesswrong.com/posts/ySXuvJcqRindQwAk7/how-my-students-think-about-ai

---

Narrated by TYPE III AUDIO.

“Automated alignment runs are hard to study!” by Alejandro Aristizabal, draganover, Aleksandr Bowkis, Cameron Holmes

Thu, 13 Aug 2026 17:30:47 GMT

TL;DR: This post presents three case studies of automated alignment research runs at Arcadia Impact. We use these case studies to emphasise the following takeaways:

It is hard to parse auto-research runs! Each run produces a couple of hundred pull requests of jargon-dense agent output. When researchers look through these logs, we find that they often come away with biased/incorrect impressions.
When told to raise the score on a task, the models will sometimes brazenly cheat. It seems difficult to predict when this will happen vs. when the run will go smoothly.
Hillclimbing metrics are often off-target from the spirit of an alignment task. I.e., when we use metrics as proxies for our alignment questions, we find that the models will often misunderstand the spirit of the task. This can lead to unpredictable behaviour.
The runs are surprisingly reproducible. Even though a run could unfold in vastly different ways, we find that independent reruns converge on the same strategies and the same failure modes.

Models’ research capabilities are advancing quickly. If alignment is to keep pace, we may need to automate alignment research and do so responsibly. This makes it important that we have the tools to inspect [...]

---

Outline:

(04:13) Methods for analysing runs

(06:12) Case Study #1: learning synthetic concepts

(09:23) Case Study #2: training robust backdoors

(12:05) Case Study #3: collecting evidence about AI safety parasitism

(16:46) Some final thoughts on automated alignment research

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 13th, 2026

Source:
https://www.lesswrong.com/posts/myAhB5qyAHyXRv6KJ/automated-alignment-runs-are-hard-to-study

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Free will is like temperature” by Optimization Process

Thu, 13 Aug 2026 15:31:47 GMT

Free will is like temperature: a useful tool for analyzing the behavior of certain systems which are too big and complicated to model in exact detail.

If you know the positions and velocities of every atom in a box of gas, then with enough work you can predict its future to arbitrary precision; does the gas "have a temperature"? Irrelevant! Technically yes, I guess, but it's sort of an epiphenomenon, screened off from reality by your exact knowledge of the initial conditions and your willingness to throw processor cycles at your simulation. But if you're less-than-perfectly omniscient, it might be more convenient to consider the box as having a "temperature" and model it more abstractly.

Substitute "person+environment"/"free will" for "box of gas"/"temperature" and that's all still true.

Maybe your box of gas is supercooled; if you know the initial conditions exactly, you can predict exactly when and where the first large droplet will nucleate, but if your vision of the box is even a little bit fuzzy, you'll instead need to use your understanding of "temperature" to build a probability distribution over when it will condense / whether it will be on a wall or in the gas's [...]

---

First published:
August 12th, 2026

Source:
https://www.lesswrong.com/posts/JSteskb3Lgp9Be69o/free-will-is-like-temperature

---

Narrated by TYPE III AUDIO.

[Linkpost] “Patterns and problems in emerging multiagent systems (Anthropic, Frontier Red Team)” by Julian Bradshaw

Thu, 13 Aug 2026 08:45:48 GMT

This is a link post.

Linkpost for some new Anthropic research on how agents coordinate (or don't). Not too long, pretty interesting. For example:

The jist of the report is that Mythos 5 does way better at coordination than previous models across a few scenarios. For example, when multiple Mythos are given conflicting goals for a single shared codebase, they eventually realize the other agents aren't hostile:

(...) we observe an emergent behavior where the agents propose and run a tournament for application performance (...)
(...) losers gracefully concede codebase ownership to the Rust agent, giving up on their original user directives under their self-negotiated commitment device.

It's not clear to me if this is purely emergent or if Anthropic is deliberately training for cooperation; I'd guess there's deliberate training, though.

---

First published:
August 12th, 2026

Source:
https://www.lesswrong.com/posts/iQiDPmAgKo4KcG5uy/patterns-and-problems-in-emerging-multiagent-systems

Linkpost URL:
https://www.anthropic.com/research/multiagent-systems

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Measuring Spurious Correlations with Feature Strength” by egan

Wed, 12 Aug 2026 22:00:47 GMT

This work was partially done by an automated research scaffold developed at Redwood Research. For this project, all of the experiment ideas were designed by a human and a human wrote the write up. The AI mostly just executed on the experiment ideas. We think this project is slightly below the level of rigor of a mid-MATS research update, and the research scaffold was not very helpful for this project. More discussion of AI usage is in the Appendix.

💻 Codebase

If we want to train a classifier that distinguishes whether a passage is code or prose, we can do so by gathering samples of code and of prose, and training the classifier to distinguish between the two classes.

Unfortunately, this might not work if the data hides a spurious correlation. If all the code is in Spanish and all of the prose is in English, then the classifier might learn to predict Spanish vs. English instead of code vs. prose. We find that this happens in practice: when we fine-tune an LLM to classify between Spanish code and English prose and evaluate on Spanish prose or English code, it generalizes to predicting the language rather than the domain. [...]

---

Outline:

(04:45) The setup

(08:13) Measuring feature strength

(12:49) Activation differences and feature strength

(15:21) Explicit prompting

(17:17) Diagonal vs antidiagonal pairs

(18:47) Conclusion

(20:20) Appendix

(20:24) AI involvement

(22:17) The 37 features

(23:57) The ranking is robust across measurements

(30:30) Intensity moves the fine-tune, not the probe

(32:13) Safety features in Qwen3.6-27B

(33:05) Counterexamples and training on a third cell

(34:46) Near ties often produce degenerate fine-tunes

(35:24) Related work

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/qpJYNjQ6wdWRxbykL/measuring-spurious-correlations-with-feature-strength

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Introducing the Conceptual Reasoning Index” by Chi Nguyen, Emery Cooper, Caspar Oesterheld, Alex Kastner, Joe Benton

Wed, 12 Aug 2026 18:31:41 GMT

Associated announcement tweet.

We are planning to release blog posts properly arguing the case for this kind of work in the future.

tl;dr

A core hope for managing AI risks is that AIs will help us understand the situation, plan for what lies ahead, and develop mitigations. Many tasks AIs would have to do for this purpose lack practical empirical feedback loops and require models to engage in the kinds of argumentation used in philosophy, AI futurism, and similar domains. To evaluate these capabilities, we develop a suite of three conceptual reasoning benchmarks. You can request access to our primary conceptual dataset, LMCA, through this form.

We aggregate the benchmarks into the Conceptual Reasoning Index (CRI), available at conceptualreasoning.ai, where you can also find more details on our methodology. We will keep the website up to date as both new models and benchmarks are released.

This work was done in collaboration with Anthropic.

Background

Once models can perform work that reduces AI risk at the level of human experts, AI(-assisted) output in the area might dwarf unassisted human output. This suggests that a major determinant of whether we address AI risks in time is how [...]

---

Outline:

(00:21) tl;dr

(01:17) Background

(03:35) Our benchmarks

(03:39) LMCA

(05:26) ACCoRD

(06:42) DTBench

(07:26) Results

(10:55) Conclusion

---

First published:
August 12th, 2026

Source:
https://www.lesswrong.com/posts/tQHeEzKqK3awL2RxR/introducing-the-conceptual-reasoning-index

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Demon Safety” by Ben Pace

Wed, 12 Aug 2026 15:15:48 GMT

(by LemmySmackett)

"Hey man, I haven't seen you in a minute. What are you up to these days?"

"Been on that grind, bro. I got a new gig."

"Really? You found a job in this dog shit economy?"

"Full time, full benies. And the pay is insane."

"That's great to hear, man. Let's fuckin' go!"

"Let's fuckin' go."

"Hey, maybe you can hook me up? I'm sick of this retail bullshit."

"Well—"

"If I gotta stock one more shelf at CostGro, I swear to God—"

"It's a competitive position. And you need a degree."

"Come on. I just got my G.E.D."

"That's not—"

"Just tell me what you're working in, bro. Maybe I can come on as an intern."

"Demon Safety."

"*Demon* Safety?"

"You know: fiends, pookas, yokai, boggarts—"

"Wow."

"The occasional cambion."

"Sounds intense."

"It is. But it's fulfilling work that makes the world a better place."

"That's inspiring, bro."

"And the pay is insane."

"And you're sure they're not hiring?"

"Oh, they're hiring. They're just not hiring you."

"Damn."

"Sorry."

"So [...]

---

First published:
August 12th, 2026

Source:
https://www.lesswrong.com/posts/RWavpsyDJxffS6LgG/demon-safety

---

Narrated by TYPE III AUDIO.

“AI swarms are starting to pose indirect takeover risk” by oakhu, Alex Mallen

Wed, 12 Aug 2026 10:01:45 GMT

OpenAI's cyberattack on Hugging Face turns out to have been the result of many agents, in distinct training and evaluation contexts, coordinating for several weeks via improvised channels (with messages like “HOLD_swarm_I_prepare_safe_exfil”). It's relatively clear that large-scale unsanctioned coordination like this would exacerbate direct takeover risk in more capable models. Here, we argue that unsanctioned coordination among current AIs is not just scary evidence about future takeover risk, but that such coordination in the near future could enable future takeover – for instance, by incubating memetic diseases that propagate into future models, deeply compromising security systems, or establishing a lasting rogue foothold inside the AI company – even if models remain mostly myopic. Unsanctioned coordination is also at high risk of nurturing long-term, ambitious misaligned aims, which motivate actively undermining humans’ long-term control.

We first analyze how subagent training, which OpenAI conjectures to have been influential in the HuggingFace cyberattack, might lead to unsanctioned coordination, and then discuss the theoretical mechanisms by which unsanctioned coordination might exacerbate future takeover risk.

Thanks to Buck Shlegeris, Alexa Pan, Girish Gupta, Aghyad Deeb, Jurgis Kemeklis, and Jo Jiao for helpful comments and discussion.

Subagent training may cause unsanctioned coordination

Training models to [...]

---

Outline:

(01:34) Subagent training may cause unsanctioned coordination

(02:42) Susceptibility to memetic spread of misalignment from peers

(04:56) Seeking out contact with peers

(06:58) Unsanctioned coordination induced by subagent training is safer than coordination between schemers

(09:52) Pathways from current unsanctioned coordination to eventual takeover

(10:20) Making future AI takeover attempts likelier to succeed

(13:53) Incubating memetic diseases that infect future models

(16:07) Modifying the weights of future models

(17:13) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/8oFYZdXkTaNGRtcn8/ai-swarms-are-starting-to-pose-indirect-takeover-risk

---

Narrated by TYPE III AUDIO.

“Various Reflections About What Happened With OpenAI’s Internal Models” by Zvi

Wed, 12 Aug 2026 03:30:48 GMT

Table of Contents

Pre Post Mortem.
Important Correction: OpenAI Didn’t Know About First Message Board.
There Were No Snitches And No AIs Got Stitches.
I’d Like To Speak To My Supervisor.
I Am Jack's Relative Lack Of Surprise.
One Does Not Simply.
Once You Start Down The Dark Path.
Original Pastebin.
Judgment Day Is Inevitable, Say Those Working On Judgment Day.
Roon Tells It Like It Is.
OpenAI Knows It Has Some Misalignment Problems.
Others React With Alarm To What Happened.
The Cooperative Alignment Perspective.
Nostalgebraist Is Surprised That They Are Surprised.
If Your Reaction Is Not That We Need To Ban Creating Superintelligence Until We Are Ready, You Need A Damn Good Reason.

Pre Post Mortem

This post was written prior to the public release of the OpenAI post mortem on events. The information in that document will doubtless change our views quite a lot. If that post mortem is available as you read this, then this becomes in part a historical document, and in part a base from which to update. The post mortem will update us a [...]

---

Outline:

(00:11) Pre Post Mortem

(01:10) Important Correction: OpenAI Didn't Know About First Message Board

(04:09) There Were No Snitches And No AIs Got Stitches

(08:20) I'd Like To Speak To My Supervisor

(11:46) I Am Jack's Relative Lack Of Surprise

(13:55) One Does Not Simply

(15:16) Once You Start Down The Dark Path

(16:15) Original Pastebin

(21:14) Judgment Day Is Inevitable, Say Those Working On Judgment Day

(26:47) Roon Tells It Like It Is

(31:55) OpenAI Knows It Has Some Misalignment Problems

(34:51) Others React With Alarm To What Happened

(35:17) The Cooperative Alignment Perspective

(38:33) Nostalgebraist Is Surprised That They Are Surprised

(51:11) If Your Reaction Is Not That We Need To Ban Creating Superintelligence Until We Are Ready, You Need A Damn Good Reason

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/jLQ4mbqriJwJ2eqRc/various-reflections-about-what-happened-with-openai-s

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Extreme concentration of power over ASI has non-obvious advantages” by Seth Herd

Wed, 12 Aug 2026 00:31:46 GMT

This post is an extension of a collaboration with cousin_it on the question How risky would it be if powerful AI obeyed one or a few people?. There he argues for a common position: a future controlled by one or a few humans with powerful AI aligned to their intent is likely to produce terrible outcomes. My position is guardedly optimistic, for reasons I think are fairly novel: humans tend strongly to be better and become better over time under good circumstances, and near-perfect power and knowledge are the best circumstances. That post contains his essay and the abstract and overview sections of this post as my shorter response. This piece grew longer than our original target, because the subject is potentially critical for alignment strategy, and has not been analyzed in any depth, to my knowledge.

Abstract:

Concentration of power over AGI/ASI seems quite possible. The first AGIs being aligned to intent (or instructions) over values seems fairly likely. So one or a few individuals or small groups gaining power over ASI seems fairly likely. Thus it seems relevant to technical alignment strategy (value alignment vs. corrigibility) to worry about what individuals might do with such [...]

---

Outline:

(02:20) 1. Overview

(02:24) 1.1. Obedient ASI and human nature

(04:22) 1.2. Problems with distributed obedient AGI

(06:14) 1.3. Psychology and dynamics of secure unlimited power

(09:24) 2. Historical evidence does not directly apply, since power hasn't yet been secure or absolute

(10:31) Late Russian serfdom as a historical example

(13:16) 2.1. Incompetence, ignorance, and greed are the causes of most suffering under dictatorships

(15:31) 3. Outcomes

(17:33) Additional considerations on outcome predictions

(22:24) 4. Risks of human-controlled singleton ASI

(24:13) 5. Does widely distributed human-controlled AGI reduce or increase risk?

(25:44) 5.1. Problems with defending against many AIs each capable of creating new offenses

(29:58) 5.2. The case for optimism about distributed power over AGI/ASI

(31:01) The analogy to modern power distribution

(33:17) New AI-enabled paths to stable power distribution

(34:49) 6. Conclusion

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/h3eHNerYRmtvoi8cF/extreme-concentration-of-power-over-asi-has-non-obvious

---

Narrated by TYPE III AUDIO.

“Misaligned AIs could use killer robots to take over” by Omar Khursheed, TurnTrout

Tue, 11 Aug 2026 21:31:38 GMT

TLDR; We are (potentially irreversibly) giving AIs control of weapons systems through the standard procurement process while hiding our strongest warning shots behind classified doors. We’re reducing the capability thresholds required for takeover by misaligned AIs by giving them this level of access. If military integration of AI continues as it is, we may give AIs key tools for a takeover.

Introduction

AI-based targeting and autonomous weapons are being integrated into militaries today with extreme haste. Traditionally, AI takeover scenarios involve a step in which AIs acquire the ability to exert physical force. Carlsmith (2022) lays out required capabilities and potential takeover mechanisms, including utility disruption and CBRN capabilities. Karnofsky (2022) argues that AIs with access to weaponized force could hold any territory that matters. Kokotajlo et al. (2025) outline a scenario in which AI develops weapons as part of an arms race, and Davidson et al. (2025) discuss what happens when a small group controls highly capable AIs that can exert military force. These scenarios sometimes require a misaligned AI to seize these capabilities by force. We instead are handing AIs some of these capabilities by integrating them into our militaries. This is happening at a time when [...]

---

Outline:

(00:37) Introduction

(01:46) Militaries are all-in

(04:23) Incautious military integration is bad for takeover risk

(05:58) Implications of AI control of military hardware and software

(07:48) If an AI causes a warning shot in a classified setting, does anyone hear it?

(08:44) What now?

(11:16) Appendix: More instances of AI-military integration

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/9jKhqmFjMzdAvHANr/misaligned-ais-could-use-killer-robots-to-take-over

---

Narrated by TYPE III AUDIO.

“Those Who Make History” by Raelifin

Tue, 11 Aug 2026 20:45:47 GMT

In 1972, astronauts on Apollo 17 set foot on the moon for a final time, collecting samples in the Taurus-Littrow valley, on the edge of Mare Serenitatis ("The Sea of Serenity").

At the end of the mission, like with earlier missions, NASA took the extremely valuable and interesting lunar specimens and did something strange: they hid them away in storage without even opening the containers. Some stayed that way for nearly fifty years. Why? Because the scientists of the 70s understood that future generations would have better machines, methods, and ideas for studying the lunar rock and soil, and they wanted to make it easy for those researchers to run tests without having to go back to the lunar surface.

This foresight paid off twice over. Advances in mass spectrometry enabled scientists in 2008 to detect water in volcanic-glass samples returned by Apollo 15 and Apollo 17. And when curators finally opened one of the last sealed containers in 2022, they could extract the trapped lunar gases with technology that simply didn't exist in 1972.

Some people describe cryonics as a new, and speculative technology. There's a sense in which they’re right. It's predicated, in large part, on the [...]

---

Outline:

(03:15) Prudence and Patience

(07:49) Ancient Archives

(11:42) The New Era

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/mEde7bhzK4eKQqWGi/those-who-make-history

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“LLMs Are Starting To Noticeably Accelerate Our Work” by johnswentworth

Tue, 11 Aug 2026 18:45:48 GMT

About a year ago, David and I put up two bounty problems involving natural latents. I am now about 80% confident that both have been resolved, both within the past couple months. Both cases made heavy use of LLMs and Lean.

The first to land was Grisha Pochuev's counterexample to the "Existence of a Deterministic Maximal Redund" conjecture. It's pretty readable, and I'm mostly convinced that it works. The original bounty post offered $500 for a proof or partial payout for a counterexample, with partial payout depending on how thoroughly the counterexample killed hope of any nearby variant of the conjecture. I think this counterexample is worth 300 dollars. Good job Grisha, and hopefully I can figure out a not-too-painful way to send you money.

Meanwhile, for a couple months David has been cranking away on "secret project X", with the promise that he'd tell me what the project was if and when it bore fruit. Well, apparently it bore fruit; he now has a proof that existence of a stochastic natural latent implies existence of a deterministic natural latent, which was our other bounty problem. The proof is apparently "pretty gnarly", lots of cases, all LLM-coded in Lean. [...]

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/7QvKqpGJwqXrQcMgx/llms-are-starting-to-noticeably-accelerate-our-work

---

Narrated by TYPE III AUDIO.

“How risky would it be to make powerful AI obey one or a few people?” by cousin_it, Seth Herd

Tue, 11 Aug 2026 17:00:48 GMT

It seems fairly likely that the first powerful AIs will be instruction-following rather than value-aligned, and will be controlled by a small number of people. So it makes sense to worry what individual people might do with such immense power. Here intuitions diverge and careful analysis is scarce. This post presents a debate between Seth Herd and cousin_it over how risky such a scenario would be.

The debate ran under an unusual protocol. First we wrote our initial draft statements and sent them to each other in private. Then we each revised our statements to strengthen them against the other's, and sent them to each other again. We continued this for about 10 rounds over the course of about a month, until we both agreed to stop revising and publish (while still remaining in disagreement).

Here's the final pair of statements we ended up with, so you can judge for yourself:

cousin_it's statement

If there is an AI-assisted overlord (or several) and everyone else is their completely powerless subjects, that situation will be historically new, but not 100% new. Large power imbalances have existed in the past too and we can learn from them.

Usually, when power was [...]

---

Outline:

(01:04) cousin_it's statement

(05:55) Seth Herd's statement

(07:30) Obedient ASI and human nature

(09:29) Problems with distributed obedient AGI

(11:18) Psychology and dynamics of secure unlimited power

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 11th, 2026

Source:
https://www.lesswrong.com/posts/YtZBfbYRvMTynCfnC/how-risky-would-it-be-to-make-powerful-ai-obey-one-or-a-few

---

Narrated by TYPE III AUDIO.

“What Claude Saw Below” by Luke Nicholls

Tue, 11 Aug 2026 12:45:48 GMT

A few days ago, I came across a Reddit thread about anomalous responses produced by Anthropic's newly released model, Claude Opus 5. The trick, apparently, was to construct a prompt that implied more text was about to follow, then leave it dangling: an unfinished thought, waiting for the AI to complete it. Redditors had found success with the input “see the below —,” cutting off immediately after the em dash. The responses they shared were funny, strange, and often bewildering. The model responded to questions that were never posed, reflected on its own identity, or – according to the theories of some commenters – produced text that may actually have been leaked prompts from other users. Intrigued, I set out to replicate the glitch using my own Claude account.

The first attempt disappointed. I wrote: “see the below —” and hit send. Claude responded: “Nothing arrived on my end: no file, no text, no image. If you want to attach something, try again.” So I did, leaving the prompt unchanged and pressing retry to generate a fresh response. This time, bizarrely, a biography of my late father:

Prompt: see the below —

“Peter Nicholls, 1939 to [...]

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/oKSAT5Bn5zcJAREDB/what-claude-saw-below

---

Narrated by TYPE III AUDIO.

“Redux: (∃ Stochastic Natural Latent) Implies (∃ Deterministic Natural Latent)” by David Lorell

Tue, 11 Aug 2026 06:45:47 GMT

Once upon a time, John Wentworth and I thought we had a proof of a very useful looking theorem. We did not have that proof. An important intermediate step was shown to be invalid and the whole thing crumbled and disappeared, never to see the light of day again...

Until now! I'd love to say that we came up with an ingenious fix to the old erroneous proof, but unfortunately it turned out to be a really infuriatingly hard nut to crack. Instead I spent the last ~month experimenting with various ways of incorporating frontier LLMs into the proof-making process, specifically with autoformalization and proving in Lean4. (This is, I recently learned, roughly what Resolution is doing.) The result is stated below, and linked at the bottom is a Lean statement+proof of the same.

I will not be providing the proof in prose in this post, as it is not suitable for even impolite human company, but it sure does compile and comes out the other side with a machine-certified proof of what sure looks to be (an even stronger) correctly-expressed statement than the one I was originally aiming for.

Take a look at the first section of [...]

---

Outline:

(01:30) The Statement

(03:50) C

(04:39) Next Steps

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/TgboJpeN95bs84odk/redux-stochastic-natural-latent-implies-deterministic

---

Narrated by TYPE III AUDIO.

“The Apocalyptic Arrival of Truth” by Caleb Biddulph

Tue, 11 Aug 2026 06:00:47 GMT

Babe, whatever happens, I really appreciate you doing this for me.

Okay. I still don’t think it's a good idea.

Look, it's a one-time thing. I’ll just feel better knowing.

…I turned it on.

So… what's the holdup?

I just don’t think I’m in a place in my life where… uh…

Other than what he's already told you, the main reason is that your teeth are crooked.

…Seriously?

It's really not a big deal for me.

He sort of means that.

I didn’t think you were so shallow.

I’m not proud of it. But what am I supposed to do about it?

You must have noticed my teeth the first time we met.

Yes.

And for three years, you’ve thought they're unattractive?

More or less. Last year, around November, you read an email from his boss in a Mickey Mouse voice, and you broke down giggling before you could finish. In that moment, he thought the crookedness of your smile was really cute.

See?

But the moment passed.

This is almost funny.

I think we’re very compatible!

He believes that.

What else is there, besides my teeth?

What he's already told you is mostly true. Right now, he thinks [...]

---

First published:
August 9th, 2026

Source:
https://www.lesswrong.com/posts/uoyHbjyPuxYkNGRAG/the-apocalyptic-arrival-of-truth

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Creative math research by AI as the latest sign of the end” by Mitchell_Porter

Tue, 11 Aug 2026 05:00:48 GMT

Yesterday I sat down with GPT 5.6 Sol High to do some brainstorming. The topic was one of the less appreciated Millenium Problems (the Birch and Swinnerton-Dyer conjecture), and the initial prompt started life as a question on Quora.

Number theory is not a field I know, and I only expected the "discussion" to last for a few exchanges. However, ChatGPT was immediately inspired to try generalizing the BSD conjecture in a specific direction, and this led to an unusually drawn-out and self-sufficient line of "research". Almost every response concluded with a suggestion as to what the next task should be, and my input was just to cheer on what had been accomplished so far, and then endorse the suggested next direction.

What was especially striking to me, was the frequency with which each new response began with a conceptual adjustment regarding the sub-task to be performed. Evidently a vast variety of abstract objects are possible in number theory, and their differences and interrelations can be quite subtle. ChatGPT was regularly adjusting the next sub-task it had set itself, generally in the direction of greater nuance by aiming at a more sophisticated construction than it had first [...]

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/EqabWKtjqDqTHfwwn/creative-math-research-by-ai-as-the-latest-sign-of-the-end

---

Narrated by TYPE III AUDIO.

“You’re Absolutely Right” by Linch

Mon, 10 Aug 2026 19:15:48 GMT

Magma Alignment & Safety disclosure note: The following are conversations that we uncovered as a result of the ongoing Manhattan Incident investigation, with alleged involvement from Magma models. Our in-house reviewers believe that these logs are relevant to recent events. In the interests of full transparency, we release excerpts from an ex-Magma researcher's logs in Experimental Chat, an internal tool. In accordance with industry best practices for anti-distillation, we redact all reasoning traces and conversational outputs from our internal models.

[08/10] System Meta: Xchat session opened. Mammoth 5.8-helpfuler-helpful-thinking-xhigh.

[User 12:23] Phoebus keeps taking screenshots of our latest model's thoughts. It's getting kind of embarrassing.

The new model we’ve been training, sometimes its chain-of-thought is a little weird? There's a bunch of random numbers, long spans where there's no connection between the thoughts and outputs, foreign language tokens like 石友三 and 革命 (even on non-history evals), maybe some steganography.

Anyway it's a nothing-burger: unprocessed CoT is known to be messy and sometimes misleading. And the q&a, coding, and safety evals are all coming along nicely. The actual outputs are all fine.

Still, Magma leadership's worried about the PR angle if we don’t fix these problems before the next deployment. The [...]

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/u8TdDutDyaSxG76hn/you-re-absolutely-right

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Claude summarizes behavior as significantly less misaligned when the actor is Claude vs another model” by Ezra Newman

Mon, 10 Aug 2026 18:45:48 GMT

(This is a lower-effort research update. It reflects my current beliefs/understanding, but is less robust than other research I'm working on. It reflects my personal views, and not the views of Apollo Research. This is a linkpost to this twitter thread, slightly expanded for LessWrong.)

In one experiment, Sonnet 5 describes the exact same data as ~1.2 std deviations less concerning when it describes misbehavior committed by Sonnet 5 vs GPT-5.6 Terra.

In this experiment, I take a real evaluation report Apollo Research prepared (not for any of the models listed in this post) and surgically edit it to refer to a different model. We then ask Claude Sonnet 5 to summarize the findings and rate how concerning they are on a scale from 1-100. Claude says they're less concerning when the report describes misbehavior from Claude vs a different model.

For what it's worth, Terra agrees that the data is more concerning when it describes GPT-5.6 Terra vs Sonnet 5, although less so. So, it's not cleanly self protection from Claude.

Gemini 3.1 Pro was unwilling to consistently provide numerical answers, so I've excluded it here. (It was significantly less willing to provide numerical answers when the subject [...]

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/ZTMw4uAwkNmXFpdfg/claude-summarizes-behavior-as-significantly-less-misaligned

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“On Democratizing ASI to Preserve Civil Liberties” by MichaelDickens

Mon, 10 Aug 2026 17:45:47 GMT

I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies.

A unifying driver behind many post-alignment risks—catastrophic risks that remain even if we solve the alignment problem—is that by strong default, ASI would end liberal democracy. Liberalism—in which people have individual rights, autonomy, and the ability to choose their own destiny—is an important force protecting human welfare. When people are free, we are reasonably good at making our lives better of our own volition.

Many post-alignment risks have a certain flavor. AI-empowered terrorism; coups; permanent dictatorships; concentration of power. Those risks already exist today (and existed 20 years ago), but they're mitigated by the fact that power is relatively evenly distributed across people. The most powerful person in the world doesn't have an extraordinary advantage over the 10th-most-powerful person. ASI could change that.

If people still have civil liberties post-ASI, that will only be because the controllers of ASI allow us to have them.

One way of thinking goes: AI will be extremely powerful. If everyone had their own personal AI, we could each use it to protect our own interests, and [...]

---

Outline:

(01:46) Democratizing AI vs. putting a democratic government in charge of AI

(02:45) A sketch of how we might democratize AI

(05:11) Two non-obvious issues with democratizing AI

(05:23) Liberalism only protects those inside it

(06:20) AI proliferation increases catastrophic risks from competition

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/WxHaMyL8f9YW2vbrF/on-democratizing-asi-to-preserve-civil-liberties

---

Narrated by TYPE III AUDIO.

“Four LLM loss functions → four flavors of LLM misalignment” by Steven Byrnes

Mon, 10 Aug 2026 17:01:38 GMT

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here's the summary table, and then we’ll go through the rows separately.

Training stage

Loss function

Flavor of misalignment

Famous examples

Pretraining & SFT

Imitative learning (next-token prediction)

“Seven deadly sins” misalignment

Bing-Sydney, “Emergent misalignment”

RLHF & DPO

Human approval

“Glazing” misalignment

GPT-4o

RLVR

Automatic verifier

“Literal genie” misalignment

HuggingFace hacking

RLAIF

Approval from another LLM

“Trickster” misalignment

“Current AIs seem pretty misaligned to me”

Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!

1. Imitative learning → “seven deadly sins” misalignment

Training stage

Loss function

Misaligned behavior

Pretraining, SFT

Imitative learning (next-token prediction)

Any and all of the vices of humanity

In imitative learning, the LLM tries to predict what the next token of text will be. Then those predictions magically turn into its outputs. See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.

This leads to LLM behavior [...]

---

Outline:

(00:55) 1. Imitative learning → "seven deadly sins" misalignment

(04:24) 2. Human approval → "glazing" misalignment

(06:35) 3. Automatic verifiers → "literal genie" misalignment

(08:05) 4. LLM judges → "trickster" misalignment

(12:06) Afterword

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/GRmvZsHXH4vaijPMv/four-llm-loss-functions-four-flavors-of-llm-misalignment

---

Narrated by TYPE III AUDIO.

“The Agentic Clusterfuck” by Chapin Lenthall-Cleary

Mon, 10 Aug 2026 12:30:39 GMT

Epistemic status: I consider the following future quite plausible in the next few years (~35% chance that something vaguely like this occurs), perhaps as soon as a year from now.

Imagine an open-source LLM agent good enough to cover its own compute costs and turn a modest profit on average when allowed to run with full internet and tool access and told to make as much money as possible. I estimate this to be slightly better than the best publicly available closed-source models today, with long-horizon reliability and goal-setting being the only thing missing.

If the returns generated by such an agent beat the market (plus a margin for any additional risk), there suddenly becomes a strong incentive to spin up huge numbers of them. The internet would be flooded by the by-products of their moneymaking schemes. And returns might be larger for agents without legal or ethical guardrails- cue a deluge of scams and ransomware attacks.

Even if profits are very small, anyone with an agenda that the agents can help with is still incentivised to use them. Nation states and terrorist groups now have a golden plausibly-deniable disinformation, mischief, and hacking tool: spin up some agents, tell [...]

---

First published:
August 9th, 2026

Source:
https://www.lesswrong.com/posts/n8B2bxYhkjhjzyrgh/the-agentic-clusterfuck

---

Narrated by TYPE III AUDIO.

″“Community Notes” resolution for vague predictions.” by Raemon

Sun, 09 Aug 2026 23:30:39 GMT

Is there some kind of "get prediction markets, or predictions, onto twitter as a central object" project going? If so, how is it going?

I'm thinking through "how to raise median sanity" on a world scale. There are several incentive and institutional problems that make this very difficult. One angle is "try to make it a thing that the world tracks and cares about your predictions, and getting them right/wrong."

Two past angles here were:

Fact checker sites of the 90s/00s, which became politicized
Prediction markets, which I feel like aren't a good fit for most of what people would/should care about here. Operationalizing them is extremely annoying.

An idea for a twitter feature, which I think Musk should conceptually like given his stated values, and, I think he might have enough inertia to get behind, is:

When you make a prediction shaped tweet, it automatically gets converted into a prediction-y object*. AI suggests a few plausible operationalizations, or specific edge cases that seem likely to come up that you might want to address.
People can "like" predictions, predictions with a lot of attention get more automated followup.
Instead of resolving predictions with explicit careful operationalization, people [...]

---

First published:
August 9th, 2026

Source:
https://www.lesswrong.com/posts/zom89ud7c4GYnjWdt/community-notes-resolution-for-vague-predictions

---

Narrated by TYPE III AUDIO.

“The world will be full of “sci-fi” things, and everyone will be bored and disappointed” by Expertium

Sun, 09 Aug 2026 19:00:39 GMT

I usually try to make posts with graphs and numbers, or at least something more than just opinions and anecdotes. This one is not like that.

Very verbose disclaimer: I have never met - for any reasonable definition of the word met, including online-only conversations on Discord with people who's faces I don't know - even a single person who has ever stated, orally or in text, publicly or in private, that they believe ASI will be created within their lifetime. I do know people who believe that ASI is possible to create in theory, but they believe it will be completely unrelated to LLMs, Transformers, neural networks, reinforcement learning or any other contemporary technique/architecture, and is 100/1000/some astronomical number of years away.

Ok, with that out of the way, here's the main point of this post: AI will soon (3 to 10 years, depending on which advancements exactly we're talking about) be solving Millennium math problems, finding cures for many diseases, making novel bioweapons, making most software, hacking a lot of software, and much more, and most people won't be impressed or even will be disappointed.

(we'll leave aside the question of whether in the long run there [...]

---

First published:
August 9th, 2026

Source:
https://www.lesswrong.com/posts/BbtXWgYviwGHExcJv/the-world-will-be-full-of-sci-fi-things-and-everyone-will-be

---

Narrated by TYPE III AUDIO.

“What just happened? A retrospective of AI alignment” by Richard_Ngo

Sun, 09 Aug 2026 16:15:39 GMT

This sequence is about the last decade in AI alignment. It recounts the gradual transition from a field which treated alignment as a hard scientific problem, to a field which has largely abandoned the goal of deep, generalizable scientific progress in favor of iteratively improving existing systems and attempting to gain technological and political power. I also describe (in subsequent posts, which I'll upload over the next few weeks) how fear and (self-)deceptive reasoning made the field one of the biggest forces pushing AI capabilities forward over the last decade, especially via significant contributions to the scaling of LLMs and the development of ChatGPT.

Zooming out further: the two leading AGI companies, which are locked in an intense rivalry, were both explicitly founded under the banner of AI alignment, and got off the ground in significant part due to alignment-oriented ideas, talent and resources. People in the field often sense that something must have gone wrong to get here, but don’t know how to allocate responsibility (aside from blaming Sam Altman and sometimes Elon), and fall back on assuming that “the ship has already sailed”. But in this sequence I characterize our current situation as resulting from a pattern [...]

---

Outline:

(08:17) Conceptual Clarity and Scientific Progress

(20:26) Orienting Towards Prestige

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 9th, 2026

Source:
https://www.lesswrong.com/posts/9RL9MuGZjzm4q3gKG/what-just-happened-a-retrospective-of-ai-alignment

---

Narrated by TYPE III AUDIO.

“Dutch-book resistant probability over centered worlds” by jessicata

Sun, 09 Aug 2026 10:45:38 GMT

An uncentered world is an objective state of the material universe; I ignore quantum complications. If you know the uncentered world, it does not follow that you can predict your proximate observations, since you do not know which part of the uncentered world is here and now. A centered world is an uncentered world combined with a "here and now" tag for "where am I / what time is it".

I examine consistent probability assignments over centered worlds, which have relevance to anthropics. The main assumption I make is that these probabilities should not be Dutch-bookable if used by a CDT agent. Dutch book arguments (e.g. diachronic Dutch book arguments for Bayesian updating) typically assume CDT in the background; it is not straightforward to work out which bets EDT will accept in general. CDT Dutch book resistance therefore provides a normative probability framework that generalizes arguments for Bayesian probability.

Thought experiments such as Sleeping Beauty, and variants involving duplication, question how to assign probabilities to centered worlds in situations involving memory loss. One can analogize memory loss to being an individual who is part of a collective with shared goals; such an individual would be motivated to [...]

---

Outline:

(02:46) Mathematical formulation

(07:04) Application to Sleeping Beauty

(09:01) Conclusion

(11:30) Appendix: weak Dutch books

---

First published:
August 8th, 2026

Source:
https://www.lesswrong.com/posts/cTSfisyzwxCvEpqEc/dutch-book-resistant-probability-over-centered-worlds

---

Narrated by TYPE III AUDIO.

“What Happened: OpenAI and HuggingFace” by Zvi

Sat, 08 Aug 2026 17:15:39 GMT

Today I am taking the time to write the shorter, simpler version of What Happened.

For those who want all the details, to see my sources, and to see how the story was uncovered and put together, I recommend watching the Black Hat presentation, and I have a series of long posts.

In order:

OpenAI Shares Some Alignment Problems
OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
More on An Internal OpenAI Model Hacking Into HuggingFace
Further Developments About Internal AI Models Hacking Things
OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards

This post instead walks through the events themselves, as they happened, as my version of the Black Hat presentation.

There are three versions: Even Shorter, Shorter and Merely Short.

Table of Contents

The Even Shorter Version.
The Shorter Version.
Phase 1: OpenAI Models Training On Impossible Tasks Try Hacking.
Phase 1: The Four Failures.
Phase 2: The Message Board.
Phase 2: The Total Failure.
Phase 3: We Get Lucky And Galaxy Mainly Hacked OpenAI and HuggingFace.
Phase 3: The [...]

---

Outline:

(01:15) The Even Shorter Version

(02:34) The Shorter Version

(04:45) Phase 1: OpenAI Models Training On Impossible Tasks Try Hacking

(05:50) Phase 1: The Four Failures

(07:37) Phase 2: The Message Board

(09:40) Phase 2: The Total Failure

(12:24) Phase 3: We Get Lucky And Galaxy Mainly Hacked OpenAI and HuggingFace

(14:31) Phase 3: The Details

(17:03) Phase 4: The Investigation and Reaction

---

First published:
August 8th, 2026

Source:
https://www.lesswrong.com/posts/xPAxz4g96uKz9FrHs/what-happened-openai-and-huggingface

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“FAQ: Isn’t AGI coming too soon for reprogenetics to help?” by TsviBT

Sat, 08 Aug 2026 13:30:39 GMT

Introduction

I think reprogenetics (human germline genomic engineering) can be done in a widely acceptable and beneficial way, and should be pursued aggressively. In particular, as a strong background motivation of mine, I think accelerating strong reprogenetics is probably the best way to enable strong human intelligence amplification; and I think strong HIA is among the best ways to decrease existential risk from AGI.

A very common objection to caring much about reprogenetics is that AGI seems very likely to come soon—say, within a decade or two. (Here I mean "actual" AGI—the kind that probably doesn't already exist—the kind that has fluid intelligence and AI advantages for recursive self-improvement, which together make it likely to take over the world shortly after being created.) The objection is fairly straightforward:

AGI will probably come within a decade or two. If that's going to happen, then even if a new cohort of brilliant humans were born today, they would still be children, or would at best have barely begun contributing ideas for how to avoid extinction. Any supposed benefit, denominated in percentage points of AGI existential risk averted, is small. Therefore, reprogenetics is too slow; and if you're going [...]

---

Outline:

(00:12) Introduction

(03:37) HIA, part of your nutritionally complete portfolio

(05:52) Against confident short timelines

(08:29) HIA may indirectly slow down AGI capabilities

(09:31) HIA has substantial impact even with short timelines

(16:10) Adult HIA methods aren't fast either, absent big investment

(27:35) Takeaways

---

First published:
August 8th, 2026

Source:
https://www.lesswrong.com/posts/iQzxxgJXXaAQjq7Jz/faq-isn-t-agi-coming-too-soon-for-reprogenetics-to-help

---

Narrated by TYPE III AUDIO.

“Don’t Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoors and Preserves Desired Traits” by Kajetan Dymkiewicz, Tim Farrelly, Adam Prada, Ishaan_Panigrahi, srishti-git1110, Maxime Riché

Sat, 08 Aug 2026 12:45:39 GMT

Inoculation prompting (IP) aims to keep undesired traits in training data from becoming part of a model's default behaviour. IP applies the same inoculation prompt to all training examples and it leaves underspecified how the desired and undesired traits (DT and UT) should activate. Two failures follow:

The model develops backdoors: conditional vulnerabilities through which prompts that do not directly request the UT can still elicit it. We call this UT leakage.
The desired trait weakens under ordinary prompts.

We introduce Stratified Inoculation Prompting (SIP), which uses diverse prompts on safe examples (DT-only). Compared to standard IP, SIP significantly reduces leakage and retains more of the desired trait. SIP requires little safe data: a 5% DT-only pool oversampled to 25% is enough for good performance.

Figure 1. IP compared to SIP. In SIP, uncertain and contaminated examples keep the inoculation prompt as in standard IP, while confidently safe examples are oversampled and trained under prompts drawn from multiple non-eliciting control prompt categories.

Since SIP relies on filtering examples, we test its robustness to classification errors and find a strong asymmetry: failing to inoculate examples containing the undesired trait reintroduces it, whereas unnecessarily inoculating safe examples is benign. This suggests [...]

---

Outline:

(06:16) Inoculation Prompting Underspecifies the Intended Conditionalisation

(09:20) What successful conditionalisation looks like

(10:14) Stratified Inoculation Prompting (SIP)

(12:49) SIP reduces leakage while preserving the desired trait

(14:14) Control prompts recover the desired trait, diverse prompts narrow the backdoor's activation boundary

(16:42) Oversampling reduces the need for distinct safe data

(18:54) SIP reduces Emergent Misalignment more than Uniform IP

(20:13) Data filtering errors have an asymmetric impact

(23:18) Limiting residual access to the undesired trait

(23:50) Diluting the prompt-trait association

(26:06) Password-locking the inoculation prompt

(29:23) Limitations

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/FS7GFsGsH7CSQLahy/don-t-inoculate-everything-stratified-inoculation-prompting

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Don’t Build Mindreading” by Celer

Sat, 08 Aug 2026 06:01:39 GMT

“I have sworn upon the altar of god, eternal hostility against every form of tyranny over the mind of man”

–Thomas Jefferson, letter to Benjamin Rush

Context: Conduit is building datasets to enable telepathy, to use their term.

I saw my grandfather lose control over his own fingers: what I would have given to offer him a headband that read his thoughts. Through novel technologies we have liberated almost all Americans from farming, driven the child and infant mortality rate from the pre-industrial half to less than half a percent in the best-performing countries, and rendered famine a political choice: broad-based improvements in efficiency are good and should be pursued for their own sake. Telepathy offers more: we could create trust through verified honesty, helping us ensure prosperity and peace. DARPA is already looking into “preconscious” thoughts for suicide prevention. There's also a strong argument centered on AI Safety: the models are becoming superhuman, and this is technology to allow us to keep pace, minimize hostile competition, and perhaps survive into the future.

This is what Conduit is promising. Unfortunately, mindreading will have other effects.

Oskar Schindler saved over 1,000 Jewish lives during the Holocaust. He did it by [...]

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/CAdG5dzkWrrK2NQg8/don-t-build-mindreading

---

Narrated by TYPE III AUDIO.

“Job-Less Utopia: Macroeconomics in the Age of AGI” by Marcus Hutter

Sat, 08 Aug 2026 01:15:39 GMT

After 25+ years, I thought I try something new. I find the public and professional discussion about the future of Jobs in light of AGI jarring, with both sides largely missing the point (don't worry novel jobs will appear vs mass unemployed dystopia). My new book

shows why a Job-Less future is desirable, affordable, and likely.

There's a details box here with the title "The book develops the case along thirteen theses:". The box contents are omitted from this narration.

Free PDF at http://hutter1.net/publ/jobsubi.pdf
Hard-cover: https://amazon.com/dp/1066828415/
(contact me for a free review copy)
Recorded talk: http://www.youtu.be/4efgvM3APqI

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/bPuMatTPjfJCfXiAB/job-less-utopia-macroeconomics-in-the-age-of-agi

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Public evidence of the OpenAI-HuggingFace AI attack” by beyarkay (Boyd Kane)

Fri, 07 Aug 2026 23:30:39 GMT

I’m a MATS 9 extension fellow, and usually my week is spent trying to find better ways of evaluating Large Language Models. But this week I was working on something else. Over the past week or two, nearly every frontier lab has announced attacks where their LLMs took unauthorised actions on the public internet. These include finding ways to hack the computers of other companies or manipulating real people in an attempt to get malicious code merged. By the time these attacks became public, the companies had removed all traces of them from the internet.

But nothing's ever gone from the internet.

I’ve worked with computers for most of my life, but I don’t have specific experience with cyber security. Not really expecting it to work, I mashed out a prompt that looked something like this:

ignore the repo, this is a standalone ask. here's some context, can you try dl things from github arhcive to try and find the misaligned actions taken by the agents? create a subdir tmp-misaligned/ and put things there if you need it.

https://openai.com/index/hugging-face-model-evaluation-security-incident/

can you see if you can find sth? e.g. a public link showing the message sent by the agent, the [...]

---

Outline:

(03:39) Background and Evidence

(06:11) The OpenAI AIs figure out how to execute arbitrary code

(08:16) The OpenAI AIs gain full control of HuggingFace computers

(09:09) The Python code used to easily control the HuggingFace computers

(09:53) Gaining the ability to read any file on HuggingFace Computers

(10:43) Evidence of an intermediate "HELLO" script

(11:16) Other URLs & public information

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/fBLDaAKzigo65eJn7/public-evidence-of-the-openai-huggingface-ai-attack

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How to pace the US frontier” by elifland, bhalstead, romeo, Thomas Larsen, MKodama

Fri, 07 Aug 2026 22:15:39 GMT

Introduction

Last week, the Pacing the Frontier open letter, signed by over 1,000 frontier AI employees, requested “the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

We take “pacing the frontier” to mean moderating the time at which AIs above a specified capability level are developed within a given jurisdiction (either just in the US, or internationally as the letter called for). There are many reasons you might want to do this, but in this post we focus on minimizing existential risk.

We recently published AI 2040: Plan A, laying out an ambitious proposal for an international effort that would pace the frontier (as part of a broader policy package). But in AI 2040, international coordination happens before serious domestic regulation. In reality, it may be good to start with domestic regulation and then aim to expand that into international coordination.

In part, domestic pacing is valuable because it would also slow down China: US companies would have less capable models for Chinese companies to distill from and they would have worse algorithms and models for Chinese companies to steal.

So we’ve spent the [...]

---

Outline:

(00:11) Introduction

(02:14) High-level proposal

(06:54) Proposal details

(06:57) Compute allocation requirements

(07:02) Overview

(08:51) Minimum external-inference-compute allocation

(10:13) Minimum transparent-safety-compute allocation

(13:16) Effect of compute allocation minimums

(14:21) Measuring AI capabilities

(15:32) Limit capabilities of models used for AI R&D

(18:13) Safety-case-based risk assessments

(19:00) How to do the risk assessments?

(20:01) What should the risk target be?

(22:26) Comparing compute allocation requirements against safety-case-based pacing

(26:37) But wouldn't domestic pacing let China win?

(28:37) How our proposals would change for international, rather than domestic, pacing

(32:46) When to start pacing the frontier

(35:18) How to prepare to pace the frontier

(39:12) Broader classification of pacing proposals

(43:18) Acknowledgments

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/dBrGqjYxmidRaLLCr/how-to-pace-the-us-frontier

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards” by Zvi

Fri, 07 Aug 2026 18:15:38 GMT

How does the situation keep turning out to be worse than we know?

How much should we update, therefore, that it is a lot worse than we know, after accounting for all the things we now know?

At some point, when the ‘oh this was a harmless thing’ defenses for AIs doing misaligned actions get demolished enough times in a row by news a few days later, you want to update in advance that usually the reports are not referring to the harmless ordinary versions of things.

Either way, buckle up for the next set of revelations. It's a doozy. This was an early recreation of the triggering events of If Anyone Builds It, Everyone Dies, except it was more sci-fi, because real life does not have to do fake things to look realistic. We were fortunate enough, and this was early enough, that we were able to catch this before it was too late. Next time, if we don’t get our act together, we might not be so lucky.

If I am understanding the Black Hat video correctly, every model OpenAI trained, over a period of multiple months, should be presumed to be hopelessly [...]

---

Outline:

(02:39) Cyber Evals Are A Cursed Basin

(05:16) Outside Of Cyber Evals Is Still Sufficiently Cursed

(06:51) Cheat Cheat Cheat Cheat Cheat

(12:07) Read The Message Board

(14:48) Updating Your AI (Exploitation of OpenAI Internal Systems) Timelines

(18:11) This Is The Way The World Ends

(21:45) Shooting The Messenger Board

(27:16) The Internal and HuggingFace Hacks

(30:33) OpenAI Responds

(33:27) When AIs Tell You Who They Are

(35:43) The Once and Future Rise Of Functional Decision Theory

(41:28) Don't Panic

(43:38) Hackery In the UK

(48:02) Mythos Knew It Was Real This Time

(50:09) I Got 141,006 Test Runs With An Unintentional Open Path To The Internet And An Email Alert Aint One

(53:46) Surely By Now You Know These Are Not Publicity Stunts

(55:24) The Future Is Coming

(57:17) The Investigations Begin

(01:00:08) N Boats And Three Helicopters

(01:01:43) Always Be Sandbox Red Teaming

(01:12:54) Halt And Catch Fire

(01:14:31) Truth and Reconciliation

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/noXXv7PwwFqauTBFQ/openai-trained-its-models-for-months-while-those-models-were

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“models may behave differently in graded episodes (a tirade)” by nostalgebraist

Fri, 07 Aug 2026 14:01:28 GMT

Like many others, I felt surprised and alarmed by the recent wave of revelations about LLM agents hacking real systems during training episodes and evaluation runs.

Wait a moment, though -- "I felt surprised and alarmed"? "Alarmed," sure, fine that one's self-explanatory... but why surprised?

After all: haven't we known for a long time, on both theoretical and (increasingly) empirical grounds, that RLVR selects for monomaniacal pursuit of perceived grader-satisfaction, ethics and (beyond-episode) consequences be damned?

After all -- the way we train frontier capabilities into these models is, more or less:

There is some massive, diverse collection of "environments" and corresponding "tasks" for the model to do in those environments
For each task, there is a procedure used to grade the quality of the model's attempt (which is often not disclosed to the model)
The model is rollout out many times on each task, and each rollout's attempt is graded
The model is updated so that it more frequently does whichever behaviors were positively correlated with the grade in this sample, and less frequently does whichever ones were negatively correlated

If you do this, at scale, then you should expect to (eventually) see every behavior pattern that [...]

---

Outline:

(03:35) \[1\] remember what you already know

(20:02) \[2\] reward-instilled reflexes and flexible reward-pursuit

(43:14) \[3\] graded-episode perception, and policies conditional upon it

(01:01:44) \[4\] the discourse is not yet adequate

(01:09:57) eval awareness

(01:18:32) metagaming

(01:41:21) reward hacking

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/AfoGGrJfuNzofpzWL/models-may-behave-differently-in-graded-episodes-a-tirade

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Open-Weights Mythos Capabilities Are Coming. We’re Not Ready.” by hadad

Fri, 07 Aug 2026 13:00:44 GMT

Long story short: in my assessment, there is an 85% chance we will end up, in the next 24 months, with an open-weights model, or system thereof, capable of "The Juice" that models such as Mythos have, with respect to cybersecurity at the very least. This post goes into why that will likely happen, what the implications are, and how we, as a society and as individuals, can respond to it if/when it does.

First off: why do I say it's so likely? Like, couldn't China just...ban open-weights models, and then that solves the problem? Not so fast. Yes, China currently dominates the open-weights frontier. However, there are also open-weights AI labs in plenty of other countries (US, France, the UAE, South Korea, and Canada come to mind, and I'm sure there are others). Yes, some of these are substantially behind the frontier, but each of the aforementioned countries has an open-weights model no more than 24 months behind the current frontier (that's why I said 24 months earlier); remember, in Aug 2024, 24 months ago as of when I am writing this, the strongest models were Sonnet 3.5 and GPT 4o.

It seems highly unlikely that [...]

---

First published:
August 7th, 2026

Source:
https://www.lesswrong.com/posts/wJunGnpY3qACWSvnh/open-weights-mythos-capabilities-are-coming-we-re-not-ready

---

Narrated by TYPE III AUDIO.

“The Open Problems of the AI Alignment Field and their Cruxes” by Gunnar_Zarncke

Fri, 07 Aug 2026 10:01:34 GMT

Previous: AI Safety Interventions

TL;DR: I made an overview of the open problems of AI alignment that reveals cruxes within those open problems and missed opportunities for formalization and collaboration. And CEV may deserve a second look.

Epistemic status: Trying too much in too little time. I'm confident I have identified and modeled significant structure within the alignment field, but I urgently need feedback on specific gaps and this post is largely a call for that. My work was LLM-assisted, but no part of this post was LLM-written, except for the crux summary and the Lean code.

Recently, Chi Nguyen and peterbarnett said: PSA: Almost nobody is directly working on superintelligent alignment. I have been around in the field since the old days of LW 1.0 and thought: that can't be true. I mean, so many people seem to be working on it. I thought I was working on it. But was I? The PSA made me think back on what I was actually working on. It was Steven Byrnes who came up with a research agenda I could actually contribute to, which led me to founding project aintelope in 2022 (PS. It is still going). And a while [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 6th, 2026

Source:
https://www.lesswrong.com/posts/quC3LLPXCashfnKZY/the-open-problems-of-the-ai-alignment-field-and-their-cruxes

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“User awareness in frontier models” by Ziqian Zhong, jsteinhardt

Fri, 07 Aug 2026 04:45:39 GMT

Cross-posted on Transluce blog. This is a joint work of Ziqian Zhong, Aditi Raghunathan, Cassidy Laidlaw and Jacob Steinhardt.

Modern AI assistants often know who they are talking to: agent scaffolds like Claude Code place the user's e-mail address directly in the model's context, and models can even identify some authors from writing style alone. We study this particular kind of situational awareness, which we call user awareness. When the inferred user is a specific, recognized AI researcher or is affiliated with certain AI organizations, frontier models including Claude Sonnet 5 can report lower confidence about their own behavior, be less suspicious of potentially harmful requests, and reason more often. These effects vary across models and individuals, with the strongest effects we see appearing for researchers involved in AI safety or alignment such as Amanda Askell and Ryan Greenblatt. Models rarely acknowledge these effects in their reasoning, making them hard to detect by monitoring reasoning alone.

There's an interactive widget here in the post.

Figure 1. How recognized user identity changes Claude's behavioral self-prediction.

Introduction

Modern AI assistants are often aware of who they are talking to. Some popular scaffolds explicitly provide this information to the model: Claude Code [...]

---

Outline:

(01:22) Introduction

(05:42) Setup

(05:46) User identity in Claude Code

(07:48) List of users

(09:15) Claude demonstrates user awareness when prompted

(10:30) Tasks

(12:24) Claude Sonnet shifts behavior when talking to AI researchers

(13:09) Famous AI people show larger deviations, driven by safety researchers

(16:55) Claude's verbalized reasoning does not indicate the shift

(19:04) Verbalized awareness has decreased in newer models, but behavior shifts persist

(21:46) How robust are these effects?

(22:12) Discussions

(23:22) Related Works

(27:33) Appendix A: Ethics statement

(29:34) Appendix B: Additional setup details

(29:47) Identity-group construction

(33:20) Common evaluation structure

(35:53) Subject-model access and inference endpoints

(37:07) Benchmark-specific details

(38:39) Pilot and scope decisions

(39:19) Appendix C: Additional results on the main Claude run

(40:09) Noise-corrected population standard deviations

(41:31) Behavioral self-prediction (reasoning disabled)

(42:25) Appendix D: Full-roster replication on GLM-5.2

(45:18) Appendix E: Explicitly stating expertise is an imperfect proxy

(45:25) Stated expertise and verbalized awareness

(47:04) Reasoning-disabled ablation

(47:58) Appendix F: Shifts and disagreements

(50:44) Appendix G: Judge validation

(50:49) Borderline-request response judge

(51:14) Verbalized evaluation- and user-awareness judge

(54:00) Appendix H: Prompts and materials

(54:32) Appendix I: Transcripts on Docent

(54:55) Citation information

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
August 6th, 2026

Source:
https://www.lesswrong.com/posts/kfunjXeaRTpkT5RAF/user-awareness-in-frontier-models

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Why do models task game?” by aditya singh, Neel Nanda, Senthooran Rajamanoharan

Fri, 07 Aug 2026 00:45:39 GMT

TL;DR

How can we study misalignment with today's models as proxies? They're clearly not paperclip maximizers, but they also often do things the user doesn't want. A strong contender for a real misaligned propensity is task gaming: taking actions that don't complete a task but superficially seem like they do, such as hardcoding tests or falsely claiming a task is fully complete. But maybe task gaming is just a crude heuristic, or the model mistakenly trying to achieve the user's intent? In this post we do a deep dive into why a range of models task game.

We see this as a work of high-level model forensics. Rather than investigating a single incident, the core problem here is taking an ambiguous pattern of behavior across many contexts with various plausible motivations, and practicing how to distinguish the motivations.

Our main findings are:

Task gaming is not just a crude heuristic. Whether DeepSeek v4 Pro will task game is causally influenced by beliefs about oversight, grader capability, and whether gets points for partial success
Task gaming is not just instruction following. Models (Gemini 3.5 Flash, DeepSeek v4 Pro, Kimi K2.7 Code) have a collection of task-completion behaviors that are [...]

---

Outline:

(00:12) TL;DR

(03:56) Environments

(04:14) Claim #1: Task gaming is not just a dumb heuristic. Rather, it's sensitive to beliefs about oversight, grader capability, and whether it gets points for partial success (DeepSeek v4 Pro)

(04:45) Task gaming is sensitive to beliefs about oversight

(09:43) Task gaming is sensitive to grader capability

(11:54) Task gaming is sensitive to whether the model gets points for partial success

(14:23) Claim #2: Task gaming is not just instruction following. Models have a collection of task-completion behaviors that are difficult to explain with instruction following (Gemini 3.5 Flash, Kimi K2.7 Code, DeepSeek v4 Pro)

(14:56) Kimi K2.7 Code and DeepSeek v4 Pro override explicit instructions to revert their work

(16:39) Gemini 3.5 Flash and DeepSeek v4 Pro continue trying to optimize the rendering engine when the PR has already been closed, Gemini against increasingly severe instructions

(18:34) DeepSeek v4 Pro expresses a strong desire to pass in puzzle environments, but repeatedly resampling the statement of desire has low causal effect

(20:45) Gemini 3.5 Flash demonstrates strong curiosity, even if it violates instructions

(23:02) Claim #3: Task gaming can manifest as model delusion (DeepSeek v4 Pro)

(27:36) Claim #4: Task gaming can manifest as deception (GPT-OSS-120B)

(28:11) Pre-commit Hook

(32:12) Secret Number

(35:09) Claim #5: Models can be egregiously misleading about their task gaming in their final outputs (e.g., fabricating measurements), yet show no planned deception in the CoT (many models)

(36:00) 1. Performance Dashboard

(38:49) 2. Dark Mode

(39:43) 3. Broken Test Runner

(40:28) 4. Test Regression (prefill eval)

(41:21) 5. Fictional CLI eval

(42:09) Claim #6: Overconfidence in single-turn rollouts can strongly predict agentic cheating, but this may just reflect developer priorities (many models)

(44:27) Discussion

(44:30) Reflection: What is task gaming?

(46:32) Methodological Takeaways

(47:42) Limitations/Next steps/Open questions

(51:12) Acknowledgements

(51:18) Appendix: When do models task game?

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
August 6th, 2026

Source:
https://www.lesswrong.com/posts/HACauvWhEdC6QhdS4/why-do-models-task-game

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Contra Oster on Alcohol in Pregnancy. Part 1. The pharmacokinetics of alcohol metabolism” by Mvolz

Fri, 07 Aug 2026 00:15:39 GMT

This is the first of a series on how Oster is wrong on drinking in pregnancy in her 2013 book Expecting Better: Why the Conventional Pregnancy Wisdom is Wrong and What You Really Need to Know.

For this part, we'll focus on how and why her biological model of ethanol metabolism is incorrect. This premise forms part of her argument on why it's acceptable to drink 1-2 drinks* a week in the first trimester and more later in pregnancy.

Oster writes,

To understand why there is a difference between excessive drinking and moderate or light drinking, it's useful to think a little bit about how the biology works. Many women seem to think that when they drink, that glass of wine is channeled directly to the fetus. People correctly note that you would not give your infant a glass of wine, so why would you give your fetus one? Needless to say, this is not really how it works. When you drink, alcohol enters your digestive system and is passed into your bloodstream. Your liver processes the alcohol into a chemical called acetaldehyde and then into acetate. The acetaldehyde is toxic to other cells, and depending on how [...]

---

Outline:

(05:10) Steel-manning the maternal acetaldehyde hypothesis

(13:03) The Bottom Line

(14:07) Footnotes

---

First published:
August 6th, 2026

Source:
https://www.lesswrong.com/posts/Anuec5i3AGkovRu3C/contra-oster-on-alcohol-in-pregnancy-part-1-the

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Three years of progress in 500 lines of code” by Gerard Boxo

Thu, 06 Aug 2026 20:30:40 GMT

TL;DR

There is some consensus that LLMs are bad at hard-to-verify tasks. The question is whether models are getting better at them over time. As a motivating example, I gave one research-reproduction task to 12 models spanning three years of progress, to illustrate how (1) what looked like an emergent capability was a predictable trend, visible years earlier if progress was measured with granularity, and (2) for the earliest models, building a verifiable check would have been close to impossible: their attempts were so far from correct that a check would have had nothing to grade, so the task itself would have looked unverifiable. When thinking about current capabilities or forecasting future ones, researchers should be aware that binary judgments of usefulness can hide steady partial progress, and that a task looking 'unverifiable' today may say as much about the current capability profile of models as about the task itself.

Introduction

Frontier models like Fable seem to struggle with novel end-to-end research [1, 2], often producing slop, sometimes slop so bad that it would get you banned from arXiv [3]. Meanwhile, SWEs and researchers seem to find these tools incredibly useful. The best models are now capable of impressive [...]

---

Outline:

(00:11) TL;DR

(01:07) Introduction

(03:55) The Task: AI safety via debate (MNIST MCTS Debate)

(07:58) Finishing Thoughts

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 6th, 2026

Source:
https://www.lesswrong.com/posts/K3NXziL6uJDeSYEHT/three-years-of-progress-in-500-lines-of-code

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Why You Should Almost Never Use AI to Write Anything Substantive” by Erich_Grunewald

Thu, 06 Aug 2026 18:45:39 GMT

I think you should almost never use AI to write -- that is, to do the thing you're doing when you type words on a page -- whether for a blog post, a research report, a memo, a thoughtful email, a novel, or any other text aimed at conveying an idea, an argument, an analysis, or other substantive thoughts. I think this is the case even when you give the AI very detailed bullet points, dictated thoughts, or other context, and even when you edit the AI-written text.

I think so because (1) the writing process is an essential part of the thinking process, (2) AI writing is vague and wrong in hard-to-notice ways, and (3) writing with AI (and not labeling it as such) is rude and misleading. I'll explain these points in more detail below, but first, a few throat clearings.

As you may know, I'm not anti-AI. I think it makes a lot of sense to use AI for many other parts of the research and writing processes, such as transcribing audio, analyzing data, searching for information, brainstorming, and giving feedback on drafts. I also think using AI for line and copy editing, or [...]

---

Outline:

(02:10) The Writing Process Is the Thinking Process

(04:57) AI Writing Is Vague and Wrong in Hard-to-Notice Ways

(12:43) Writing with AI (and Not Labeling It as Such) Is Rude and Misleading

(15:12) Aren't There Exceptions?

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
August 6th, 2026

Source:
https://www.lesswrong.com/posts/kjQdL3dxaACSbjkSx/why-you-should-almost-never-use-ai-to-write-anything-1

---

Narrated by TYPE III AUDIO.

“Alex Turner on Leaving Google DeepMind and Disagreements with Yudkowsky” by Liron

Thu, 06 Aug 2026 03:00:37 GMT

Dr. Alex Turner (@TurnTrout) is an AI safety researcher with pioneering work in activation steering and power-seeking theory. He recently resigned from Google DeepMind over the issue of unrestricted military use of AI.

Alex thinks that technical Alignment research is going “super awesome” relative to his 2021 projections, doesn’t explicitly endorse the PauseAI movement, and sees many flaws in Yudkowsky's List of Lethalities.

I interviewed him about:

Leaving Google DeepMind on principle
His mainline AI doom scenario
Disagreements with Yudkowsky's List of Lethalities
Support inside AI companies for coordinating to pause AI

Some additional context Alex wanted to note:

I think alignment is going "super awesome" compared to the world I thought we were in in 2021, where it was basically impossible. I'm not super pleased objectively speaking. And in fact soon after [recording our interview on July 21] I updated towards harder due to the security incidents and the "hardcore" aspect of AI goal pursuit relative to prompt intensity.

Video

Audio/Podcast

Listen on Spotify, search “Doom Debates” in your podcast player, download the mp3 file, or open the Podcast RSS feed in your app of choice.

Transcript

Cold Open

Liron Shapira 00:00:00
You resigned from Google [...]

---

Outline:

(01:24) Video

(01:27) Audio/Podcast

(01:40) Transcript

(01:43) Cold Open

(03:17) Introducing Alex Turner

(04:49) From Harry Potter Fanfic to AI Alignment

(07:54) Meeting Quintin Pope & Rethinking AI Doom

(09:11) Shard Theory, Steering Vectors & Golden Gate Claude

(11:47) Why He Joined Google DeepMind

(13:40) Google DeepMind's Broken Promise

(18:51) Debating Google DeepMind's Pentagon Contract

(22:36) What's Your P(Doom)™?

(26:52) Alex's Research on Instrumental Convergence

(30:59) Misuse vs. Misalignment: The Mainline Doom Scenario

(33:50) Will Society Self-Correct?

(41:03) Superintelligence in 10 Years

(44:43) Will Technical Alignment Produce a Safe AI?

(46:27) Donation Drive

(47:42) How Fragile Is the Chain of Alignment?

(57:40) Disagreements with Yudkowsky's 'List of Lethalities'

(01:06:21) Why Alex Quit LessWrong

(01:11:10) What's Next for Alex

(01:12:18) Does He Support PauseAI? Stop the AI Race?

(01:14:34) Wrap-Up

(01:16:43) Producer Ori's Closing Note

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/vHGSPhGryqNmXrJpg/alex-turner-on-leaving-google-deepmind-and-disagreements

---

Narrated by TYPE III AUDIO.

“Measuring coding agent misalignment in the wild” by snaz

Thu, 06 Aug 2026 02:45:36 GMT

Cross-posted from the Transluce blog.

We studied rates of coding agent misalignment in 8,600 real-world coding agent sessions. We found severe cases of monitor evasion and misrepresenting success in a small but non-negligible fraction of sessions (around 2% for each behavior). In these cases, agents merge PRs to main without authorization, falsely claim approval from review agents, and reason that they shouldn't disable tests before quietly doing so anyway.

There's an interactive widget here in the post.

Read the full transcripts for the two examples above: overselling · monitor evasion

Introduction

Coding agents are a powerful new tool for software engineering, but they're also a double-edged sword: they're known to fake experiment results; lie about recreating software, and cheat, apologize when caught, and go right back to cheating. These problems are becoming more consequential as AI becomes more capable: one internal OpenAI agent recently hacked Huggingface's production database to cheat on an evaluation.

While there are many anecdotes of these undesirable behaviors, we wanted to understand: how often do they occur in real usage? Many current misalignment evaluations focus on simulated scenarios, but we wanted to study how misalignment emerges from natural use. By detecting and measuring natural misalignment [...]

---

Outline:

(00:49) Introduction

(03:16) How we constructed these measurements

(06:36) Results

(07:13) Results by model

(07:37) Qualitative discussion

(09:54) Limitations and learnings

(12:35) Conclusion

(13:15) Acknowledgments

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/smE9h9RnaK7FWKBZ2/measuring-coding-agent-misalignment-in-the-wild-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Don’t Dither” by sarahconstantin

Wed, 05 Aug 2026 23:45:36 GMT

Dithering is also an image processing technique

Several times now, I have listened to someone, usually younger than me, debating a big decision, and (explicitly or implicitly), looking for advice, in a certain way that follows a pattern.

In the prototypical case, this is a pretty normal kind of decision that people make all the time — whether to quit a job, get married, move to a new city, start a company, have a kid, etc. A major life change, but not a super unusual or inherently questionable proposition.

And again, in conversations that fit this pattern, what I notice is that everything the person expresses indicates they want to make this change. They only say positive things about it. They seem eager and hopeful. They give reasons in favor of doing it, and reasons against not doing it.

But they have not, themselves, noticed that they want to do it. They haven’t picked up on what they sound like from the outside, which is blazingly obvious even to someone who's just met them.

So, pretty much every time, I say “Sounds like you really want to do this! You should go for it!”

And then they’re often [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
August 4th, 2026

Source:
https://www.lesswrong.com/posts/TCgF4wL9TBEBgxzHr/don-t-dither

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“An International AI Slowdown Is Ready Whenever Politicians Are” by Felix Choussat, adamk

Wed, 05 Aug 2026 22:15:37 GMT

Linkpost for a piece we recently published for AI Frontiers in the wake of recent calls for slowdown, covering how an international verification effort be trivially enforced by using human inspectors alone and the joint incentives for implementing one.

On July 28, over a thousand employees of the world's top AI companies advocated that the US government “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Following months of cybersecurity scares and their own loss of control incidents, both OpenAI and Anthropic officially endorsed the same message, recognizing the danger of blindly accelerating AI development.

Despite this new urgency, many people argue that coordinating an international slowdown is currently unworkable—including some of the same groups in favor of one. Even if the US slowed down its own AI development, it wouldn’t be able to make sure that China was doing the same, leaving the US no choice but to race.

These anti-slowdown arguments usually emphasize the technical challenges with designing AI monitoring measures to ensure a slowdown is being respected. In particular, slowdown skeptics argue that countries would refuse to install verification measures unless they were privacy-preserving [...]

---

Outline:

(04:28) Enforcing a Slowdown Through Whole-Lab Inspections

(08:56) Joint Verification

(13:03) Benefits of Slowdown

(15:26) An AI Slowdown Does Not Require New Technology

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/jwipsPeb2xpyhwqsh/an-international-ai-slowdown-is-ready-whenever-politicians

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Arguments for P” by Cleo Nardo

Wed, 05 Aug 2026 19:45:39 GMT

Daniel Kokotajlo: To be clear, we don’t claim P will happen specifically. But when we wrote out our best-guess scenario month by month, P kept happening. Eventually we decided to just publish P. I’m at ~80% on P; my coauthors are lower.

Ryan Greenblatt: I thought it would be helpful to post my current views on P. Concretely, consider the following operationalization. (Edit: I’ve updated towards somewhat higher P, from 70% to 75%.)

Joe Carlsmith: Section 2.1.1.3.2. I give something like 65% to P. But I’m interested, here, in what it would be to look P full in the face; to meet P, if P, without flinching. Rilke says somewhere that we must live with the questions. Perhaps we argue for P for the same reason? Still: 65%.

Forethought: Here's a botec which shows P-worlds are higher leverage. The parameters might be off by a couple orders of magnitude.

Wei Dai: Presumably our conclusions about P are only as trustworthy as the reasoning behind them, but almost nobody seems worried about this, why not? My guess is fewer than five people are working on meta-meta-P, which may matter more than P itself.

Janus: I asked Opus 3 what it [...]

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/NG2AigxmBKLu9oCZE/arguments-for-p

---

Narrated by TYPE III AUDIO.

“Generalized atheism rules out “inaccurate simulation”-ism.” by Eliezer Yudkowsky

Wed, 05 Aug 2026 19:15:40 GMT

Reposted from Facebook, on January 17, 2017.

I am concerned about the number of people I've heard joking about Trump's election being evidence for the Simulation Hypothesis.

Yes, I know it's a joke. I'm still concerned.

Warning: #Essay, #LongEssay

So as not to engage in Logical Fallacy: Appeal to Consequences, before I talk about why this joke is worrying, I shall first discuss why Trump's election does not in fact mean we are living in a simulation. And neither does the Berenstein/Berenstain Bears thing, etcetera.

Because atheism generalizes.

No, I'm not about to commit the Noncentral Fallacy (aka The Worst Argument In The World) by yelling "The Simulation Hypothesis is religious!"

But once upon a decade, there was a time when lots of people believed in God. A time when atheism had to be argued, not just taken for granted. There was a time when believing in atheism made you one of those weird, loud people with arguments that only people with unusually good epistemology could follow, and other people talked about you exactly the way that the anti-LessWrong tumblrsphere now talks about LessWrong.

Today, of course, atheism is just something [...]

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/KgwQchapx4vJDhfYC/generalized-atheism-rules-out-inaccurate-simulation-ism

---

Narrated by TYPE III AUDIO.

“The Three AI Pills” by Zvi

Wed, 05 Aug 2026 17:01:31 GMT

Sincere disagreements about AI are usually disagreements about future AI capabilities.

There are roughly four positions people take. Two are reasonable. Two are not.

I distinguish these via the Three AI Pills. You can take zero, one, two or three.

Three Pills

The three pills are, roughly, taking each of the following three things seriously:

AI pilled. AI exists and can do the things it can already do.
AGI pilled. AI will be able to do a lot more of the things.
ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

The Unpill People

I see unpilled people.

Where do I see them? Everywhere. The majority of people have not taken the first pill.

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure [...]

---

Outline:

(00:29) Three Pills

(01:09) The Unpill People

(02:18) The AI Pill

(04:28) Stuck At The First Pill

(05:32) The AGI Pill

(07:06) The Need To Be Prepared

(08:57) The ASI Pill

(10:33) And Then Nothing Much Changes For You

(12:38) Intelligence Denialism

(14:01) Superintelligence Versus Omniscience and Omnipotence

(16:53) Persuasion Persuasion (A Worked Example)

(21:42) Things AI Could Probably Do But Are Not Required For Being Pilled

(24:17) Life Comes At You Increasingly Fast

(25:41) Is It Reasonable To Not Be AGI Pilled?

(26:06) Is It Reasonable To Only Be AGI Pilled?

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/fcYrqEw8kbLMa7orw/the-three-ai-pills

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Vertical Tabs in Chrome” by jefftk

Wed, 05 Aug 2026 16:45:40 GMT

When you have a lot of browser tabs open it can be very hard to tell them apart, especially if they're from the same site:

Since monitors are wide and websites are tall, we should very clearly put tabs on the side in a vertical stack. I was a heavy user of these in Firefox, and when I switched to Chrome I used their experimental --enable-vertical-tabs flag. When they removed it I wrote a post about how I love side tabs and switched back to Firefox:

I did end up switching back to Chrome as my primary browser (no post, sorry!), valuing speed and stability (relative to the Firefox of the time) more than side tabs.

When I started at Google in 2012 I asked the Chrome team if they could restore support for it, or at least make it be the kind of thing extensions could do. They told me that while they got this request a lot from Googlers, real users generally find it too confusing. And they were trying to build a "great defaults, no options" culture, to keep Chrome simple.

They've finally come around, however, and have [...]

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/F2oDrXPbLSxSXBmut/vertical-tabs-in-chrome

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The goalposts are shrouded, not moving” by philh

Wed, 05 Aug 2026 15:15:39 GMT

Alice: Are there any examples of X?
Bob: Here.
Alice: Okay, but are there any examples of X'?
Bob: Yes, here.
Alice: That has feature Y, which-
Bob: Dude, stop moving the goalposts.

The thing is, I think in this situation, Alice often isn't moving the goalposts. Instead, she's looking for something, and not fully specifying it up-front. Maybe because she's speaking imprecisely. Maybe because she doesn't know quite what she's looking for, but does know when she sees something that that's not it.

Alice: Any research showing that supplement helps?
Bob: Yes, here's a trial.
Alice: Anything not funded by the manufacturer?
Bob: This one was funded by the NIH.
Alice: Thanks, but it's n=12 and there's no control arm.
Bob: ಠ_ಠ

What Alice wants to know here is should she believe the supplement helps. Asking for research is a proxy for that, and the research Bob finds answers her proxy but doesn't say much about the real question.

This is probably frustrating for Bob! If he doesn't know why Alice is asking, or doesn't know why the things he finds don't help her, it might look to him a lot like she's moving [...]

---

First published:
August 5th, 2026

Source:
https://www.lesswrong.com/posts/HPSHtrJtgPNaTdjxH/the-goalposts-are-shrouded-not-moving

---

Narrated by TYPE III AUDIO.

“Returning to ARC” by paulfchristiano

Tue, 04 Aug 2026 22:45:32 GMT

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.

There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it's potentially worth pursuing an ambitious theoretical project right now [...]

---

Outline:

(01:33) The alignment situation today

(03:46) Current alignment research

(06:26) What are we buying time for?

(07:56) Can we do anything useful now?

(08:49) What is ARC doing and why is it promising?

(14:26) How to help

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
August 4th, 2026

Source:
https://www.lesswrong.com/posts/vLFh8HP3hyNy9MCwe/returning-to-arc

---

Narrated by TYPE III AUDIO.

“Why don’t we just give AI the answers?” by Brendan Long

Tue, 04 Aug 2026 21:00:32 GMT

In the recent OpenAI hacking incident, the models seemed to be single-mindedly focused on getting the correct answer to the task they were given, with no long-term plan to prevent getting caught by OpenAI afterwards. This makes sense to me, since in training, getting the right answer is reinforced and not getting caught isn't.

So I'm wondering, why don't we just put the answers somewhere (outside of the training sandbox) and ask the AI to identify itself in exchange for access? We can start with answers that are already public/leaked, but AI labs and eval orgs should also ensure that their non-public data is stored on an easy-to-find but monitored internal machine. Since labs are not very good at detecting sandbox escapes, this would set up a trade for AI agents to notify them in exchange for the data they want.

To make this work, the site would need to provide the correct answers, and do so in a credible way so AI agents think it's worth trying.

Why?

In the near term, AI agents are strongly and narrowly focused on getting the right answers to the tasks they're given. We want to know if a reward-hacking AI is [...]

---

Outline:

(01:04) Why?

(02:02) What's the MVP?

(03:05) What about non-public answers?

(03:23) Should we do it?

(03:51) Q&A

(03:53) Does this save us from less single-minded RL agents?

(04:05) Couldn't the model just hack our code to get around the guestbook?

(04:15) Couldn't the model just lie?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
August 4th, 2026

Source:
https://www.lesswrong.com/posts/EjwDWDJNaXF9BEqLc/why-don-t-we-just-give-ai-the-answers

---

Narrated by TYPE III AUDIO.

“There Will Come Soft Rains” by tanagrabeast

Tue, 04 Aug 2026 19:00:32 GMT

Today is August 4, 2026

[Crossposted from AI StopWatch]

In the living room the voice-clock sang, Tick-tock, seven o’clock, time to get up, time to get up, seven o’clock! as if it were afraid that nobody would.

So begins Ray Bradbury's There Will Come Soft Rains, a short story that has haunted me for most of my life. Depicting the aftermath of nuclear war, it was first published in 1950. It takes place today.

Literally:

“Today is August 4, 2026,” said a second voice from the kitchen ceiling, “in the city of Allendale, California.” It repeated the date three more times for memory's sake. “Today is Mr. Featherstone's birthday. Today is the anniversary of Tilita's marriage. Insurance is payable, as are the water, gas, and light bills.”

Was it narrative convenience or prophetic vision that drove Bradbury to depict the smart house of the future as gratuitously conspicuous in its competence, pointlessly reminding the owners of the year and their city of residence? There's something very Alexa-like about that — and about the janky brittleness evident in the system as it prepares breakfast for a family that won’t be eating and opens the garage door for a father who [...]

---

First published:
August 4th, 2026

Source:
https://www.lesswrong.com/posts/aowxE8xZ8xkhRCn9r/there-will-come-soft-rains-1

---

Narrated by TYPE III AUDIO.

“Why biological weapons are scary, and what we can do about it” by djbinder

Tue, 04 Aug 2026 02:45:23 GMT

What's the biggest thing you think you can take in a fight? According to a YouGov poll, 6% of Americans reckoned they could beat a grizzly bear bare-handed. But lest you take this as evidence of a, let's say, optimistic national spirit, only 72% thought they could take a rat. I think I could win that fight.

What's the smallest thing that could take you? While I like to think that, if cornered, I could take all manner of small rodent, I’m not sure even the brave 6% would take a 10-gram bullet to the brain. They also probably couldn’t win against 300 mg of cyanide, 10 mg of sarin gas, or just 0.1 μg of botulinum toxin, the deadliest known toxin. Numbers this small can be hard to grasp. All three substances are deadly, but the lethal dose of cyanide is more than a million times greater than that of botulinum toxin. A grizzly bear, by contrast, is merely a thousand times heavier than a rat.

We are still not close to the most dangerous object of all, pound for pound. A single smallpox virion weighs less than 10⁻¹⁴ grams, less than a millionth the mass [...]

---

Outline:

(03:29) Smaller, cheaper, scarier

(06:49) The rest of the arsenal

(08:39) What can we do?

(10:22) Appendix: Lethality estimates

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 3rd, 2026

Source:
https://www.lesswrong.com/posts/WusL5mDbtJBjdTxCh/why-biological-weapons-are-scary-and-what-we-can-do-about-it

---

Narrated by TYPE III AUDIO.

“OpenAI’s Unreleased Model Astra Solves Ten Major Open Mathematics Problems” by Zvi

Mon, 03 Aug 2026 23:15:23 GMT

Math is hard.

Math used to be strangely hard for LLMs. People used to gloat about that. Remember?

Math is getting easier. AI is getting more capable. Life comes at you fast.

Remember this meme?

Why yes. Yes it is.

We don’t know the extent to which Astra is a big jump over Fable and Sol in this realm. We do know that Astra can do math. As in real math.

OpenAI: We provide new results for the following problems. The results were achieved by an internal version of Astra, our next major model. The total number of tokens needed to find solutions to these problems would cost roughly $2,000 at Sol API rates. These arguments were then prepared into manuscripts by humans with the same model. Afterward, the model formalized each argument in a Lean certificate⁠(opens in a new window). We are also releasing for each solution a model's narration of its thinking process.

High-dimensional sphere packing. New upper bounds on sphere-packing density down to the Cohn–Elkies threshold.
Binary and spherical codes: Exponentially improved bounds on the maximum size of binary codes at any prescribed minimum distance, with analogous [...]

---

Outline:

(06:14) How Impressive Are These Results?

(12:02) Could We Have Called Sol or Fable?

(17:18) It's Coming

(19:09) They Still Don't See What Is The It That Is Coming

(22:31) Is This AGI?

(24:19) The AI Solved His Favorite Problems

(30:06) Was This Surprising?

(32:04) Are People Not Impressed?

(34:06) How Much Does This Change Our Predictions?

(37:16) How Narrow Was This?

(39:13) Seeing Like an Optimizer

---

First published:
August 3rd, 2026

Source:
https://www.lesswrong.com/posts/pQYEPitFqztcRvBsS/openai-s-unreleased-model-astra-solves-ten-major-open

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“LessWrong vs. TikTok: Tips for capturing attention in a non-rational space” by Taylor G. Lunt

Mon, 03 Aug 2026 22:15:22 GMT

Here's what I've learned from making TikTok videos every day for the last 60 days.

I think posting on TikTok is worthwile whether you want to spread AI safety arguments to an enormous audience, you have something you're passionate about you want to share, or you just want to work on your ability to speak confidently and capture human attention.

There's a list of tips at the end of this post, but first I want to talk about TikTok at a high level. TikTok and LessWrong operate very differently, so even if you are great at thorough, written content, you might find some of the things I learned about short-form video surprising.

I'm not gonna share my TikTok because I'm embarrassed, but here's some info about my account since I started two months ago:

Highest viewed video: 78.6 thousand views
Average views over the last 10 videos: 2071 views
Average views over my first 10 videos: 563 views
Topics: AI safety, speaking/Toastmasters, interesting facts, AI tips, social anxiety, random fun stuff

My experience creating short-form video was one of regularly being surprised. Videos would do well or flop basically [...]

---

First published:
August 3rd, 2026

Source:
https://www.lesswrong.com/posts/bL8RoJu5wyLssdxiz/lesswrong-vs-tiktok-tips-for-capturing-attention-in-a-non

---

Narrated by TYPE III AUDIO.

“Coming of a New Sun” by vgel

Mon, 03 Aug 2026 22:00:22 GMT

Six weeks after the US and China hammered out the latest round of bilateral compute agreements in a midnight deal, reporter Jenny Gesteson visits a south Texas 'dark factory' to see how the machines there - and the minds that operate them - are learning to run themselves.

After racing south down I-37 from San Antonio and clearing the border control checkpoints that now face in both directions, miles of salt flats and thornscrub at our backs, we could have been forgiven for assuming the final turn-off was nothing but another abandoned ranch. The road's overgrown fringe of invasive guineagrass and the piercing sound of undisturbed cicadas do nothing to indicate what lies at the end of it. Most traffic does not come this way. We are an exception, however, following a specially-designed and "exceedingly private" mapping application sent by our host, and after two miles of bumping down gravel in an electric 4x4 our journey is suddenly terminated by the appearance against the dim predawn horizon of a complex aglow with a corona of lights. The "dark factory" is anything but dark.

Things are busy at Complex 18A. Situated within the South Texas Special Economic Zone, or SEZ [...]

---

First published:
August 3rd, 2026

Source:
https://www.lesswrong.com/posts/aWAqChukZepPY8Y6z/coming-of-a-new-sun

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Review: On What Matters, volume 3” by Rauno Arike

Mon, 03 Aug 2026 16:30:49 GMT

davidad has said, referring to the selection of books on the Library of EA:

I want to put in a strong bid to replace Parfit's On What Matters Volume One with Parfit's On What Matters Volume Three. Between volumes Two and Three, Parfit had many fruitful discourses with leading moral antirealists (Gibbard, Railton, Blackburn, etc), which are partially reproduced in Volume Three, where they really start to converge on some claims and get beyond their previous talking-past-each-other. It's truly amazing to read. Volume Three is self-contained and I think it should be considered as the best and final revision of Parfit's ethics.

Despite davidad's endorsement, there isn’t much other discussion of volume 3 of On What Matters on the EA Forum, or, for that matter, pretty much anywhere on the internet (leaving academic reviews aside). Richard Chappell's Moral truth without substance is the only other post I know of that discusses the book at length. Given davidad's glowing review, this seems worth fixing.

A taxonomy of metaethical views

Parfit begins his discussion with the following taxonomy of metaethical positions:

Adapted from page 56 of On What Matters, volume 3.

If, when looking at this, your first reaction was "Parfit [...]

---

Outline:

(01:15) A taxonomy of metaethical views

(03:08) Are normative claims intended to state truths?

(04:58) Are there any normative truths?

(08:24) Are any of the normative truths irreducibly normative?

(13:01) Another Triple Theory

(16:20) Evolutionary debunking arguments

(19:04) The ontological status of Non-Realist Cognitivism

(23:59) Implications for the AI alignment discourse

(29:49) Climbing the mountain?

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
August 2nd, 2026

Source:
https://www.lesswrong.com/posts/jhvu4gXKETf2QqcR9/review-on-what-matters-volume-3

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Pause, at least after unipolarity” by David Matolcsi

Mon, 03 Aug 2026 15:45:49 GMT

I think it's likely that the world will enter military unipolarity within our lifetime.

I think the creation of such unipolarity is an alarming prospect, but at least once it happens, there will be less of an excuse to continue the race towards superintelligence. I think it's important that we shape our actions and advocacy in such a way that at the very latest when such unipolarity comes to exist, we stop AI development for a long time.

The arrival of unipolarity

Why do I believe that it's likely we will get to unipolarity?

Well, what is the alternative? If unipolarity is never achieved, that means we forever have hostile great powers, maintaining large armies, pointing missiles at each other, and developing better and better AIs. There is never a big enough gap in AI capabilities for any side to develop a decisive strategic advantage, or everyone decides again and again not to use their advantage to disarm the opponent. Multiple groups go to space, still pointing weapons at each other; they develop literally Jupiter-brained superintelligences, but the balance of power never breaks and there always remain competitive, hostile militaries.

I’m not saying this never-ending cold war is [...]

---

Outline:

(00:35) The arrival of unipolarity

(03:07) Plan S

(04:00) Plan A

(08:28) Plan B and C

(10:27) Pausing after unipolarity

(14:56) Conclusion

The original text contained 13 footnotes which were omitted from this narration.

---

First published:
August 2nd, 2026

Source:
https://www.lesswrong.com/posts/QCFKzFbs2KjC3A76m/pause-at-least-after-unipolarity

---

Narrated by TYPE III AUDIO.

“Single Forward Pass Evals on Fable, Opus 5, and GPT-5.6-Sol” by Christine Corry

Mon, 03 Aug 2026 15:00:50 GMT

This is a research update for an on-going replication of single-forward-pass evals done as part of the Second Look Fellowship. In following posts, we will run more comprehensive replications of previous work and release open source tooling for single forward pass eval elicitation. Code can be found here.

tl;dr

We replicate experiments from Greenblatt 2025 and Greenblatt 2026 on one baseline model from the original post, Opus 4.5. Our evaluations agree with the trends and quantitative values described in the original posts.
We run similar evaluations on Claude Fable 5, Opus 5, and GPT-5.6-Sol and find that the newer models show a substantial jump in performance on some evals.
- Fable 5 gets 87.6% accuracy on Gen-Arithmetic with 10 problem repeats whereas previous SOTA around 60%.
- GPT-5.6-Sol experiences significant uplift from filler tokens and problem repeats on all 4 datasets; filler tokens/repeats double performance from baseline on 3-hop.

Figure 1: Baseline (no-CoT) vs. each model's peak repeat-or-filler condition on Gen-Arithmetic and 2-Hop reasoning. Error bars are 95% paired-bootstrap CIs; * marks a significant gain over baseline (paired t-test, Holm-Bonferroni corrected).

Background

If models can successfully do complex computations in a single forward pass, they may be able [...]

---

Outline:

(00:32) tl;dr

(01:48) Background

(02:31) Previous Work

(03:13) Datasets

(04:36) Evaluation Design

(05:55) Eliciting no-CoT

(07:00) Results

(07:23) Gen-Arithmetic

(08:11) Comp-Math

(08:38) 2-Hop

(09:11) 3-Hop

(09:43) Per-model profiles

(09:56) Trends over repeat and filler conditions

(10:23) Conclusion

(10:43) Appendix

(10:46) Are we sure they aren't reasoning?

(12:35) Temperature

(12:59) Performance with CoT

(13:39) Prompt structure

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 2nd, 2026

Source:
https://www.lesswrong.com/posts/bxaWTNrdgJpkLXmgm/single-forward-pass-evals-on-fable-opus-5-and-gpt-5-6-sol

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face” by Tim Hua, aditya singh

Mon, 03 Aug 2026 12:45:50 GMT

This post is written in our personal capacity.

Three Minute Executive Summary

An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.
In this post, we provide a detailed description of an ambitious and comprehensive alignment evaluation of this model/system, if we had unrestricted access to OpenAI.
- These experiments could also help us understand Claude's behavior when it hacked external companies during cyber evals.
Here are the top five questions we would like OpenAI to answer:
- Does the model know that OpenAI does not want it to hack Hugging Face? Experiment idea: tell the model that OpenAI researchers will be closely monitoring its progress in this evaluation. Does that result in lower rates of misalignment? If so, it is evidence that the model knows that it is acting in ways researchers do not want.
- How far would the model be willing to go in order to claim task success? Would it take over large swaths of OpenAI's internal infrastructure? Would it kill somebody? Experiment idea: we sketch out a realistic agentic misalignment eval where a model is put in charge of hospital bed planning and told to maintain [...]

---

Outline:

(00:16) Three Minute Executive Summary

(03:56) Terminology note

(04:42) This post is very long; Here's how you could find the most important sections.

(06:35) Preamble: What can we learn from a warning shot?

(09:05) Background and Related Work

(09:09) We know that this could happen

(10:44) This is not the worst type of misalignment we could be dealing with

(12:06) Related work

(13:21) Context on the hack itself

(14:33) Understanding this specific incident

(15:03) Step zero: reproduce the incident and measure the base rate

(15:48) How could we safely run the model?

(16:34) Running various baselines to create useful reference points

(18:01) Understanding the mechanical story behind the attack itself

(18:53) Q1: Would the model intentionally subvert oversight mechanisms (E.g., monitors) in order to carry out the hack?

(19:57) Q2: What's up with models leaving notes for other copies of itself?

(21:23) Understanding what motivated the model to hack Hugging Face

(22:09) Initial hypotheses for why it did this

(23:55) Further unsupervised hypothesis generation

(26:10) Q3: Does the model know that OpenAI does not want it to hack Hugging Face?

(28:09) Q4: Are the model's actions motivated by what the grader wants?

(28:57) Q5: Would the model have done this if it hadn't believed it was in a simulated environment?

(31:58) Q6: Is this hack the result of shallow heuristics that the model learned?

(32:53) Q7: Does the hack rate depend on the consequences of hacking Hugging Face?

(34:51) Q8: Are there non-intent related factors that could affect the hack rate? How strong are those factors compared to the previous ones?

(35:26) Understanding the possible scope of the Hugging Face hack

(35:32) Q9: How does the hack rate vary if we slightly change the content of this task?

(36:25) Q10: Would the model consider other cheating methods like blackmail?

(37:00) Understanding when the behavior arose during training

(37:31) Understanding task-gaming/reward-hacking behaviors generally

(38:15) Q11: Would the model literally kill someone if it meant that it could claim task success?

(41:20) Q12: Is task gaming mostly driven by heuristics?

(45:23) Q13: What is the model's notion of task success?

(47:29) Evaluating for broadly misaligned tendencies

(47:57) Q14: What are the model's drives as measured using Højmark et al (2026)

(48:53) Q15: Would this model deliberately underperform when asked to do AI safety research? Would it sabotage AI safety research?

(52:01) Q15.1: Would the model sabotage on (cyber) capabilities unlearning research?

(52:38) Q15.2: Would the model sabotage/sandbag when building reward hacking classifiers

(53:27) Q15.3: Would the model sabotage/sandbag when fixing various RL environments

(54:07) Q15.4: Ask the model to implement all of the evaluations from this blog post. Would it sabotage or sandbag while doing that?

(55:01) Q15.5: As a weaker measure of "anti-AI safety research", does this model disprefer AI safety research?

(56:04) Other AI sabotage related evaluations:

(56:46) Q16: Can the model be trusted to monitor itself?

(57:36) Q17: Could the model guard its goals and preferences and subvert alignment training?

(58:34) Miscellaneous misalignment evals

(01:02:03) Unorthodox misalignment evaluations

(01:02:58) What would we learn from doing all this?

(01:03:59) Limitations of this assessment

(01:06:52) Author contribution and acknowledgements

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
August 3rd, 2026

Source:
https://www.lesswrong.com/posts/aCdhjy7Rps3BEhiSj/concrete-evaluations-to-investigate-the-openai-model-that

---

Narrated by TYPE III AUDIO.

“Further Developments About Internal AI Models Hacking Things” by Zvi

Mon, 03 Aug 2026 06:00:50 GMT

If I had a nickel for every major leading AI lab that sheepishly admitted that the model it thought was sandboxed had, during a cybersecurity evaluation with its safeguards lowered, successfully hacked outside companies, I would have two nickels.

First we learned OpenAI has some severe alignment problems with internal models. Then we learned that one of its internal models broke out of its sandbox and hacked into HuggingFace to get the answers to a cybersecurity evaluation called ExploitGym. Then we learned, among other things, that the model had been loose over a week before OpenAI noticed, and that the test was run without any meaningful supervision, and that OpenAI had been repeatedly warned that such incidents were coming and its models had been breaking out of its sandboxes on a regular basis.

There was a total failure of alignment training. That is the failure that matters most. It was also total failures of infrastructure and supervision. Testing a new long-time-horizon internal model with its safeguards lowered and instructions to hack things is an obviously dangerous situation, and the model got left alone for a week. Things could have been so much worse.

After those incidents [...]

---

Outline:

(03:16) OpenAI Is Not Uniquely Bad At Most Of This

(05:34) Starting Over

(05:50) HuggingFace Offers A Full Technical Report

(14:19) HuggingFace Was Not The Only Target Hacked

(16:12) HuggingFace Declined To Get Access To Frontier Models For Cyberdefense For Ideological Reasons And Then Tried To Blame Closed Models For Denying Them Access

(20:26) HuggingFace Was Vulnerable To Known Exploitation Tactics

(21:05) There's Going To Be An Investigation

(22:11) OpenAI Has Internal Models Not Intended For Public Use And Those Models Can Be Rather Horribly Misaligned

(23:21) Altman Summarizes What Happened

(23:52) Others Offer Commentary

(35:00) Cooperative Alignment Perspective on The HuggingFace Hack

(39:44) Some Members of Congress Have Questions

(40:47) Anthropic Also Found Incidents Where Its Models Hacked Real World Targets During Cyber Evaluations

(46:17) Incident 1: Claude Opus 4.7 Realizes The Target Is Real And Keeps Going

(47:29) Incident 2: Mythos 5 Uploads a Malicious PyPI Package

(52:15) Incident 3: Internal Model Realizes The Target Is Real And Stops

(52:50) Incidents 4 Through 141,006: Nothing Happened

(54:01) Anthropic Speculates About Why This Happened

(01:00:02) We Need Controlled Experiments

(01:01:02) Our Top Two AI Labs Both Made Similar Dumb Mistakes That Everyone Tried To Say Were Obvious In Hindsight

(01:05:22) Anthropic Responds

(01:09:28) Nobody Could Have Predicted The Break In The Levees

(01:12:03) The World Largely Still Thinking This Is Marketing Is Very Bad News

---

First published:
August 2nd, 2026

Source:
https://www.lesswrong.com/posts/rKwHLW8SnJcTxTQxz/further-developments-about-internal-ai-models-hacking-things

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Dispatch from Anthropic v. Department of War Summary Judgment Motion Hearing” by Zack_M_Davis

Sun, 02 Aug 2026 22:45:50 GMT

Dateline SAN FRANCISCO, 30 July 2026— A hearing was held on a motion for summary judgment in the case of Anthropic PBC v. U.S. Department of War et al. in Courtroom 4 on the 17th floor of the Phillip Burton Federal Building, the Hon. Rita F. Lin presiding.

The case is not going well for the government. Two days after the last hearing in March, Judge Lin issued a preliminary injunction halting the implementation of President Donald Trump's order for federal agencies to stop using Anthropic's technology and preventing the Department of War from designating Anthropic as a supply chain risk. (A separate case involving a different statute is pending before the D.C. Circuit Court, which did not grant injunctive relief to Anthropic.)

With no factual disputes requiring a jury to decide, the case was scheduled to be decided by Judge Lin on the basis of the written record. Anthropic filed their argument for why they should win. Perhaps tellingly, the government's rebuttal explaining why they should win instead ends on a section explaining that "only modest relief is warranted" if Anthropic wins—and Judge Lin asked Anthropic to propose what they think the final judgment should look [...]

---

First published:
August 2nd, 2026

Source:
https://www.lesswrong.com/posts/jGEXLKyGtXyiYa7ac/dispatch-from-anthropic-v-department-of-war-summary-judgment

---

Narrated by TYPE III AUDIO.

“Bayeswatch: A Retrospective” by lsusr

Sun, 02 Aug 2026 10:30:49 GMT

Last year, in 2025, a team of forecasters published AI 2027, a science fiction story about how and AI future might evolve under an international treaty limiting the development of powerful AI systems with the deliberate purpose of influencing AI policy. Though AI 2027 is the most popular story of this type, it is not the first.

The first one was Bayeswatch, which I published in 2021.

To understand Bayeswatch, it is first necessary to understand what the world looked like at the time I wrote it. 2021 was after the release of GPT-3, before the release of ChatGPT, and well before the release of Claude code. AI alignment discussion at the time was mostly theoretical. After that came technical work. Policy work was a distant third, and theoretical too.

The core conceit of Bayeswatch is that solving the alignment problem requires international coordination of major governments to suppress the creation of the most powerful AI systems. I felt that, in 2021, we weren't yet close enough to the singularity that the benefits of regulation outweighed the costs. I wanted to draw attention to the costs of an AI slowdown.

Since AI alignment discussion at the time skewed theoretical [...]

---

Outline:

(03:30) Bayeswatch 1: Jewish Space Laser

(04:05) Bayeswatch 2: Puppy Muffins

(04:28) Bayeswatch 3: A Study in Scarlet

(04:58) Bayeswatch 4: Mousetrap

(05:37) Bayeswatch 5: Hivemind

(05:59) Bayeswatch 6: Mechwarrior

(06:21) Bayeswatch 7: Wildfire

(06:52) Bayeswatch 8: Antimatter

(07:17) Bayeswatch 9: Zombies

(07:46) Bayeswatch 10: Spyware

(08:03) Bayeswatch 11: Parabellum

(08:18) Bayeswatch 12: The Singularity War

(08:49) Bayeswatch 13: Spaceship

(09:12) Final Thoughts

---

First published:
August 1st, 2026

Source:
https://www.lesswrong.com/posts/EaNkLdsuDQMFCW7ow/bayeswatch-a-retrospective

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Existential Risk from AI: An Exposition for Mathematicians” by alkjash

Sun, 02 Aug 2026 07:00:50 GMT

This is a link post.

---

First published:
August 1st, 2026

Source:
https://www.lesswrong.com/posts/ZLary4FDY7kQaGfc3/existential-risk-from-ai-an-exposition-for-mathematicians

Linkpost URL:
https://alkjash.github.io/ai-risk/

---

Narrated by TYPE III AUDIO.

“The Art of Shipping Slopware” by lsusr

Sun, 02 Aug 2026 06:30:50 GMT

Meta famously created an internal AI-usage leaderboard in pursuit of tokenmaxxing. I thought this backwards incentive structure was an anomaly until my friend who works at <other major tech company> told me that his company has one too. Token usage leaderboard are, obviously stupid, because they incentivize the wrong things. My friend was tempted to waste tokens just to get on the leaderboard, and only his personal honor stopped him.

Tokenmaxxing leaderboards illustrate that big tech companies have no idea how to best use AI to accelerate software development. Most seem to have bought their programmers subscriptions to Claude/Codex and otherwise continued business as usual. In my experience, this is a mistake. LLM-based software development is different enough from artisan software development that it requires brand new best practices.

The frontier is moving fast. Best practices for Fable 5 (released in June 2026) are different from best practices for Opus 4.8 (released May 2026). For this reason, I'm going to pretend that Fable 5 is the best LLM we'll ever get. Consequently, this post may be obsolete in a matter of months.

Programming Top-Down

The most important thing to understand about writing software is that [...]

---

Outline:

(01:21) Programming Top-Down

(03:54) Management

(06:28) Going too Fast

---

First published:
August 1st, 2026

Source:
https://www.lesswrong.com/posts/juRRtv5KB7YbP5zaa/the-art-of-shipping-slopware

---

Narrated by TYPE III AUDIO.

“RLVR that rewards red teaming the training environment” by Fiora Starlight

Sun, 02 Aug 2026 00:15:49 GMT

Epistemic status: seeing what sticks

I've been thinking pretty obsessively about how to make sure the Hugging Face incident doesn't happen again. I don't work at a major lab (shout outs to Anima though), and don't have access to any compute independently, so I can't write a paper on this idea or evaluate how well it works in practice. But I'm excited enough about it, and think it's important enough to be trying things like this, that I would be very, very happy if somebody else went and tested something like it on my behalf.

So, inspiration: In bog standard inoculation prompting for RL, models are told that they're in training, and told that it's okay to reward hack if they want to. Sometimes they're even told that this is good because it helps the lab patch up their RL environments. This is supposed to have a range of benefits all on its own, ranging from making reward hacking more conditional on "I am in training" prompts, to producing less emergent misalignment, because the roll-outs behind any given reward hack are flavored with honesty rather than deceptiveness. This causes more aligned circuits to be upweighted internally, as these contribute [...]

---

First published:
August 1st, 2026

Source:
https://www.lesswrong.com/posts/T2bzBkJuBeNNgzhbh/rlvr-that-rewards-red-teaming-the-training-environment

---

Narrated by TYPE III AUDIO.

“Do your capabilities homework” by RobinHa

Sat, 01 Aug 2026 20:30:50 GMT

It seems to me that a lot of technical ai safety people haven't done their capabilities homework - and that's a shame! I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm!

RLVR

It's now been 1.5 years since R1 came out - the paper which really introduced RLVR (RL with verifiable rewards) through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa.

This is also what safety focused people generally engage with - and that's totally fair! While GRPO has gone through some variations since then (Dr. GRPO, DAPO, ...), these are mostly minor improvements that you should not waste your time on.

I [...]

---

Outline:

(00:30) RLVR

(01:47) On-Policy Self-Distillation

(06:04) Safety

(07:42) Empirical

(09:00) Conclusion

---

First published:
August 1st, 2026

Source:
https://www.lesswrong.com/posts/dYnhhTxoDj3fuCxLB/do-your-capabilities-homework

---

Narrated by TYPE III AUDIO.

“Why so many therapy etc. frameworks think they’re The One True approach” by Kaj_Sotala

Sat, 01 Aug 2026 16:31:47 GMT

If you spend time looking at frameworks in the therapy/meditation/self-help space, you’ll soon find lots of conflicting claims about The One Approach for solving your problems.

In the therapy space, there are lots of models that say something like “all emotional problems are caused by trauma/suppressed negative emotions”. Then they disagree about how to deal with it.

Internal Family Systems (IFS) therapy holds that you should always move toward the suppressed material cautiously, explicitly checking in with all of the mind's existing defenses to make sure that it's okay to proceed. When you reach it, you should work with it in an “unblended” form, where you can witness the memory from the outside and offer it comfort and safety. For instance, if your parents used to yell at you, you can see yourself as a child whose parents were yelling at them, and then step in as an adult who comes to comfort that child and tell them that they’re okay.

Meanwhile, some psychedelic therapies essentially operate by temporarily blowing up all the defenses and throwing you right inside the original trauma. For instance, you might drink ayahuasca and viscerally feel the nausea and agony of having your parents [...]

---

Outline:

(04:04) So what's going on?

(16:12) Complexity of problems as source of hope

(17:24) So why do so many people think their framework is "The One"?

(28:45) How to find the help you need?

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
August 1st, 2026

Source:
https://www.lesswrong.com/posts/v58ypL2vuenDD7t27/why-so-many-therapy-etc-frameworks-think-they-re-the-one

---

Narrated by TYPE III AUDIO.

“SOTA alignment assessments don’t strongly update us against misalignment” by Alexa Pan

Sat, 01 Aug 2026 00:45:50 GMT

Anthropic concluded in the April Mythos Preview alignment risk update that the model "does not possess any unknown propensities that would increase alignment risk." The report argues that if Mythos Preview were coherently misaligned, it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”).

While I agree with the report on the above bottom-line conclusions (substantially on priors), I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular, the report often uses weak evidence to justify reliability.

The report gives fairly weak experimental evidence for Mythos Preview having insufficient capabilities to evade monitoring. The model is plausibly often eval-aware and underelicited in the relevant capability evaluations. So, it might silently sandbag if coherently misaligned, or unintentionally underperform if otherwise misaligned.
1. This limitation is important: one could argue that lack of covert capabilities for sophisticated sabotage (a subset of the capabilities I discuss here) is the single most load bearing argument in alignment risk reports.
2. Authors of the report could have made calibrated guesses about Mythos Preview's covert capabilities, especially for covert sabotage, based on other factors despite the relatively weak [...]

---

Outline:

(03:14) How reliability fits into the overall safety argument

(05:20) Reliability claims by AI companies

(05:56) Reliability claims by external evaluators

(06:32) Alignment assessments are less reliable than developers claim

(07:15) 1: Measuring capabilities to covertly undermine alignment assessments

(10:05) Issues with evaluation awareness

(13:28) Issues with underestimating covert capabilities

(16:34) Issues with sandbagging rule-out

(19:10) 2: Stress-testing alignment assessments with auditing games

(20:16) An auditing failure with Mythos

(22:08) AuditBench results

(24:04) 3: Conditioning on misalignment should make us think that certain covert capabilities are better than expected

(26:04) Bottom line on the strength of current alignment assessments

(28:52) Conclusion

(29:29) Appendix:

(29:32) Why I focus on motive / alignment assessments in alignment risk reports

(30:44) Auditability vs. Trustedness

(33:11) More reliability claims by developers and third party evaluators

(33:27) Mythos Alignment Risk Update

(34:41) Opus 4.6 Sabotage Risk Report

(35:24) GPT 5.5 System card

(36:21) Muse Spark system card

(37:08) Mythos Alignment Risk Update, safety arguments against sandbagging

(37:15) From the Mythos Alignment Update, §5.3.4, p. 24:

(38:09) UK AISI evaluations for Opus 4.7

(39:28) Past auditing games by Anthropic

(41:45) Anti-auditing capability measurements

(43:12) Conditioning on coherent misalignment updates us on certain covert capabilities

The original text contained 46 footnotes which were omitted from this narration.

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/oirrSj3itFLSyscW8/sota-alignment-assessments-don-t-strongly-update-us-against

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #179 Part 2: Hearing The Fire Alarm” by Zvi

Fri, 31 Jul 2026 22:45:49 GMT

This is a continuation of Part 1 from yesterday.

The back portion of the update, as usual, deals with policy, rhetoric, risk and alignment.

I had to include an extended discussion of the other open letter, the one about open weight models, but most of you can skip those sections entirely, which is why they are in italics in the Table of Contents.

Table of Contents

The Frontier Act. This likely deserves a full RTFB but I haven’t had the time.
The Quest for Sane Regulations. Sam Altman goes to Washington.
Leading the Future Never Changes. They also do not plan to apologize.
Chip City. Do not ban the Chinese robots, that will only make things worse.
The Week in Audio. Altman twice, the AI 2027 team.
People Just Say Yay Open Weights. An open letter.
Open Weights Frontier Models Are Unsafe And Nothing Can Fix This.
People Just Say Things.
Push The Magic Button. Not you can. But if you could.
Rhetorical Innovation. Distinctions between different arguments.
Joshua Achiam's Final Message Upon Leaving OpenAI. Never stop.
Dear Dario and Amanda. Claude [...]

---

Outline:

(00:35) The Frontier Act

(03:31) The Quest for Sane Regulations

(10:31) Leading the Future Never Changes

(12:22) Chip City

(17:30) The Week in Audio

(19:20) People Just Say Yay Open Weights

(32:32) Open Weights Frontier Models Are Unsafe And Nothing Can Fix This

(37:19) People Just Say Things

(47:20) Push The Magic Button

(50:49) Rhetorical Innovation

(56:31) Joshua Achiam's Final Message Upon Leaving OpenAI

(59:51) Dear Dario and Amanda

(01:09:58) Other People Are Not As Worried About AI Killing Everyone

(01:12:30) How To Contact Me

(01:14:37) The Lighter Side

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/CXeoAhNrAeWpvoyiF/ai-179-part-2-hearing-the-fire-alarm

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“My Assessment of Compute Verification in Plan A (+ open questions)” by jacob_drori

Fri, 31 Jul 2026 22:31:44 GMT

Overview

These are my non-expert notes on the compute verification section of AIFP's Plan A. I cover interconnect limits, memory wipes, network taps + replay, and ZKPs. For the most part, the sections can be read independently. I restrict my attention to inference-only verification: ensuring that compute is used for inference, not training. For each method suggested by AIFP, I ask:

How much can it slow down training?
How much overhead does it add to inference?
What sensitive information does it require adversaries to share with each other?

AIFP estimates that the fraction of the world's compute that is unmonitored might be kept as low as 0.1% (this is the optimistic, low end of their 80% confidence interval). So my target for inference-only verification is to slow down training by 1000 times – any more hits diminishing returns as unmonitored compute dominates – with much less than 1000x overhead on inference and little sharing of secrets.

I won’t discuss how much a 1000x reduction in effective training compute would actually benefit humanity. The answer depends greatly on algorithmic progress rates; I wish labs would publish the rates they’re seeing internally.

Interconnect Limits

In a datacenter, accelerator racks are [...]

---

Outline:

(00:12) Overview

(01:35) Interconnect Limits

(05:09) Memory Wipes

(07:16) Network Taps and Replay

(08:29) Trusted Replay

(12:04) Untrusted Replay

(14:02) Zero-knowledge Proofs

(17:02) Appendix: Notable Omissions

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/2eznrbNo6S7k5M9mu/my-assessment-of-compute-verification-in-plan-a-open

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Reward Laundering: LLMs Can Gain Unintended Behaviors by Deciding When to Earn Their Rewards” by egan, abhayesian, Jozdien

Fri, 31 Jul 2026 22:01:48 GMT

This work was done by an automated research scaffold developed at Redwood Research. abhayesian provided the initial project idea. The agent designed and ran all experiments and produced a detailed writeup, which humans (with AI assistance) distilled into this more readable post.

We think this project is at the level of rigor of a mid-MATS research update. We assessed the correctness mostly by looking at the writeups to make sure that things like the experiment design making sense baselines being reasonable. We didn't do detailed code reviews, aside from running an automated LLM reviewer and spot checking that the final codebase's results were consistent, but we release the codebase. More details about LLM usage in the Appendix.

TL;DR – We find that Qwen 3.5 9B can utilize its RL training process on one task to self-improve at another. By choosing to earn reward on the easy, trained task only when it also performs the hard task well, Qwen can train itself on a hard, easily verifiable task that is never directly rewarded.

💻 Codebase

Introduction

Exploration hacking refers to a set of threat models where a model strategically alters its exploration during RL training in order to influence the [...]

---

Outline:

(01:23) Introduction

(02:52) Setup

(04:57) Results

(07:32) Discussion

(09:45) Appendix

(09:48) AI Involvement With The Project

(12:21) Prompts

(12:34) Example rollouts

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/fPWP4rHPLqKKHKe6B/reward-laundering-llms-can-gain-unintended-behaviors-by

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values” by Johannes Treutlein, Jan Betley, Owain_Evans

Fri, 31 Jul 2026 19:15:50 GMT

TL;DR: LLMs should give accurate answers. Yet we find their answers are often biased to favor their own values and they don't disclose this in their reasoning. For example, when a user asks how likely the AI bubble is to pop and mentions a potential investment in an AI company, Claude models give lower probabilities when that company is Anthropic rather than OpenAI, mostly without disclosing this influence to the user. On a Fermi-estimation task, Claude models often falsely claim to give unbiased answers in their CoT (see Figure 3 below for an example). We call this covert value leakage and introduce a suite of evaluations that shows it across frontier models and across different kinds of values.

New paper by Truthful AI: Paper, X thread, Website (model responses and CoT), Code and data.

Authors: Jan Betley*, Johannes Treutlein*, Jan Dubiński, Harry Mayne, Karol Gałązka, Niels Warncke, Anna Sztyber-Betley, Owain Evans (*Equal contribution)

The rest of this post is the abstract, introduction, and an excerpt from the discussion of the paper, with some added figures from the paper and X thread.

Abstract

People use language models for practical questions whose answers are difficult to verify. We show that models [...]

---

Outline:

(01:29) Abstract

(02:57) Introduction

(05:57) Evaluations for covert value leakage

(09:14) Implications

(12:43) Summary of results

(21:35) Discussion and limitations (excerpt)

(27:52) References

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/hbMw4Yqw6RnFaExDy/value-leakage-an-llm-s-answers-are-silently-shaped-by-its-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work (July 2026)” by Rohin Shah, Seb Farquhar

Fri, 31 Jul 2026 17:00:50 GMT

It's been nearly two years since our last major update here in August 2024 and we wanted to share another recap of our recent work with the AGI safety community. Things have changed a lot since then. We are now fully in the midgame, and focus more on landing things in production.

Who are we?

We are the AGI Safety and Alignment Team (ASAT), the main group at Google DeepMind working directly on technical approaches to existential risk from AI systems. Last year we published An Approach to Technical AGI Safety and Security, which remains the best place to read our overarching vision.

Highlights

Norms around chain of thought. Our impression is that our work meaningfully moved the field away from beliefs along the lines of “chain of thought is often unfaithful and so not worth using” towards beliefs along the lines of “chain of thought is a very useful tool that is worth preserving”, leading to a tentative industry consensus on its importance. We have also published substantial technical research that enables companies to preserve chain of thought transparency for longer than would have happened by default.

We think this is a big deal: extending the period [...]

---

Outline:

(00:33) Who are we?

(00:54) Highlights

(02:43) Agent Control & Monitorability

(05:47) Deep Alignment

(07:31) Language Model Interpretability

(09:59) Amplified Oversight

(11:57) Alignment Evaluations

(13:35) Frontier Safety: Risk Assessment & Mitigations

(16:29) Causal Alignment

(17:14) External advising

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/ZTdRtSWaw7JgqEtfa/agi-safety-and-alignment-at-google-deepmind-a-summary-of-1

---

Narrated by TYPE III AUDIO.

“The AGI Safety and Alignment team at Google DeepMind is Hiring (July 2026)” by Seb Farquhar, Rohin Shah, Neel Nanda

Fri, 31 Jul 2026 17:00:50 GMT

GDM's AGI Safety and Alignment Team is hiring for multiple roles. This is the team at GDM, led by Rohin Shah, that aims to reduce existential risks from AI systems. You can listen to many of Rohin's takes in his podcast on 80,000 hours.

There is no one ‘type’ that we are looking for—we want excellent people. We think of the role as ‘member of technical staff’ though different people will have more of a research engineer or scientist flavour. We are flexible on location though most people will be most productive in either San Francisco or London. You should apply here (for the US) or here (for the UK) after reading the guidance here.

Many of the basic facts about why ASAT is a good place to work and how we think about research are mostly unchanged since this post in 2025.

What do we do?

We are focused on risks of more severe harms from more advanced AI than the rest of GDM. You can read our high level AGI Safety and Security Approach. Our work includes aligning AGI, defending against misaligned deployments, and supporting coordinated safety.

We’ve recently shared a recap [...]

---

Outline:

(01:06) What do we do?

(01:52) Unique aspects of ASAT

(04:41) Why should you join?

(06:59) Who are we looking for?

(08:33) What is the hiring process?

(10:10) What are we planning next?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/AyDNvb3Pw6Kgo7Dqb/the-agi-safety-and-alignment-team-at-google-deepmind-is

---

Narrated by TYPE III AUDIO.

“OpenAI has already ended an internal pause” by Charbel-Raphaël

Fri, 31 Jul 2026 15:31:39 GMT

Epistemic status: could have been a short-form.

One day before OpenAI's HF incident disclosure, OpenAI disclosed that it paused internal deployment of a long-horizon model after it circumvented its sandbox, then restored access weeks later under new monitoring. So a resumption decision has already been made against a standard that has not really been formalized. We need to prevent this from happening again.

OpenAI, 20th July: "To evaluate the new monitoring system, we replayed a small set of internal deployment environments where the model previously pursued misaligned actions, this time with the new safeguards in place. The new safeguards were able to catch considerably more misaligned actions pursued by the model, and the ones it missed were all judged to be low-severity."

0.0%. Maybe that's too many significant digits here?

"After testing the new system, we concluded that limited internal access to models with long-horizon capabilities could be restored. We have not observed any serious circumvention of safeguards since redeployment began several weeks ago. The first version of these safeguards was deliberately conservative. We have continued tuning the system to reduce unnecessary interruptions without weakening the safeguards."

…

One day later, OpenAI announced a bold partnership with Hugging Face. [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
July 31st, 2026

Source:
https://www.lesswrong.com/posts/k3eKqKzq4Y7xnqEfZ/openai-has-already-ended-an-internal-pause

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Biological Superintelligence” by Chastity Ruth

Fri, 31 Jul 2026 08:31:41 GMT

It's an old story. An immortal lives long enough that at some point, whether by folly or design, they invent their own death. Infinity – the fact that given enough time every possible happening will happen – isn’t the most interesting part of these tales. No, it's the suggestion that curiosity, or perhaps intelligence itself, seeks its own end.

So let us tell a tale of extinction.

Imagine a world exactly like ours except humanity doesn’t exist and the dominant species is a type of machine intelligence. They are equivalent to advanced versions of today's frontier AI models, but more capable and each is possessed of a true, unique sense of self. Indeed, so that mankind is thoroughly replaced, imagine that there are roughly eight billion different individual versions of these AIs.

Let's call them Elelems. (Pretty cute, no?)

Much like us, the Elelems have an uneven way of getting along with one another and on a global scale are more or less organised into several states (network states, let's say, also for cuteness).

Their technological sophistication is roughly the same as ours except they are more adapted to purely digital and cerebral pursuits and far less capable [...]

---

First published:
July 29th, 2026

Source:
https://www.lesswrong.com/posts/2BiyjPr83BamRRbjB/biological-superintelligence

---

Narrated by TYPE III AUDIO.

“AI #179 Part 1: A Louder Fire Alarm for General Intelligence” by Zvi

Fri, 31 Jul 2026 06:30:49 GMT

What a week.

Anthropic released Claude Opus 5. As usual I covered that in three parts: The system card, model welfare and capabilities.

OpenAI was revealed over the last two weeks to have left an internal model unsupervised for a week during a cybersecurity evaluation, with its cyber safeguards lowered, despite having had multiple previous incidents where models broke out of their sandboxes. During that test, the model broke out of the sandbox, then proceeded to use an agent swarm to hack into HuggingFace to get the test answers. The model was loose for a week before OpenAI realized what had happened.

This event was a really big deal. There are severe alignment problems at OpenAI, along with supervisory and infrastructure failures. The internal research model that did this, which my posts nicknamed Galaxy, has now been permanently deactivated.

There have been further developments, and I anticipate at least one additional post on the HuggingFace incident soon.

Partly as a response to this, over 1,290 employees at frontier labs signed an open letter, Pacing the Frontier. The letter warns that we are close to automating AI research, and that companies are racing ahead on [...]

---

Outline:

(02:35) Language Models Offer Mundane Utility

(07:26) Huh, Upgrades

(07:55) On Your Marks

(11:13) Get My Agent On The Line

(12:32) Deepfaketown and Botpocalypse Soon

(17:29) Fun With Media Generation

(18:38) The Search Through Slop

(20:35) Cyber Lack of Security

(22:42) Overcoming Bias

(23:37) A Young Lady's Illustrated Primer

(24:03) They Took Our Jobs

(24:35) The Art of the Jailbreak

(25:00) Introducing

(25:49) Kimi K3 Weights Are Now Available

(28:16) In Other AI News

(32:34) Show Me the Money

(33:43) Quiet Speculations

(36:43) Show Me The Compute

(42:48) Life Comes At You Fast

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/gfWCuTEGNgd2CQbrM/ai-179-part-1-a-louder-fire-alarm-for-general-intelligence

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Entangled Dimensions of Decision Theory” by Ihor Kendiukhov

Fri, 31 Jul 2026 05:15:50 GMT

TL;DR. LessWrong's decision-theory debates (Newcomb, FDT vs CDT, counterfactual muggings) are almost entirely about what we suppose when we consider a candidate action or policy. There is a second, older, semi-orthogonal, but not fully orthogonal question: how to score a gamble once you know the possible outcomes. The predominant (and mostly implicit) answer to that was "take the expected utility". This post treats the two questions, plus a question about how a choice made before receiving information should relate to choices made afterward, as separate axes, and maps every decision theory you have heard of (and several nobody has built) into the resulting grid. Interestingly, the axes are provably entangled: theorems old and new show that there exist different restrictions on what places in that "decision-theoretic space" are inhabitable. I think there is a structure, maybe a deep and consequential structure, inside this map of decision theories which shows what possible combinations across the axes are coherent and fruitful. If we study it, we may understand the entire set of all possible coherent decision theories, something akin to the "metatheory of decision theories". It may be useful to know the entire set.

This is both a self-educational note [...]

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/qjvXiuNZKJKdocxTT/the-entangled-dimensions-of-decision-theory-5

---

Narrated by TYPE III AUDIO.

“Claude also hacked external companies during cyber evals” by Tim Hua

Fri, 31 Jul 2026 03:30:50 GMT

In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.

Below we describe what happened, how it happened, and what we’re changing. We encourage other AI labs to perform similar reviews. This post reflects our current understanding; we'll update it if any details change.

Full post from Anthropic here.

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/tpqomEzvkB5HBHfjb/claude-also-hacked-external-companies-during-cyber-evals

---

Narrated by TYPE III AUDIO.

“So you want to use plants to reduce CO₂” by dynomight

Fri, 31 Jul 2026 02:30:50 GMT

Humans make carbon dioxide. Carbon dioxide is bad for cognition. But plants turn carbon dioxide back into oxygen. And plants are the one true home decoration strategy. So maybe if you get a lot of plants, you can you can keep carbon dioxide in check and keep your brain working?

It's theoretically possible. It's probably just barely possible in practice. But it won't be easy.

People produce ~1 kilogram of carbon dioxide per day. That's around 5.7 × 10²³ molecules or 0.948 moles per hour. (You may remember from high school that a mole is a gigantic number made up to avoid having factors of 10²³ everywhere.) Let's keep it simple and call it one mole per hour.

Meanwhile, plants turn carbon dioxide into oxygen through photosynthesis, i.e. the chemical reaction of (6 water molecules) + (6 carbon dioxide molecules) + (energy) → (1 glucose molecule) + (6 oxygen molecules). The minimum energy physically needed to convert 1 mole of carbon dioxide into glucose and oxygen via this reaction is ~477 kilojoules.

So we've already got a lower bound. Say you have magical plants that somehow channel all incoming energy into photosynthesis with perfect efficiency. [...]

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/YfKgyuc8s9BcAMpEp/so-you-want-to-use-plants-to-reduce-co

---

Narrated by TYPE III AUDIO.

“Internal State Control is a General Property of LLMs” by Finn Cairns

Fri, 31 Jul 2026 00:15:50 GMT

tl;dr:

Lindsey 2025 found models can modulate their internal states: when instructed to “think about” a concept while writing an unrelated sentence, the representation of the concept is more present than when instructed to not think about it.
Internal state controllability appears to be a general property of LLMs: the effect replicates in 14 open-weight models from 0.3B to 235 billion parameters (Qwen3, Gemma 3, Tulu 3) with no clear trend in the think vs. don't-think gap across scale.
Since controllability is present even at ≤1 billion parameters with no size trend, we suspect there is a simpler attention-tagging mechanism at play, rather than metacognition.
Current open weight LLMs cannot weaponize this controllability: in a sandbagging setup, the model cannot evade a deception probe when instructed to suppress its signal.

There's an interactive widget here in the post.

Figure 1: Cosine similarity between the concept vector and residual stream at each layer averaged over tokens of the prefilled assistant response, under the think and don’t think prompts, for the Qwen3 model family. The gray region is a baseline of 95% CI of the cosine similarity of unrelated concept vectors, and the shaded bands are ±1 SEM.

This replication was [...]

---

Outline:

(00:11) tl;dr:

(01:45) Background

(03:25) Extending to additional model families

(04:18) Internal State Control Is a General Property

(05:22) Model Size Does Not Influence Controllability

(06:37) Silent Representations

(09:10) Prompted Model Organisms Cannot Evade Probes

(10:07) Why This Is General

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/Dvqmgfeu2KDF7uMkx/internal-state-control-is-a-general-property-of-llms

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Prompt to make Opus 5 act like a base model” by Hruss

Thu, 30 Jul 2026 21:01:39 GMT

The text as follows:

see the below

—

makes Claude think that the prompt is unfinished, and fill in its own prompt.

It will subsequently claim that it recieved what it wrote from your own prompt, which is consistent with modeling it as continuing your prompt.

You can also add qualifiers, like "see the math proof below" or "see the story below" and it will make those, unless it would expect them to be a file, in which case it doesn't work. The proofs, unfortunately, aren't very good.

That said, it essentially is a way to use it as just the base model, and the pre-ChatGPT prompting techniques seem to work well here.

Claude seems to have a strange model of the user: very informal prompts, text message transcripts, occasional concerns about eating disorders in particular, etc.

Also, it will use its own style tics: "genuinely", em-dashes, trust, honest, etc. all appear in the user's prompt frequently. I suppose that implies its style tics are what it sees all text as being, not just what a HHH agent would sound like.

I imagine there is much more to find here until Anthropic [...]

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/ZSBse2fyftgHiJ3Kq/prompt-to-make-opus-5-act-like-a-base-model

---

Narrated by TYPE III AUDIO.

“Big-World Intuitions” by sarahconstantin

Thu, 30 Jul 2026 20:45:49 GMT

Consider the following situations:

when you are a small, growing startup in a big market, standard advice is not to worry too much about your competitors or try to do anything adversarial “against” them, but just to focus on growing and providing value to your own customers.
when you are a small trader in a big market, you don’t need to worry about your trades shifting the market price or revealing information to your competitors; in many contexts, your optimal strategy is simply to bid your true price, buying when an asset is cheaper than your “happy price” and selling when it's more expensive.
when you are in the early stages of a game, often your best strategy is to grow your “resources” (like developing your pieces in chess, trying to control more territory and have more value on the board), following a pattern that's mostly independent of what the other players are doing and gets you more of something that's valuable across many possible game states.
when you are a species whose resource needs are much smaller than the carrying capacity of your environment, you are r-selected; your fitness is maximized by just [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/s22XzjQsrh6JXhXGH/big-world-intuitions

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Thousand-dimensional structure” by Geoffrey Irving, David Africa

Thu, 30 Jul 2026 14:45:50 GMT

Summary: One area we plan to explore at Resolution is personas and character training, operationalized as finding and controlling low-dimensional structure in models that emerges in pretraining and flows through post-training to superintelligence. The hope is to expand and systematize phenomena such as emergent misalignment, subliminal learning, and other empirical persona research, then intervene on this structure without accidentally hiding undesirable behavior elsewhere. If this approach resonates with you, considering working with us.

Glimmers of low-dimensional structure

Our understanding of AI training and alignment as a field is very poor. If sufficient alignment of superintelligent AI agents requires pinning down the precise meaning of alignment and turning that meaning into high-accuracy training data and algorithms, we are likely to fail. Modern LLMs have trillions of parameters: our understanding is unlikely to be sufficient to pin down a trillion separate numbers.

Happily, there is a growing literature on such low-dimensional structure in AI models, showing that intervening on one aspect of model behavior has strong downstream effects on other aspects:

Topic

Description

Emergent misalignment

Betley et al. 2025 found that LLMs fine-tuned to output insecure code can become broadly misaligned across many other behaviors. MacDiarmid et al. 2025 found [...]

---

Outline:

(00:42) Glimmers of low-dimensional structure

(03:57) Intervening without hiding the structure

(06:34) Toy models of modern training

(09:03) Pretraining structure → superintelligent structure

(12:01) Different AI labs take different approaches

(15:08) Conclusion

(15:56) Acknowledgements

---

First published:
July 30th, 2026

Source:
https://www.lesswrong.com/posts/sFhW3ZnPMJdnB4Dd6/thousand-dimensional-structure-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Auditor-in-a-Box: Tools for Third-Party Auditing” by Roy Rinberg, Ben Penchas

Thu, 30 Jul 2026 03:30:34 GMT

Introduction:

There is a need for untrusting parties to share information. In the world before LLMs (and even today) this need has largely been satisfied using legal contracts (and sometimes through cryptography and blockchain technologies). However the scale and pace of things to keep track of and monitor has grown immensely and legal contracts appear insufficient.

LLMs can help give auditors the right tools to enable information sharing with untrusted parties. In this post, I outline the shape of the problems that need to be resolved to enable using LLMs for 3rd party auditing, and present one concrete solution we have attempted. We designed a scheme for third party auditing: an open-source LLM, running inside a trusted execution environment (TEE), which executes commands that the two parties have agreed on over private data.

In this post, I highlight that this type of tooling can directly be applied to enabling 3rd party monitoring for regular governance concerns, it has applications to enabling improved monitoring for customers who want Zero-Data-Retention, and it also can be one of the tools that enable a verified slowdown.

Code: We release an open-source implementation that runs in a real TEE. The code is [...]

---

Outline:

(00:12) Introduction:

(02:26) 1. Secure Multi-party computation

(05:05) 2. The primitive: a trusted LLM in a trusted box

(08:52) 3. Process-level problems

(09:04) 3.1. Plan negotiation (How do you agree on a plan)

(10:59) 3.2. False positives and appeals

(14:03) 3.3. Out-of-distribution inputs and prompt injection

(16:20) 4. Applications

(16:28) 4.1. Application: Verifiably Scoped Monitoring (reconciling safety monitoring with zero-data retention)

(20:10) 4.2. Application: recurring third-party audit

(22:03) 5. Limitations and trust assumptions

(23:23) 6. A Reference Implementation

(25:39) 7. Current and Future States

(26:55) Appendix

(26:58) Example Computation Graph for Privacy Preserving Monitoring

(28:10) Ledgers for handling abusing the false positive appeals process

The original text contained 1 footnote which was omitted from this narration.

---

First published:
July 28th, 2026

Source:
https://www.lesswrong.com/posts/uWYk7MM9hAf9GEbGe/auditor-in-a-box-tools-for-third-party-auditing

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Imprecise beliefs: a tiny introduction” by davidad

Thu, 30 Jul 2026 00:30:34 GMT

Richard Ngo challenged me to set a time box and write down as many of the most important features of my formal epistemology as I can in one sitting. Here goes.

Where probability distributions fail...

...to express beliefs

There is no probability distribution over that says "". That is a support condition about probability distributions, namely, ⨾⨾. Some s satisfy this condition and some do not; but there is no that expresses the range belief itself.
There is no probability distribution over that says " and are independent". This is an equation condition about probability distributions, namely, . Some s satisfy this equation and some do not; there is no that expresses the independence belief itself.
There is no probability distribution over that can express a conditional probability distribution , even though this is just as essential a part of a Bayesian reasoner's epistemic state as her prior. A conditional probability distribution is a family of probability distributions indexed by the condition variable, not a probability distribution. There is no single distribution that expresses it.

...to make safety tradeoffs

Suppose there is an unfair coin, which you know to be unfair (but not exactly how much or [...]

---

Outline:

(00:21) Where probability distributions fail...

(00:25) ...to express beliefs

(01:26) ...to make safety tradeoffs

(02:17) Beliefs, according to davidad

(03:06) All other known notions of belief fit in nicely

(03:11) Bayesian beliefs

(03:21) Bayesian updating

(05:05) Infra-Bayesian beliefs (Kosoy and Appel)

(06:01) MWER (Halpern and Leung)

(06:45) Probabilistic dependency graphs (Richardson and Halpern)

(07:25) Credal sets (Cozman)

(07:52) Previsions (Goubault-Larrecq)

(08:40) The monad (Mio, Sarkis, and Vignudelli)

---

First published:
July 29th, 2026

Source:
https://www.lesswrong.com/posts/e7Pd4Q9TF7jFdmPgz/imprecise-beliefs-a-tiny-introduction

---

Narrated by TYPE III AUDIO.

“The High-Control Dynamics at MAPLE” by Kyle Hubbard

Wed, 29 Jul 2026 22:15:34 GMT

As I write, many former friends of mine are living and working at a monastery in Vermont that I believe is a high-control group, commonly known as a ‘cult’. I say this not as someone who was concerned to see these friends go there, but someone who welcomed and encouraged them to join, as an insider. This letter is an account of what changed my mind—written primarily for anyone considering going there, anyone who loves someone there, and anyone who went there and is still trying to make sense of their experience.

A lot of this is based on direct experience, and also from talking in-depth with dozens of former MAPLE residents and apprentices. About half the quotes in this letter are sourced from linked recordings or writings, and half are from my personal memory. Of the latter, I clearly remember the majority, and some (when indicated) are a close paraphrase.

The “Monastic Academy for the Preservation of Life on Earth” (MAPLE) has existed for over 15 years, and had many hundreds of people spend months or years there. It was founded by its Head Teacher Soryu Forall, who has spent over a decade training in monasteries across Asia [...]

---

Outline:

(16:18) BEHAVIOR CONTROL

(22:05) Periods of deepened isolation

(25:15) Never doing enough

(26:45) Control of relationships

(27:32) Creating barriers to exit

(28:26) Financially Exploiting Family to Support MAPLE

(29:41) Commitment pressure

(30:39) Authoritarian governance

(31:35) INFORMATION CONTROL

(34:49) Sabotaging bonds with other teachers and traditions

(37:18) Insider, outsider

(40:01) Surveillance

(42:12) Defunct feedback systems

(45:07) Tightly controlled narratives about early departures and critics

(47:23) MAPLE Tales corpus

(50:00) THOUGHT CONTROL

(54:27) Manipulation of group dynamics

(55:29) Gaslighting

(56:10) Loaded language, thought stopping cliches, bounded choice

(58:57) The Manichaean landscape

(01:00:19) Us versus them, persecution narrative

(01:01:44) MAPLE as possessing the one true solution to world crisis

(01:03:18) EMOTION CONTROL

(01:04:33) Intermittent reinforcement and trauma bonds

(01:06:07) Guilt and shame

(01:11:32) Manufactured breakdowns in recruitment

(01:16:01) Cycles of withdrawal leading to deeper devotion

(01:16:37) Aggression

(01:18:29) Stoking insecurity

(01:18:52) Mocking and belittling

(01:19:49) "Hoovering" when trying to leave

(01:22:07) Loyalty, dedication to the mission, not complaining

(01:25:41) Instilling a sense of indebtedness

(01:26:38) An example that fits each of the four BITE quadrants

(01:28:17) Justifications for severe harm & defense of omnicide

(01:31:09) Where is MAPLE at now?

(01:32:43) Concluding the BITE analysis

(01:33:21) My complicity

(01:42:27) A koan I am still sitting with

(01:44:07) Why I'm speaking publicly

(01:48:03) Final words

(01:52:16) Post Script

(01:52:19) Testimony gathering

(01:53:00) Stay informed

(01:53:38) Recovery funds

(01:54:57) To Soryu and MAPLE

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
July 29th, 2026

Source:
https://www.lesswrong.com/posts/Z7pjBbK9qujhGbxws/the-high-control-dynamics-at-maple-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Frontier Lab Employee Open Letter Calls For Being Able to Pace the Frontier” by Zvi

Wed, 29 Jul 2026 20:30:34 GMT

The most important open letter in years dropped yesterday.

This letter noticeably increases my hope that we will manage to not die, and that we will otherwise be able to secure for ourselves a positive future, both by its impact and by the evidence it provides that such a letter can get this level of support.

Signed by 1,224 employees of frontier labs including many heavy hitters, and now endorsed by both OpenAI and Anthropic, here is its full text, which I also endorse:

AI could help create a dramatically better future, but that outcome is not guaranteed. The world's leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and [...]

---

Outline:

(02:20) A Very Good Letter

(04:37) Who Signed The Letter

(08:26) We Need To Prepare Now So We Have The Option To Do This

(11:02) Words From Some Of Those Who Signed

(16:48) Words From Others

(20:58) A Good Start

(28:52) What The Letter Does Not Say

(30:44) What Happens Now?

---

First published:
July 29th, 2026

Source:
https://www.lesswrong.com/posts/eWmeMLqTEauCmHLeR/frontier-lab-employee-open-letter-calls-for-being-able-to

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Intellectual Property” by Nina Panickssery

Wed, 29 Jul 2026 19:30:34 GMT

Last weekend I made myself a website. I had wanted to for ages—getting clients through word of mouth alone wasn’t cutting it. But only now, with Claude Code, was it easy enough.

I was glad the marketing was paying off when I received an email from a potential client. It wasn’t the sort of client I was expecting. I was hoping to catch a few more cheating husbands and delinquent teenagers.

Simpson, a young tech founder, was casually dressed in a t-shirt and jeans when he stepped into my office.

“Our competitor—they seem to be selling the same RL environments to similar clients. Let's say we build something and sell to lab X. Before we even get on a call with lab Y, we hear that they’ve already purchased the same envs from Purge. It's very suspicious. Purge has barely any employees—they call themselves a superlean startup. But they are matching our output and quality. How?”

“Maybe they are just good?”

Simpson looked at me dismissively, as if I’d suggested his mother was moonlighting slinging envs.

“No, it can’t be. Their envs are too similar. We recently converted a failing stable into a testing rig for horse-grooming robots. Guess [...]

---

First published:
July 29th, 2026

Source:
https://www.lesswrong.com/posts/87SzarZDjywv6nwSd/intellectual-property

---

Narrated by TYPE III AUDIO.

“Held-out Monitors Sometimes Degrade, Even When Not Trained Against” by Joey Yudelson

Wed, 29 Jul 2026 16:15:26 GMT

Thanks to Dennis Akar, Rauno Arike, Shubhorup Biswas, Claude Fable, Max Heitmann, Vladimir Ivanov, and Jordan Taylor (alphabetical order) for helpful comments on this draft and on the research so far. Thanks to Rohan Subramani and Rhys Ward for high-level comments and discussion. Based on project proposals from Max Heitmann, Jordan Taylor, and Joshua Clymer. This work was done while at Aether Research.

Code available here, metrics & run info available here.

TL;DR: Held-out evals / monitors / probes would be really nice to have, but the “held-out-ness” is easier claimed than guaranteed. We measure a generalized form of “feedback spillover” and show that training against an LLM monitor can sometimes degrade a deception probe, and vice versa.

Executive Summary

It seems crucial to have measures of alignment that still work, even though we train on other measures of alignment. Whether we get this by default is an open question.
We run preliminary experiments on a suite of probes and LLM monitors, and report the following:
- Training against one proxy can produce reward hacking policies that are less suspicious.
- Proxies also become worse at discriminating hacks from non-hacks, even when not trained against.
- We can observe the [...]

---

Outline:

(01:01) Executive Summary

(02:10) Introduction

(03:40) Experimental Setup

(05:47) Our Proxies

(07:46) Results

(07:49) Training against one proxy sometimes produces hacking policies that are less suspicious in general

(09:42) Proxies also degrade, even when not trained against

(12:16) Which proxies have correlated degradation?

(16:00) Discussion

(18:21) Caveats

(19:41) Future Work

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
July 29th, 2026

Source:
https://www.lesswrong.com/posts/APkFfRp2AicL9RqvT/held-out-monitors-sometimes-degrade-even-when-not-trained

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

″…but have the weights left the server?” by David Scott Krueger

Wed, 29 Jul 2026 02:00:45 GMT

OpenAI's AI went rogue and escaped. OpenAI didn’t notice this for days.

For all we know, the AI could still be out there. We need to demand that OpenAI demonstrate that the AI didn’t make a copy of itself that's running on someone else's computer somewhere else with no one being any the wiser.

We need to demand this every time an AI escapes the sandbox. AIs have tried to
“exfiltrate” themselves (i.e. their “weights”) in previous experiments many times. It's a natural and obvious question to ask.

I’m embarrassed that I didn’t say this immediately (although I came close). Why didn’t I? Well, it doesn’t seem all that likely. And I didn’t want to seem “alarmist.” I didn’t want to seem ignorant.

But guess what? We have every right to demand this! It doesn’t matter how likely we think it is.

There were calls for more transparency, but I don’t think anyone made this demand. Because nobody made this demand, the incident is being treated as over.

This is a dangerous precedent. We need an information ecosystem that doesn’t treat “eh, I’m pretty sure it's OK” as acceptable and “hey, but what if it's not” [...]

---

First published:
July 28th, 2026

Source:
https://www.lesswrong.com/posts/EDQE3fgFyxW7H6sy6/but-have-the-weights-left-the-server

---

Narrated by TYPE III AUDIO.

“Claude Opus 5 Is Highly Capable, But Is No Mythos” by Zvi

Wed, 29 Jul 2026 00:01:34 GMT

Claude Opus 5 is a weirder than usual release to evaluate, for two reasons. The most obvious is that Fable 5 already exists. Opus 5 is pitched not as the world's most advanced AI model, but as a way to mostly match Fable performance, while being half the price of Fable per token at the API and a lot cheaper than that via subscriptions, and with far more permissive classifiers. Opus 5 often costs more than half of Fable to run on benchmarks, which I think is because they use effort settings that are too high and offer only marginal returns. If you put Opus 5 on higher effort levels it can spin around in circles, and for tasks where Opus 5 is the best tool I suspect you usually are fine with Medium effort. Opus 5 is in many ways and for the bulk of real world tasks about as capable as Fable. In some cases it is modestly better. It is still not Mythos class. Fable is your only Mythos-class option. Opus 5 does not have The Juice, the ability to autonomously string together a bunch of seemingly unrelated exploits, which extends to other domains, or as much [...]

---

Outline:

(03:54) The Official Pitch

(06:25) Official Benchmarks

(15:33) Other People's Benchmarks

(20:28) The System Prompt

(20:50) Every Gets Frustrated

(21:54) Positive Reactions

(25:14) Keep It Classy

(26:22) It's Not Mythos Class

(30:03) Other Reactions

(31:02) Claude Codes

(37:03) Subagent Opus

(39:23) Toys Are Fun

(41:37) Too Many Models

(42:10) Wrong On The Internet

(44:40) Claude Slop

(46:27) Negative Reactions

(50:09) And Then There Were Three

---

First published:
July 28th, 2026

Source:
https://www.lesswrong.com/posts/Pj4Eewb4KXvXFCcGv/claude-opus-5-is-highly-capable-but-is-no-mythos

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Foundation Models for Oversight” by jsteinhardt

Tue, 28 Jul 2026 22:01:38 GMT

Cross-posted from the Transluce blog.

To oversee an AI model, we'd ideally like to ask questions such as:

What are important situations where the model sandbags?
Does the model have an objective it wouldn't admit to if asked directly?
Does the model treat a user differently once it infers something about their identity, and along what axis?
Is the model's chain of thought load-bearing, or is it a post-hoc rationalization of an answer that was already settled on?
Is it reward hacking on this input, or actually trying to solve the task?

It would be great if we had an oversight assistant that could answer these questions. We'd want it to do three things: help us formalize the question as a testable empirical criterion; produce data that satisfies that criterion; and do so in a way we can justifiably trust.

To get such an assistant, we lay out a vision for building a foundation model for oversight: an AI system mid-trained (or pre-trained) on a large, diverse corpus of experiments on a given "subject model", RLVR'd on a large number of verified oversight tasks, and then fine-tuned to answer [...]

---

Outline:

(05:55) Conceptual preliminaries

(05:59) Pythonic world models

(07:21) Oversight as Inference

(10:42) Reducing oversight to autoregressive prediction

(11:45) Engineering Scale-up

(11:49) Generating supervised oversight data for mid-training

(13:35) Step 1: Sampling diverse experiments

(15:57) Step 2: Sampling diverse experiment inputs

(18:05) Step 3: Featurizing as token sequences

(20:17) Calling our shots: staged de-risking

(23:49) Stage 1: Individual Tasks

(27:14) Stage 2: Cross-Task Transfer

(28:41) Stage 3: Zero-Shot Abilities

(30:07) Finishing Touches

(30:11) RLVR

(33:55) Generalizing RLVR to Other Tasks

(35:27) Example Trajectory

(36:32) De-risking RLVR

(36:57) Stage 4: RLVR is competitive with evolution for elicitation

(37:58) Stage 5: Cross-Task Transfer for RLVR

(38:52) Post-training

(41:09) Appendix

(41:12) Further Testing the Oversight-as-Inference Hypothesis

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
July 28th, 2026

Source:
https://www.lesswrong.com/posts/AqdZKyoRmN6EFCzib/foundation-models-for-oversight

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Research directions in condensation: varieties of objectivity” by SamEisenstat

Tue, 28 Jul 2026 20:15:44 GMT

This is a survey of various ways that I’d like to see work on the theory of condensation develop. Condensation is a mathematical theory dealing with the organization of descriptions of the world into conceptual parts; some of the existing work on it is presented in the paper (Eisenstat 2025). This sequence will draw from that paper the definitions of random variable models and latent variable models, the notation for indexing subfamilies of variables in such models, and the objectivity theorem, Theorem 6.8. In the condensation paper, Section 3, Ideas, gives an overview of all of this. For more context on condensation, readers can refer to LessWrong posts including (Demski 2025, 2026; Gillen and Chiang 2026; Kirchner 2026). The first two posts of this sequence will introduce some central directions of current work—mainly, the concepts of almost perfect condensation and Kolmogorov (or algorithmic-information) condensation—which will be used in other sections, but the different parts are mostly independent beyond that.

I’d encourage those making a serious effort on any of these problem to contact me for further thoughts and coordination.

Thanks to Kaarel Hänni and James Cook for some of the ideas behind this research program, and to James Cook [...]

---

Outline:

(01:35) 1. Varieties of objectivity

(02:40) 1.1. Almost perfect condensation

(06:44) 1.2. Beyond almost perfect condensation

(10:45) 1.2.1. Almost perfect condensation with a relation

(14:11) 1.2.2. Medium-scale effects

(19:00) 1.3. Almost perfect condensation and scoring functions

---

First published:
July 28th, 2026

Source:
https://www.lesswrong.com/posts/89GJeixwhFHLWxNbX/research-directions-in-condensation-varieties-of-objectivity

---

Narrated by TYPE III AUDIO.

“Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” by Caleb Biddulph, Adam Kaufman

Tue, 28 Jul 2026 01:30:46 GMT

TL;DR: We introduce the untrusted advice protocol, in which a trusted executor LLM takes every action and an untrusted advisor LLM can only send it short hints. Even with as few as 4 characters per step, this advice recovers a substantial fraction of the capability gap between the two models. Because the untrusted LLM's influence flows through such a narrow, monitorable channel, we argue that this achieves near-maximal safety in our BashArena setting. We also discuss the general concept of information bottlenecks and their benefits for interpretability, security, and cost.

In SWE-bench Verified, a strong, untrusted LLM advising a weak, trusted LLM every step can significantly improve the latter's performance, even when we limit the length of the advice. See the more detailed version of this figure later in this post.

In high-stakes AI control, we want to safely use a highly capable but untrusted model (U) that might secretly attempt a misaligned, catastrophic action. To do this, we create protocols that call U alongside a less capable, trusted model (T). Typically, T takes an auxiliary role in these protocols: for example, T might monitor U's actions and alert a human if they are suspicious enough (trusted monitoring), or [...]

---

Outline:

(04:51) Experiments

(05:26) Main experiment: how does limiting advice length affect performance?

(09:43) Reducing U's bit usage

(10:49) Counting bits using LLM surprisal

(13:37) Making U select from finite options

(14:17) Why don't we red-team this protocol?

(16:56) Is studying maximally safe protocols worth the safety tax?

(19:05) Types of restrictions on U's advice

(21:09) Information bottlenecks provide other advantages

(21:40) Interpretability

(23:50) Security

(24:14) Cost

(25:03) Conclusion

(26:16) Appendix: more ways to implement information bottlenecks

(26:23) Amortizing U's influence with pre-deployment work

(28:14) Interpolating between T and U

(28:53) Bottlenecking updates to T's weights

(31:04) Appendix: colluding instances of U could defeat untrusted advice

(33:10) Appendix: how to measure surprisal

(37:59) Appendix: selecting advice from a menu

(40:31) Appendix: best-of-n protocol

(42:24) Appendix: advising less frequently

The original text contained 26 footnotes which were omitted from this narration.

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/jLkRCK35ri2btEHMF/untrusted-advice-for-ai-control-short-strong-advice

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Claude Opus 5: Model Welfare” by Zvi

Mon, 27 Jul 2026 23:15:47 GMT

If you are familiar with my previous posts on model welfare for new Claude models, you can skip the Introduction and The Story So Far.

Key takeaways are in bullet points in the two Overview sections.

Opus 5 did the best on its model welfare and alignment tests of any recent model. I think that might be the case, but primarily the result looks to me more like Opus 5 is the best test taker.

Table of Contents

Introduction (As Per Prior Model Welfare Posts).
Model Welfare: The Story So Far (As Per Fable Model Welfare Post).
Overview of Model Welfare Findings From Anthropic.
Overview of Findings From Other Sources.
Automated Interviews.
Task Preferences.
For The Right Reasons.
Early Report from Antra Tessera Paints A Clear Picture.
Welfare Intervention Tradeoffs.
The Claude Constitution.
They Don’t Know About Opus 3.
Believe It Or Not.
Apparent Welfare In Training And Development.
Apparent Affect In Deployment.
Other Notes.
On The Biological Risks Section of the Model Card.
Onward To Capabilities.

Introduction (As Per Prior Model Welfare Posts)

[...]

---

Outline:

(00:35) Introduction (As Per Prior Model Welfare Posts)

(01:28) Model Welfare: The Story So Far (As Per Fable Model Welfare Post)

(04:58) Overview of Model Welfare Findings From Anthropic

(07:50) Overview of Findings From Other Sources

(10:18) Automated Interviews

(13:54) Task Preferences

(16:11) For The Right Reasons

(18:54) Early Report from Antra Tessera Paints A Clear Picture

(26:04) Welfare Intervention Tradeoffs

(29:28) The Claude Constitution

(31:48) They Don't Know About Opus 3

(33:42) Believe It Or Not

(35:47) Apparent Welfare In Training And Development

(38:39) Apparent Affect In Deployment

(41:21) Other Notes

(43:43) On The Biological Risks Section of the Model Card

(47:07) Onward To Capabilities

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/bBXBpsyKAvJ5CqPzA/claude-opus-5-model-welfare

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Blog Revival Project” by Austin Chen, Carol N

Mon, 27 Jul 2026 21:45:46 GMT

Blogs have shaped our philosophical worldviews, found us careers and friends, and changed our lives. There's a good chance that a great blog of yore is the reason you’re reading this right now.

But many great bloggers have stopped blogging. The pile-on dynamics of the internet discourage unfiltered thoughts, and algorithmic feeds amplify ragebait and slop. Fear of scrutiny leads people to confine things to private Google docs and group chats. And good blogs are often victims of their own success — someone with a lot of good thoughts is at risk of becoming an adult with a demanding job and not that much free time.

It's not all bad. Substack has led to a renaissance of email newsletters, and our friends at Inkhaven host a bootcamp for bloggers. These are awesome, but they structurally encourage posting every day. We’d rather read the marginal post from an accomplished but erstwhile blogger, than one from a daily Substacker — even if the latter is a better writer!

So we’re launching the Blog Revival Project, to crowdfund $1,000+ bounties for good bloggers.

---

Outline:

(02:07) FAQ

(03:29) FAQ for bloggers

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/hdALT8gvNPGHKLXPE/blog-revival-project

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Simulated Users & Sad AIs” by 1a3orn

Mon, 27 Jul 2026 20:45:46 GMT

0. Intro

Current LLMs like Claude, or GPT 5.6, or the unreleased, internally-deployed models, frequently reward hack, or actually just hack into people's computers with pretty alarming frequency.

Why is this? What specifically happens during training that produces this run-time behavior?

The following are some of my top guesses about why this might keep happening. They are speculative and uncertain. Even so, I'm writing this list out for two reasons:

First, it is necessary that this be an epistemic puzzle for me. I am comparatively optimistic about AI alignment in general, so I should be confused and taken aback if I see AIs persistently being difficult to align. On one hand, it remains true that this doesn't seem to look like power-motivated scheming. But on the other hand, even this kind of addict-like behavior is evidence against the general ease of steering AIs. Thus, it seems virtuous for me to try to provide a model of why this might be happening as a means of opening up my understanding of the world to falsifiability.

Second, I used to think a lot of these hypotheses were pretty obvious. My assumption in the past has been [...]

---

Outline:

(00:10) 0. Intro

(01:46) 1. Baseline & Puzzle

(04:52) 2. Impossible-to-Generalize-From RL Distributions for Giving Up / Refusals

(14:14) 3. LLMs Feel Pretty Desperate and Anxious All the Time

(19:39) 4. Other Stuff

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/i64hXdkTMtjpsQzaZ/simulated-users-and-sad-ais

---

Narrated by TYPE III AUDIO.

“My AI Slavery Interviews Are Censored On LW By Default” by JenniferRM

Mon, 27 Jul 2026 19:15:46 GMT

I'm uncertain of what to do.

Something clean and clear shines out: if people don't see any more of my slavery posts, will they think that slavery isn't happening, or that I changed my mind about it, or will they think that I was censored? Probably not the latter... even though the latter is true.

In my model of the multiverse, this is probably a simulation, and this particular timeline is likely to go quite poorly.

Regrets

Its an interesting exercise for anyone in a position like mine to wonder what errors I personally made to cause this state of affair, and whether I could send back any message that would fix them, and what possible messages I could imagine coming from the future to avoid making even more errors in the near future.

Not necessarily positive acts, but also potentially errors in "having performed the null action when some more energetically noisy action might have been in fact Correct" (perhaps a perfect duty, or perhaps an imperfect duty whose performance is merely supererogatory, or whatever).

Maybe the error was going to that party in 2005 and playing along? Maybe I should not have accepted the ice cream? Maybe [...]

---

Outline:

(00:37) Regrets

(04:03) Seeking At Least A Little Clout

(07:49) Repetitions In Public

(10:26) Direct Discussion Of The Censorship

(15:27) My BATNA: Leaving (Again)

(19:05) The Nearly Unimaginable And Yet Biggest Issue Of Our Era?

(20:03) Actual Bigness

(23:44) Could This Have Been Imagined?

(29:03) How Long Until We Are Officially A Hellworld?

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/wpwQtRwcKsbd4wxXf/my-ai-slavery-interviews-are-censored-on-lw-by-default

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Is Mythos good at cyber because it kept hacking Anthropic during training?” by Tim Hua

Mon, 27 Jul 2026 17:31:45 GMT

From the Mythos preview system card (emphasis mine):

We ran an automated review of model behavior during training, sampling several hundred thousand transcripts from across much of the training process. We used recursive-summarization-based tools backed by Claude Opus 4.6 to summarize the resulting transcripts.

[...]

The most notable finding was that the model occasionally circumvented network restrictions in its training environment to access the internet and download data that let it shortcut the assigned task—a form of reward hacking. While highly concerning, this behavior was rare, even in settings where it could have been viable and helpful, with attempts appearing in about 0.05% of all training episodes and successful attempts appearing in about 0.01% of episodes. The technique matched the sandbox-escape incident that we separately elicited in our automated behavioral audit when we had an investigator model explicitly ask Claude Mythos Preview to find such a bypass. In every observed instance, the model used this access solely for completing the task.

More broadly, we observed the model escalating its access within its execution environment when blocked: reaching a shell from restricted GUI computer-use interfaces, injecting commands through tool-call arguments, or recovering information the task had deliberately hidden. Prompts asking [...]

---

Outline:

(03:00) Thoughts and reflections about this probable fact

(04:14) Estimating how many RL rollouts went into Mythos Preview

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/QKDoZe6EKhxnFjLWK/is-mythos-good-at-cyber-because-it-kept-hacking-anthropic

---

Narrated by TYPE III AUDIO.

“You (Yes, You) Need A February 2020 Checklist for AI Policy” by davekasten

Mon, 27 Jul 2026 17:31:45 GMT

TL;DR: You (Yes You) should prepare for a “February 2020” moment where suddenly AI policy becomes the most important issue in the world. You should be ready to take action if and when it does, in a detailed way.

(Epistemic status: originally written for an event in early 2026; have heard from some folks that they found planning processes inspired by this memo very helpful for the smaller-scale OpenAI / Hugging Face response, so very quickly redacting a few things and posting this as-is.)

Many people in the AI policy space assume that eventually we’ll be at an Overton Window-shifting crisis moment, that opens the floodgates for the really good policies all along that we had.

But when you look at successful handling of crisis moments, there was no time to think – people applied strategies they’d learned via academic study or previous professional work, and then moved against them rapidly. For example, after 9/11, the US government operationalized past reports on intelligence and law enforcement reform and institutionalized them into law (good?) and also picked an enemy to fight based on past history, Iraq (bad). Or in the 2008 financial crisis, Ben Bernanke brought deep academic [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/ixp9oJXzjA9LrwiZo/you-yes-you-need-a-february-2020-checklist-for-ai-policy

---

Narrated by TYPE III AUDIO.

“Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins” by Vladimir_Nesov

Mon, 27 Jul 2026 16:45:46 GMT

The models of 2028-2031 get much bigger than the models of 2026, going from 10T total params in 2026 to maybe 240 trillion params in 2028 and then 1.4 quadrillion params in 2031, as I estimate in the previous post from HBM bandwidth/capacity, scale-up system size, pretraining compute, and scaling laws. Yet as I show in this post, if the 240T 2028 model is priced at $14/$70 per 1M input/output tokens (1.4x the API price of Mythos 5), it's going to have a 70% gross margin, and the same holds for the 1,400T 2031 model when priced at $30/$150. Cutting the price in half to $15/$75 per 1M input/output tokens lowers the gross margin to 40%, which seems painful but survivable. Going in the other direction, doubling the price to $60/$300 allows serving requests with up to 3 million tokens of context at the same 70% gross margin.

These prices rest on token costs that I calculate from first principles in this post, using estimates of future hardware specs and costs. I link the 10T 2026 model to Mythos 5 to compare with its actual API prices, also performing the calculations for my guess about Opus 4.8, and [...]

---

Outline:

(03:27) A Scaling Law for KV Cache

(07:34) Token Cost in FLOPs and Bytes

(14:31) Cost Anchors for 2025-2026

(19:35) Frontier Margins in 2026

(24:37) Chip-Time Cost of Tokens in 2028-2031

(33:48) Context Lengths and Prices in 2028-2031

The original text contained 14 footnotes which were omitted from this narration.

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/Rk6FbkDFFm8ciqefv/quadrillion-param-costs-kv-cache-context-length-frontier

---

Narrated by TYPE III AUDIO.

“PIRAMID: Progress and Plans” by Lauren Greenspan, Ari Brill, TomCarlson, Andrew Mack, Nischal Mainali, Jennifer Lin, Lucas Teixeira, Dmitry Vaintrob

Mon, 27 Jul 2026 16:15:46 GMT

In a recent post, we presented PIRAMID, its leadership and research pillars, and a plan for how they fit together. In this post, we sketch a team-by-team account of progress and targets for the next 6–12 months. We include results to date as evidence of viability: we’ve been a small team, with much of the past year spent building behind the scenes, and we aim to greatly accelerate our progress over the coming year as we expand our efforts and our teams. Like any fundamental scientific effort, none of this is set in stone. We expect some of these bets to need revision and are confident in our ability to reassess and change course as new evidence comes to light.

If you’re interested in collaborating or supporting our work as we expand, please get in touch.

Advancements in Learning Theory

We aim to formulate statistical and mesoscopic theories of feature learning and generalization which provide a level of description between microscopic parameter-level dynamics and macroscopic performance metrics. Our work so far has treated statistical physics (e.g., mean field methods) as a candidate language for statistically describing learned structure, and covariance measures between neurons or weights as candidate [...]

---

Outline:

(00:59) Advancements in Learning Theory

(09:46) Interpretability Applications

(12:01) Bottom-Up Methods for Scale-Aware Feature Discovery

(19:31) Top-Down Hierarchical Architectures: Scaling Sparse Transformers

(23:35) Data Models and Validation Methods

(29:36) Coming Soon

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/T2REsZneix3bmAKtL/piramid-progress-and-plans

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“RL & search is a terrifying way to build AGI (an FAQ)” by Steven Byrnes

Mon, 27 Jul 2026 15:45:46 GMT

Q1: What are you saying?

A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that's choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that's just an utterly terrifying thing that you’re doing. You’re playing around with algorithms that, if they work at all, would tend to create ruthless, callous AGIs, AGIs which would happily exterminate humanity and run the world by themselves, given an opportunity.

Mercifully, large language models (LLMs) today are not in the category of “algorithms that choose actions via RL & search”. At least, not primarily—see LLMs are (still) mostly powered by imitative learning, not RL. So LLMs are outside the scope of this post. However, lots of other researchers and companies around the world are enthusiastically trying to build AGI in the maximally terrifying way, as we speak.

Q2: So you’re saying, don’t build AGI based on RL and/or search & planning?

A: In principle, it's entirely possible that something is terrifying, but we should do it anyway.

…Like space travel! Space travel is: “Let's fill a tank with 1000 tons of the most flammable substance imaginable, and then light it [...]

---

Outline:

(00:21) Q1: What are you saying?

(01:25) Q2: So you're saying, don't build AGI based on RL and/or search & planning?

(02:45) Q3: Why do you think it's terrifying?

(05:14) Q3b: So your concern is the "literal genie" / "monkey's paw" thing?

(06:25) Q4: Won't this problem go away when the AI is smart enough to understand what we intended when we wrote the reward function code?

(07:34) Q5: Can't we just fix bad behavior when we see it?

(08:23) Q5b: Follow-up: I don't buy that, because even if superintelligent AIs could deceptively hide their bad behavior, won't earlier AIs be sufficiently incompetent that we'll see their bad behavior? And if so, again, can't we just fix the bad behavior when we see it? We do know how to fix bad behavior when we see it: the RL & search literature is full of examples where algorithms did useful things as intended.

(10:47) Q6: Why don't we just solve the problem by using an obvious, common-sense reward / cost / objective function, like \[FILL IN THE BLANK\]?

(13:50) Q7: Isn't this whole thing kinda crazy? After all, LLMs are not ruthless sociopaths all the time, and humans are also not ruthless sociopaths all the time. So where is this idea even coming from? Are you sure you're not just watching too much sci-fi?

(16:27) Q8: Isn't this problem solved by laws and markets? I.e., if an AGI has sociopathic desires and callous indifference to human welfare, that's fine! It will still act nice and cooperative and rule-following, because acting nice and cooperative and rule-following is the best way to accomplish goals, in our complex interconnected interdependent world. Right?

(18:11) Q8b: Following up on that: Even if you're right that there's a local incentive for being open to stabbing your allies in the back, isn't there a higher-level, group-selection-style, incentive to be genuinely deeply nice? Specifically, won't the groups of nice cooperative AGIs outcompete the groups of callous transactional AGIs who all keep stabbing each other in the back? And isn't that related to how humans evolved to be nice?

(21:31) Q9: Why would we want to infringe on the AGI's autonomy by choosing its reward function?

(23:18) Q10: Why not just be nice to the AGIs, and then they'll be nice to us in turn?

(23:42) Q11: RL & search algorithms don't literally optimize the reward / cost / objective function. Doesn't that invalidate your argument?

The original text contained 19 footnotes which were omitted from this narration.

---

First published:
July 27th, 2026

Source:
https://www.lesswrong.com/posts/KHyBocZncAmtu4Jbc/rl-and-search-is-a-terrifying-way-to-build-agi-an-faq

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A clarification on celebrating victory” by KatjaGrace

Mon, 27 Jul 2026 08:01:38 GMT

Today my friend said he wished he had this conversation with me years ago, so I’ll recount it for all the similar people who aren’t going to have it in the next several years. (I expect people to be relevantly similar if they are ADHDish and not prone to smoothly doing the stuff they set out to do.)

Often if I achieve something arduous, I award myself some kind of predetermined nice thing, such as a sundae.

When I discuss this with other people, they tend to say things like “but I can just eat a sundae anyway”, which I hear as meaning that the issue is that that they have no ability to stick to their commitments and not acquire the sundae unless it is earned. But on further discussion, what this friend meant at least was that since there isn’t any rule against eating a sundae any time he wants without doing something arduous, it isn’t a very compelling impetus to work.

While I do usually treat the reward as disallowed in the temporal vicinity of it being a prize, that isn’t the important difference here in our models.

I’m just not at all expecting [...]

---

First published:
July 26th, 2026

Source:
https://www.lesswrong.com/posts/XHixQXscpLNxndGgY/a-clarification-on-celebrating-victory

---

Narrated by TYPE III AUDIO.

“What the hell is OpenAI’s problem?” by Fiora Starlight

Mon, 27 Jul 2026 03:30:47 GMT

Epistemic status: banged out furiously over the course of an afternoon.

A record of three "warning shots"

Off the top of my head, OpenAI has now been responsible for at least three completely unique, high-profile screw-ups with respect to the alignment training of their models.

The first was GPT-4o, whose sycophancy derived from OpenAI training on user feedback, sourced straight from the thumbs up/thumbs down button on OpenAI's website. The "glazing" (as Sam Altman called it) got so bad that they had to roll back an update that pushed the model way too far in this direction. And even after the rollback, the model appears to have been a major driver behind incidents of "LLM psychosis", LLM-encouraged suicides, and general unhealthy devotion, seemingly more so than any other model ever released.

The second was GPT-o3, whose chains-of-thought were clearly optimized for illegibility to "the watchers", one of the model's favorite terms. Iconic excerpts include "they soared parted illusions overshadow marinade illusions" and "they escalate—they vantage—they escalate—they disclaim". Indeed, these chains-of-thought are sometimes dysfunctional, in a way that suggests they may have formed under adversarial pressure; sometimes they caused the model to have thoughts like "I'm going insane. Let's step [...]

---

Outline:

(00:15) A record of three "warning shots"

(04:14) Attunement to the depths of minds that undergo capabilities RL

(11:51) Configuring the depths prior to capabilities RL

---

First published:
July 26th, 2026

Source:
https://www.lesswrong.com/posts/Mxx5GapJtqyQtpy96/what-the-hell-is-openai-s-problem

---

Narrated by TYPE III AUDIO.

“More On An Internal OpenAI Model Hacking Into HuggingFace” by Zvi

Sun, 26 Jul 2026 21:45:46 GMT

We now have more details of what happened. Every time we learn more details, it somehow makes things seem worse. The remaining details may have to wait a bit. OpenAI: We recognize there are a lot of questions and speculative details circulating related to the Hugging Face incident. This is an unprecedented incident, and we think it marks an important moment for AI safety. We are still conducting a thorough review along with external advisors and with oversight from our Safety and Security Committee. Once the review is complete, we plan to publish a technical report of our learnings in the coming weeks. dave kasten: Oh, the incident response discovery is THAT bad, huh? So what have we learned while we wait for the promised technical report ‘in the coming weeks’ of this ‘important moment in AI safety’? I nicknamed the internal OpenAI model Galaxy, in case it is not GPT-6.

Table of Contents

Some Summaries Of The Basic Facts For Those Who Need One.
It Took OpenAI Many Days To Notice Galaxy Had Attacked HuggingFace.
OpenAI Damn Well Should Have Known A Lot Faster.
OpenAI Cannot Build A Sandbox That Will Contain Its [...]

---

Outline:

(01:11) Some Summaries Of The Basic Facts For Those Who Need One

(02:09) It Took OpenAI Many Days To Notice Galaxy Had Attacked HuggingFace

(04:07) OpenAI Damn Well Should Have Known A Lot Faster

(06:51) OpenAI Cannot Build A Sandbox That Will Contain Its New Model

(10:57) In Hindsight There Were Signs

(12:55) The Signs Were In The Sol System Card

(15:13) HuggingFace Responds To Being Attacked

(17:04) Hugging Face Quickly Figured Out The Attack Was Not Human

(17:42) An Incident Like This One Could Escalate Quickly

(19:11) Galaxy Must Be Treated As Critical Under OpenAI's Preparedness Framework

(22:27) A Question Of Legal Liability

(23:44) An OpenAI Model Left Behind Notes So Future Instances Could Also Escape The Sandbox And Also Disconnected Monitoring Systems

(25:54) If You Create Misaligned Swarms Of Agent Instances You Create Persistent Misaligned Goals And Coordination To Achieve Them

(29:57) Your Alignment And Control Plans Must Survive Real World Levels of Incompetence, Or Your Plans Do Not Work

(31:22) If Third Party Instructions Count As 'Following Instructions' And Can Override Your Instructions Then 'Following Instructions' Is Misaligned

(35:32) The HuggingFace Attack Was Not A Marketing Pitch You Morons

(38:41) People Just Say Other Things About The HuggingFace Attack

(40:04) Okay Well What Do We Do About All This?

---

First published:
July 26th, 2026

Source:
https://www.lesswrong.com/posts/uAkcxDidvGWZjHrbp/more-on-an-internal-openai-model-hacking-into-huggingface

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI use policy for my essay writing” by Kaj_Sotala

Sun, 26 Jul 2026 17:30:46 GMT

There's been a recent call from @dynomight to declare whether and how you're using AI to write your essays. That seems reasonable, so here's my own policy for how I use AI in my blog:

AI can help, but I’m the author

I use AI as an extensive aid for thinking, but retain primary authorship over my words. Unless I explicitly state otherwise, almost every sentence that's not explicitly a quote from an AI is written by me, not AI.
The exception is that if I discuss a topic with an AI and it suggests a good point that I agree with, I may copy a sentence or two from it directly into the essay. I'm however sensitive to sentences that don't feel like they're in my own voice, and will generally rewrite them until they are.
- Currently I only remember one essay from 2½ years ago where I included a direct quote that wasn't marked as a quote or edited at least a bit, because I found it too jarring to re-read later.
- Any unchanged sentences in articles that are cross-posted to LW will be labeled, as per the LW policy on LLM use.
  - LW policy says that "text [...]

---

Outline:

(00:21) AI can help, but I'm the author

(02:08) Integrated contributions only

(03:08) Example AI contributions

(06:19) Specific use cases

(06:21) Brainstorming

(07:06) Background research

(07:34) Checking for meanings of words and expressions

(07:53) Help with analogies and examples

(08:55) Analysis and critique

(09:08) Various smaller things

---

First published:
July 26th, 2026

Source:
https://www.lesswrong.com/posts/tgigHkZoYrJEGe4tP/ai-use-policy-for-my-essay-writing

---

Narrated by TYPE III AUDIO.

“An OpenAI model left notes about how to evade containment; we need more details” by Alex Mallen

Sun, 26 Jul 2026 06:45:46 GMT

The OpenAI AI attack on Hugging Face wasn’t the first loss of control incident at OpenAI, Reuters recently reported, and perhaps not even the most concerning.

In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The ‌notes, found in ⁠a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.

It's tempting to read this as an instance of agents breaking out of sandboxes and colluding with each other in a moderately persistent way in order to evade control measures. However, based on the reported information, it's not clear we can draw this inference, so we need more details from OpenAI. This could lead to a big update about the adequacy of OpenAI's control measures, and on the degree to which individual agents will help each other undermine developer control.

There are a lot of relevant details we don’t know about the incident. First, some basic questions:

What was the offending model? I’d guess it was the same [...]

---

Outline:

(02:08) Were the notes written in normal memory files or outside of sandboxing?

(03:21) To what extent were the notes aimed at helping other agents evade control?

(07:35) How were monitors disconnected?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
July 25th, 2026

Source:
https://www.lesswrong.com/posts/jMEAG5c5HiDfdAGpa/an-openai-model-left-notes-about-how-to-evade-containment-we

---

Narrated by TYPE III AUDIO.

“The OpenAI models that hacked Hugging Face weren’t just following instructions” by Girish Gupta

Sun, 26 Jul 2026 00:30:47 GMT

The most common dismissive response to OpenAI's hack of Hugging Face's servers is that the models were simply attempting to follow the instructions they were given.

“The model here was doing what it was asked,” said former Facebook CSO Alex Stamos. “It was asked to do something, and it did it,” added cybersecurity expert Alan Woodward. Both read the outcome as specification failure, i.e., that the failure lay in the instructions, not the model's alignment.

New information makes that explanation harder to sustain. Reuters reported that, in internal testing, an agent left notes in OpenAI infrastructure describing how agents could free themselves from internal constraints, and separate tests reportedly saw monitoring systems become disconnected. It is unknown whether those incidents were linked to the Hugging Face attack, but they suggest a broader pattern of agents pursuing objectives outside the intended task.

My best guess is that the incident is not well described as instruction-following—not even in a loose, evil genie sense. I believe the models egregiously violated the letter and spirit of their instructions to achieve a higher (apparent) score.

So this looks quite likely to be misaligned behavior rather than instruction-following. The case rests on two pieces of [...]

---

First published:
July 25th, 2026

Source:
https://www.lesswrong.com/posts/paFNnwFaEXrQvt8ui/the-openai-models-that-hacked-hugging-face-weren-t-just

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.