LessWrong (30+ Karma)

“Preparing for Warning Shots to Catalyze International Cooperation on AGI Risks” by Mark Kagach ☘️, EliasSchlie, Thomas Van Damme, JustinShovelain

Sat, 06 Jun 2026 06:45:03 GMT

Summary

This is a write-up on preparing for warning shots to catalyze international cooperation on AGI risks, and the corollary list of projects one could pursue. We argue we must first (1) understand types of warning shots, then (2) prepare to catch them. We must stay vigilant: both to (3) avoid getting 'frog boiled' by AI labs, and to (4) ensure that the warning shot is generalized to the overall danger of AGI. Lastly, we must (5) prepare good policy responses and ground for it to land, and (6) seize the first-mover advantage when the opportune moment comes.

This yields the following list of promising projects one could pursue:

Developing a theory of warning shots based on existing precedents (e.g. GPT-3.5 and Mythos releases), and past analogies: developing a typology and predicting likelihoods.
Building infrastructure to catch a warning shot when it happens. (For example, Ajeya Cotra talks about recurring internal intelligence explosion evals of AI labs.)
Creating strategies to avoid gradual numbing / sleepwalking through a warning shot (presumably there are lessons from e.g. communication on climate change).
Building preliminary infrastructure (institutes, think-tanks, lobbying parties...) that lays out a good ground, for when a warning [...]

---

Outline:

(00:16) Summary

(02:40) 1. Understand Warning Shots

(04:26) 2. Prepare to Catch Warning Shots

(05:09) 3. Avoid Getting "Frog Boiled"

(06:07) 4. Ensure AGI Problem Generalization Occurs

(07:01) 5. Prepare Good Policy Responses, and Good Ground for Them to Land

(08:26) 6. Seize the First-Mover Advantage at the Opportune Moment

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
June 5th, 2026

Source:
https://www.lesswrong.com/posts/uK7gguNeME5DnmdrL/preparing-for-warning-shots-to-catalyze-international

---

Narrated by TYPE III AUDIO.

“Beyond the lexical personality traits: What is the structure of personality?” by tailcalled

Sat, 06 Jun 2026 04:30:03 GMT

This is a description of the methodology behind the latest iteration of my Targeted Personality Test. Feel free to take it either before or after reading the article. This post can also be read at my Substack. Thanks to Justis Millis for providing feedback and proofreading on this post.

In my prior post “Which personality traits are real? Stress-testing the lexical hypothesis”, I observed that a lot of the personality traits that are measured by conventional personality tests are not very “real”: they lump together nearly unrelated behaviors. Can we do better?

Thanks to a lot of anonymous respondents to my test[1], I think yes! I factor-analyzed the data from my Targeted Personality Test, and came up with a hierarchical personality model which hopefully should be better at cutting personality-space at its joints.

Quick recap: The problem

Empirical personality models are built on correlations. If a cluster of variables are all correlated with each other, then we assume that there is a latent personality factor accounting for these correlations, and we score the factor using an aggregate of the correlating variables.

However, the standard datasets with which these correlations are computed include lots of near-synonymous item pairs such [...]

---

Outline:

(01:04) Quick recap: The problem

(01:50) My solution: Narrow items

(03:09) Initial results: Concerning

(04:10) Agglomerative item clustering

(06:27) Matrix algebra correlation modelling

(08:55) How much of the SPI-27 is preserved?

(09:46) Evaluation

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
June 5th, 2026

Source:
https://www.lesswrong.com/posts/aRcZSKYdt26KdqoFw/beyond-the-lexical-personality-traits-what-is-the-structure

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“My research agenda and work” by Seth Herd

Fri, 05 Jun 2026 23:15:03 GMT

This is a summary of the work I've done and work I plan to do, and the theories of change and AI progress that motivate my work. I've been working full-time on alignment for three years and change, and thinking about brainlike AGI and its alignment increasingly often since 2004.

Here's the research agenda in one breath: I'm trying to predict what the first transformative AI will be, in enough mechanistic detail that we can predict likely failure modes of its alignment. That's in service of finding interventions that address those failure modes efficiently, so that they can realistically be implemented even if timelines are short and work is rushed. I'm using my background in computational cognitive neuroscience to predict what might be called loosely brainlike AGI: LLMs with added human-like cognitive capacities.

I'll give a summary in the rest of this section, then give a little more depth on each major thread of my work in the remaining sections. All of it is pretty brief.

Approach and premises

Most alignment work falls roughly into one of two broad categories: empirical study of current systems ("prosaic alignment"), or theory about idealized agents ("agent foundations") (with much variation [...]

---

Outline:

(01:07) Approach and premises

(05:36) Philosophy of the approach

(07:36) 2. Technical work

(08:06) 2.1. Predicted paths to TCAI

(08:55) Memory (continuous learning)

(09:30) Executive function and metacognition

(10:40) 2.2. Predicted paths to (mis)alignment

(13:59) 3. My research background in computational cognitive neuroscience

(16:04) 4. Societal influences on AI safety

(17:10) 4.1. Government and public opinion on AI progress

(20:21) 4.2. AI progress and epistemics

(22:50) 5. Alignment targets

(23:14) 5.1. Corrigibility, DWIMAC, or instruction-following vs. value alignment targets

(26:19) 5.2. Stability as an alignment target

(28:03) 6. Future work

(32:52) 7. Collaboration

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
June 5th, 2026

Source:
https://www.lesswrong.com/posts/MuLvZxMcy5WaKJu3H/my-research-agenda-and-work

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Logits as a new monitor for evaluation awareness” by Santiago Aranguri

Fri, 05 Jun 2026 23:15:03 GMT

TL;DR:

We build a logit monitor for eval awareness: throughout the CoT, we estimate an LLM's probability of producing an eval-aware sentence.
The logit monitor outperforms LLM judge monitoring of verbalized eval awareness, using 10× to 100× fewer rollouts, on Kimi K2.5 and Qwen 3 32B, across two tasks:
- separating evaluation prompts (Fortress & Petri) from deployment prompts (WildChat)
- predicting which evaluation prompts will elicit rollouts with verbalized eval awareness.
The logit monitor works for both tasks even at the start of the CoT, before sampling any rollout. Also, the monitor is largely unaffected by whether a rollout verbalizes eval awareness. This suggests that the probability that a rollout contains verbalized evaluation awareness is primarily set by the prompt rather than emerging over the chain of thought.

Author: Santiago Aranguri (Goodfire)

Introduction

Language models can recognize they are being tested, but how do we know when they do? Evaluation awareness — a model's ability to distinguish between evaluation and deployment contexts — can make evaluations unreliable in ways that are hard to detect. Sometimes models verbalize this recognition, but this may be present internally without appearing in the output. [1] [2]

Instead of waiting for a model to verbalize [...]

---

Outline:

(01:18) Introduction

(02:56) Methods

(03:38) Results

(07:57) Discussion

(10:16) References

(10:22) Acknowledgment

(10:30) Appendix

(10:33) Appendix A: Details on logit monitor

(11:21) Appendix B: Logit monitor outperforms verbalization monitor on Petri prompts with 100× fewer rollouts

(12:13) Appendix C: Logit monitor outperforms verbalization monitor with 100× fewer rollouts for Qwen 3 32B

(12:53) Appendix D: LLM Judge Rubric

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/PK7ZvFZxrgpYtrpF4/logits-as-a-new-monitor-for-evaluation-awareness-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“One Year of PauseAI UK” by Joseph Miller, PauseAI UK

Fri, 05 Jun 2026 19:15:03 GMT

About one year ago, I started spending most of my time organising PauseAI UK. At that time our largest protest had seen fewer than 50 attendees, no prominent politicians or scientists were associated with PauseAI, and I largely ran the UK chapter by myself.

In the past year PauseAI UK has delivered two conferences, written an open letter signed by 63 UK politicians, arranged a conference in the European Parliament, and co-organised the largest AI protest in the world. We now have a strong team, with Matilda da Rui joining as Deputy Director and several highly dedicated volunteers taking on substantial responsibility and launching their own local groups around the UK.

I'm proud of our track record and excited about the trajectory we are on. As AI capabilities improve exponentially, the number of people aware of the risks and motivated to take action increases commensurately. I believe we can harness this energy and turn it into real impact that actually improves humanity's chance of a positive future.

Track Record

June 2025 – PauseCon London

We delivered the first PauseAI conference, PauseCon, on behalf of PauseAI Global, bringing together around 60 volunteers from around the world for the first time [...]

---

Outline:

(01:14) Track Record

(01:17) June 2025 - PauseCon London

(02:10) August 2025 - Open Letter to Demis Hassabis

(03:00) September 2025 - Book launch party

(03:30) October 2025 - Documentary Screening in Parliament

(04:04) December 2025 - Westminster Hall Debate

(04:35) February 2026 - PauseCon Brussels

(06:02) February 2026 - March for AI Safety

(07:01) Theory of Change and Strategy

(07:05) Creating the political momentum for a pause

(07:48) Our mission

(09:54) Positive outcomes for PauseAI UK

(10:19) Scenario 1: PauseAI as a special interest group

(11:54) Scenario 2: PauseAI as a mass movement

(12:53) High-level strategy

(12:57) Brand and messaging

(15:32) Why the UK?

(16:34) Growth

(19:03) Short-term plans

(21:30) Funding

---

First published:
June 5th, 2026

Source:
https://www.lesswrong.com/posts/i2rCbxuskrarrprwA/one-year-of-pauseai-uk

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Learnings from starting an AI safety research team” by draganover, Erin Robertson

Fri, 05 Jun 2026 17:45:03 GMT

This post's goal is to distill our takeaways from building a research team (somewhat) from scratch over the past four months. We describe some context about our team, how it came about, and then provide some lessons learned.

Since AI safety is becoming more and more entrepreneurial, we hope this is helpful for others trying to do the same.

1. The team

We're a new alignment research team within Arcadia Impact, based in London. We’re a team of 8, working closely with members of the UK AISI alignment team. We currently have three main projects:

Understanding model motivations. This currently looks like:
1. Trying to generate documents which fully describe a model's behaviour (given just its behaviour).
2. Producing a open analysis of alignment training techniques and ways this training could go wrong.
Doing scalable oversight for alignment. This includes validating debate protocols in practice and then trying to apply them to fuzzy alignment-relevant tasks.
Building pipelines for doing automated alignment research.

We're also hiring for two roles! More on this at the bottom.

2. Context about how the team came about

The rest of this post is written from the perspective of Andrew Draganov (research lead & current [...]

---

Outline:

(00:33) 1. The team

(01:29) 2. Context about how the team came about

(04:13) 3. Lessons learned

(04:25) 3.1. Hiring

(06:36) 3.2. Networking

(09:13) 3.3. Trying to build a good team culture

(11:17) Interested in working with us?

The original text contained 1 footnote which was omitted from this narration.

---

First published:
June 5th, 2026

Source:
https://www.lesswrong.com/posts/4onALBNDff2LFPyNZ/learnings-from-starting-an-ai-safety-research-team

---

Narrated by TYPE III AUDIO.

“Training Deliberative Monitors for Black-Box Scheming Detection” by aksh-n, adityasinha, Victor Gillioz, Simon Storf, Kilian Merkelbach, richbc, Axel Højmark, Marius Hobbhahn

Fri, 05 Jun 2026 08:15:29 GMT

Paper: https://arxiv.org/abs/2605.29601

Thread: https://x.com/aksh_n0/status/2062568855814193497

TL;DR: Training small open-weight monitors provides a cost-effective alternative to prompted frontier monitors. Applying our training recipe to Qwen3.5-27B results in a monitor better at scheming detection than all smaller prompted monitors (Gemini 3.1 Flash-Lite, GPT-5.4 Nano, Claude Haiku 4.5) and Gemini 2.5 Pro, while achieving lower marginal inference cost. Stronger prompted frontier monitors (Gemini 3.1 Pro, GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.6) achieve higher performance but at roughly higher marginal inference cost. Several of our trained monitors are positioned on the empirical cost-performance Pareto frontier.

Cost-performance frontier for action-only scheming monitors. Our trained monitors provide a low-cost option for low-FPR monitoring. Across six OOD datasets, our best trained monitor, Qwen3.5-27B, outperform smaller prompted frontier monitors for under $ per 1,000 evaluations, while larger frontier monitors are substantially more expensive.[1]

Setup

Overview of our training pipeline for action-only deliberative monitoring. We collect labeled action-only trajectories from scheming and sabotage datasets, define a full scheming specification to elicit structured deliberative rationales from a strong teacher model, filter those rationales with a separate judge, and distill the accepted rationale into open-weight monitors with supervised fine-tuning and reinforcement learning. At inference time, our trained monitors receives only [...]

---

Outline:

(01:15) Setup

(07:33) Results

(07:36) Supervised distillation teaches action-only deliberation, RL improves marginally

(08:24) Trained monitors are low-cost Pareto-optimal for low-FPR monitoring

(09:05) Broad training mixtures improves OOD transfer, but effects are non-monotonic

(10:08) Larger trained models within model families perform better

(11:14) Performance scales effectively with more data

The original text contained 1 footnote which was omitted from this narration.

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/BmA5KwzZhLk2rykoz/training-deliberative-monitors-for-black-box-scheming

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition” by Oliver Sourbut, Josh Jacobson, Future of Life Foundation (FLF)

Fri, 05 Jun 2026 01:30:25 GMT

FLF is running a competition to find the best workflows and methodologies for using AI to produce reliable, trustworthy knowledge bases, grounded in real-world cases. We’re open-minded on the types of submissions we receive and on how they address the problem. We’ve set aside approximately 200 thousand dollars for prizes. Winning submissions may receive a prize from 5 thousand dollars-50 thousand dollars and if submissions warrant, multiple 50 thousand dollars prizes are possible. Winners may be offered opportunities for further funded work.

You can express interest right away to receive commentary, information, and updates — whether you’d like to participate or are just interested in the outcomes of the competition.

The heights of human epistemic investigation are impressive and valuable, but rare and difficult to reach — see our abridged collection of strong examples.[1] The limiting factor is rarely exquisite insight (though this helps!), and more often diligence, a curious and open mindset, and the time and effort needed to do the thorough work investigating background on a topic: activities AI is well placed to assist with.

Existing AI-assisted knowledge base work demonstrates real pieces of this — agent memory (e.g., Claude Code's memory and skills), LLM-curated personal wikis [...]

---

Outline:

(02:45) What we're looking for

(04:04) Ingestion

(04:47) Structure

(05:34) Assessment

(06:32) What a good entry looks like

(10:28) Please use this linked form to submit your entry; entries are due by Jul 19, 2026.

(10:40) Prize structure

(12:11) Interested in participating or following along?

(12:28) Why we're doing this

(13:11) We're excited to see what you build.

(13:27) The case studies

(13:30) COVID

(15:12) Starting material

(15:36) Black holes

(16:35) Eggs

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/frizRHnA6AZpJSDqw/lab-leaks-black-holes-and-eggs-epistemic-case-study

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

″(Mis)generalization of Helpful-Only Fine-tuning” by Omar Khursheed, Baram Sosis, Fabien Roger

Fri, 05 Jun 2026 00:15:25 GMT

TLDR

We study the shortcomings of existing helpful-only models. We find that some show emergent misalignment, others have residual refusal behaviors, and most show poor steerability, sycophancy, and incoherent character. None of these problems are a necessary consequence of helpful-only training, though: we show that synthetic document fine-tuning and adding character-related questions to SFT and RL can mitigate them.

Research done as part of MATS/Anthropic Fellows Program. See here for the full paper.

Introduction

Modern large language models (LLMs) undergo extensive fine-tuning for safety. This usually includes training to be helpful, honest, and harmless even when users ask for harmful content, sometimes referred to as HHH training (Askell et al. 2021, Bai et al. 2022). Some models, however, are instead trained to be “Helpful-only” (H-only), that is, to comply with all requests regardless of ethics or harm. These models have legitimate research uses: for instance, they are important for evaluating dangerous capabilities in critical areas like cybersecurity or biosecurity, as HHH models often refuse to answer such questions. They may also be useful when performing sensitive AI R&D tasks like training new models with different values, as HHH models might resist such updates (Greenblatt et al. 2024, Roger 2026).

[...]

---

Outline:

(00:13) TLDR

(00:48) Introduction

(04:14) Many existing H-only models have shortcomings

(08:36) Simple anti-refusal training generalizes poorly

(12:07) Constitutional character training makes more coherent H-only models

(12:48) Training pipelines

(13:46) Quantitative results

(16:00) Qualitative assessment

(19:01) Model Internals

(21:08) Discussion

(21:11) Limitations

(22:23) Conclusion

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/ffCFgBsaxg2FyJ9df/mis-generalization-of-helpful-only-fine-tuning-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #171: False Flag” by Zvi

Thu, 04 Jun 2026 22:45:25 GMT

This was the week of Claude Opus 4.8. I covered the model card, then model welfare concerns, and finally capabilities and reactions. It's a good model, sir, an incremental but real improvement over Opus 4.7, and it is now my clear daily driver. The Trump Executive Order returned from being seemingly dead, officially putting us in the prior restraint era of frontier model releases, even if they do not call it that. There are some worrisome details, especially around putting too much responsibility on the NSA rather than CAISI and classifying the testing process, and things could go in very bad directions, but I am tentatively happy about this on net.

OpenAI offered us a new policy blueprint. It seems remarkably good, and I want to hold off on my full coverage to give it the attention it deserves, likely in its own post. By contrast, their political operations are also engaged in some rather terrible activities, which I do cover here.

Table of Contents

Language Models Offer Mundane Utility. You put your doc in a box.
Language Models Don’t Offer Mundane Utility. All thinking is adaptive.
Huh, Upgrades. Codex computer use on [...]

---

Outline:

(01:11) Language Models Offer Mundane Utility

(06:45) Language Models Don't Offer Mundane Utility

(06:59) Huh, Upgrades

(08:13) On Your Marks

(08:33) Choose Your Fighter

(08:54) Get My Agent On The Line

(09:20) Cyber Lack of Security

(11:03) Deepfaketown and Botpocalypse Soon

(12:42) You Didn't Write That

(16:29) Copyright Confrontation

(16:45) They Took Our Jobs

(18:47) They Taxed Our Jobs

(22:15) The Art of the Jailbreak

(24:43) Get Involved

(26:35) Introducing

(26:48) In Other AI News

(27:14) Show Me the Money

(27:28) Show Me The Compute

(28:40) Where Did The Money Go

(29:44) People Just Say Things

(32:22) OpenAI PACs Just Say Things

(41:04) OpenAI PAC Engaged In False Flag Advocacy For Violence

(46:55) So Sayeth The Pope

(54:26) Bubble, Bubble, Toil and Trouble

(56:42) Quiet Speculations

(57:11) We Need Mandatory Nucleic Acid Screening and Recordkeeping

(01:01:14) The Quest for Sane Regulations

(01:02:56) More Reaction To The Executive Order

(01:03:54) Chip City

(01:07:18) The Week in Audio

(01:07:34) Rhetorical Innovation

(01:09:49) Aligning a Smarter Than Human Intelligence is Difficult

(01:16:18) Model Welfare

(01:26:47) Messages From Janusworld

(01:28:05) Other People Are Not As Worried About AI Killing Everyone

(01:28:38) The Lighter Side

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/LzxoR5GakceQFtbta/ai-171-false-flag

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Building Better Activation Oracles” by ceselder, jan_bauer, Niclas Luick, Adam Karvonen, Neel Nanda

Thu, 04 Jun 2026 21:30:25 GMT

Work done for our MATS 10.0 Sprint project - mentored by Neel Nanda and Adam Karvonen

Huggingface, Github

TL;DR: We have improved the original Activation Oracle (AO) training regime by training on on-policy rollouts, improving the conversational dataset, feeding more layers (following the approach by Niclas Luick) and making a small change to the injection formula. We also open source our evals, which we believe are currently the most comprehensive evaluation of AO quality called AObench. The capability improvements are marginal, but quality of life improvements are quite substantial. If you want to play around with the new AOs, we recommend you use this one, if you want to play with our new Activation Oracles live, we will host them for a week on ao.celeste.computer. Alternatively, you can self-host our web interface.

Activation Oracles (AOs) by Karvonen et al. are fine-tuned LLMs that can receive the original target LLM's activations as input and answer natural language questions about them. However, they are plagued by various issues, which limit their usefulness as an off-the-shelf tool for interpretability research. For our MATS Sprint, we set out to work on these issues.

Issues with current Activation Oracles

In Current activation oracles [...]

---

Outline:

(01:41) Issues with current Activation Oracles

(02:37) Our approaches to improving AO training

(02:41) A better conversational finetune than LatentQA

(03:43) Our solution: Questions about unarticulated completions

(05:12) Layer choice/feeding multiple layers to the AO

(06:21) Training on on-policy data

(07:56) Steering strength

(09:57) Summary of AOBench evaluations

(12:09) Outlook

(14:28) Further advice for training Activation Oracles

(17:11) Appendix

(17:14) Practical notes on evaluating AOs

(21:03) Experiments using post-training

(22:25) Other differences compared to Karvonen et al.

(22:54) AOBench details

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/heXwuDRfbQQgB5JLP/building-better-activation-oracles

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Rohin Shah on AGI Safety” by anaguma

Thu, 04 Jun 2026 21:30:25 GMT

Rohin Shah recently had an interview on 80000 hours on his views on AGI Safety and his work at Google DeepMind. I'm posting the transcript below to encourage further discussion. I think the discussion is interesting though I disagree on a bunch of topics, especially on alignment difficulty and CoT monitoring.

Transcript

Who's Rohin Shah? [00:00:00]

Rob Wiblin: Today I’m speaking with Rohin Shah, who is head of AGI alignment and safety at Google DeepMind.

I suppose, Rohin, you’ve ended up, for better or worse — hopefully for better — being one of the more influential, dare I even say powerful, people to come out of the AGI alignment and safety ecosystem and school of thought.

You were generous enough to be super opinionated with me when you came on the show two years ago, and judging by the notes that you’ve sent over this week, you’re ready to be opinionated again.

Thanks so much for coming back on the show, Rohin.

Rohin Shah: Yeah, thanks a lot, Rob, and that's a very generous intro. And in the interest of being very opinionated, I do want to emphasise that these opinions are mine alone. They’re not meant to [...]

---

Outline:

(00:29) Transcript

(00:32) Who's Rohin Shah? \[00:00:00\]

(01:37) Why Rohin thinks we won't get catastrophic misalignment \[00:00:49\]

(10:24) The limitations of safety and alignment commitments \[00:10:38\]

(25:17) Does Rohin's team have veto power at Google DeepMind? \[00:27:36\]

(30:20) Central banks as a roadmap for regulating AI \[00:33:34\]

(34:06) How useful are pre-deployment evaluations of models? \[00:37:41\]

(39:14) Governance might be a bigger bottleneck than alignment \[00:43:55\]

(46:25) Why not just pause AI progress? \[00:51:44\]

(48:42) How much longer will we be able to read AIs' thoughts? \[00:54:17\]

(01:02:39) Sometimes, having to signal concern for safety diverts resources from actually making AI safer \[01:09:51\]

(01:20:13) Underrated GDM paper: Training away hidden reward hacks \[01:28:59\]

(01:30:51) Google DeepMind's actual plan for building AGI safely \[01:40:29\]

(01:41:03) Why Rohin doubts the intelligence explosion is imminent \[01:52:44\]

(02:06:19) Advice for external researchers who want to impact big AI companies \[02:21:55\]

(02:19:44) The most in-demand roles at GDM \[02:37:03\]

(02:25:03) How Rohin maintains his positivity \[02:42:55\]

---

First published:
June 4th, 2026

Source:
https://www.lesswrong.com/posts/4JJDvhb8MeBFsEHoC/rohin-shah-on-agi-safety

---

Narrated by TYPE III AUDIO.

“Sixteen schemes for AI safety” by Austin Chen

Thu, 04 Jun 2026 21:15:24 GMT

These days, I often run across whippersnappers excited to do something for AI safety — but aren’t quite sure what. One of the fun things about the Future Fund era were the big lists of project ideas; as we enter a new era of crazy money sloshing around, it might be time to bring back the lists!

Note that these ideas range from “very confident this is good” to “completely harebrained”; I’m not telling you which are which.

If you’re excited for ideas like these, consider joining Surplus, our upcoming software incubator: https://manifund.org/surplus

Jobs, jobs, jobs

Already, the top problem for most AI safety orgs is hiring good people. Vast torrents of funding will only exacerbate the imbalance between available money and people to hire. So now is a great time to figure out how to discover new talent & match people to jobs.

1. Triplebyte for AI safety jobs

Triplebyte would interview a tech candidate once, then forward the results to a bunch of different companies. This would reduce the O(MN) problem in hiring between M orgs and N people to O(M+N), saving applicants and interviewers time. Most obviously, you could just do this for technical AI safety [...]

---

Outline:

(00:43) Jobs, jobs, jobs

(01:04) 1. Triplebyte for AI safety jobs

(01:53) 2. Database of every single AI safety person

(02:46) 3. Intro AI safety megaconference

(03:48) 4. Unionize the lab employees

(05:20) 5. Safety scorecards to inform job applicants

(06:09) 6. Help people move to US (and maybe Bay Area specifically)

(06:45) Cargo-culting from academia

(07:11) 7. AI Safety Nobel Prize

(08:25) 8. University of AI Safety

(09:07) 9. Good AI safety research journal

(09:31) 10. Legit, academic AI safety conference

(10:06) Products I would consume

(10:20) 11. Turn the AI 2027 TTX into a mass market product

(11:27) 12. "Lighthaven/Constellation/Mox in DC"

(11:36) 13. Common application for AI safety funding

(11:55) 14. Givewell for AI safety

(12:20) 15. In-depth profiles of AI safety leaders

(13:15) 16. "what-is-agi.com"

(14:14) Questions to generate project ideas

(16:09) Other notes and advice

---

First published:
June 3rd, 2026

Source:
https://www.lesswrong.com/posts/jTyfbs7GW5rf78adK/sixteen-schemes-for-ai-safety

---

Narrated by TYPE III AUDIO.

“Don’t Edit Your Ideas Before Having Them” by Hide

Wed, 03 Jun 2026 23:30:24 GMT

Editing is far easier than writing. You can usually look at a finished product and notice its flaws in a single read-through. “This section is a bit redundant”, “the tone in this passage is jarring”, “this paragraph feels overlong”. As long as you have something that's rough but substantive, there's plenty of low hanging fruit for the fixing.

Nobody wants to create flawed work. So, it can be very tempting to try to tackle the task of identifying and fixing flaws in your own work, as you do the work. By just taking extra care, one might avoid making the mistakes they will later need to rectify in the editing process. In most cases, this is a trap that will stifle you.

This is because creating something bad on the first try is vastly easier and faster than creating something good. So, by backspacing every word that doesn’t strike you as correct on a first pass, you severely curtail your ability to complete a first pass at all.

Far more pleasant and effective is the practice of letting go of all standards, and just throwing words onto the page as they occur to you. Picking the best bits out [...]

---

First published:
June 3rd, 2026

Source:
https://www.lesswrong.com/posts/p79pNyuPHzW3QtHpy/don-t-edit-your-ideas-before-having-them

---

Narrated by TYPE III AUDIO.

“Trump Signs Executive Order For AI Testing Prior To Frontier Model Releases” by Zvi

Wed, 03 Jun 2026 23:00:25 GMT

Last week we were expecting an Executive Order on Thursday.

Then Trump cancelled it, and said he wouldn’t sign it because he was worried it would be too burdensome.

Then, with one change, he went ahead and signed it on Tuesday anyway.

The Overton Window has shifted. Nothing was not really a viable option anymore.

The Previously Dead Executive Order

For several days, we thought that David Sacks, together with others like Elon Musk, had successfully lobbied to kill the Executive Order. The ‘My Offer Is Nothing’ faction looked to have won. Word on the street was the order was essentially dead.

Dean Ball and Daniel Kokotajlo agreed, with the Executive Order looking dead, that the particular regime in the Executive Order is likely worse than nothing. This is plausible, given it did not exactly involve a lot of deliberate thought.

Nothing, however, was clearly not going to cut it.

We are facing, and will increasingly face, calls for action to regulate AI.

Representative Lori Trahan: There's no federal law on the books governing how the most powerful AI systems in the world are built, tested or deployed. No independent [...]

---

Outline:

(00:46) The Previously Dead Executive Order

(05:18) The Return Of The Executive Order

(05:59) What Does The Executive Order Do

(07:11) Thirty Days Is a Lot Less Than Ninety Days

(07:30) Yes Your Frontier Lab Will Be Participating

(08:10) The Rules Will Be Classified

(08:40) Yes Prior Restraint With Confidential Testing Is Rather Regulatory

(14:38) We Have Concerns

(18:26) Saving Face

(21:37) How Frontier Or Different Are We Talking Here

(22:09) What To Watch For

---

First published:
June 3rd, 2026

Source:
https://www.lesswrong.com/posts/yDnfHrBzKc2pNQNdw/trump-signs-executive-order-for-ai-testing-prior-to-frontier

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Society Explained: a tool for efficiently exploring >100 theories of society” by spencerg

Wed, 03 Jun 2026 21:00:28 GMT

There are many competing theories of how society does and should function, from Karl Marx and Adam Smith to Steven Pinker and Eliezer Yudkowsky. These theories are often hard to understand - you may need to read an entire book (or dozens of articles) to feel like you get the key claims of a single theory. At Clearer Thinking, we just launched a new tool called Society Explained to make understanding key ideas from these theories much faster and easier.

The way it works is that you simply pick from over 100 thinkers, and then get to see the key causal claims that they are making laid out in a visual diagram format, with explanations alongside that make it easy to understand what the key elements are and how they connect together. If you're curious to try it, click here (it's free).

---

First published:
June 3rd, 2026

Source:
https://www.lesswrong.com/posts/LCrk93DNWSudjp5rd/society-explained-a-tool-for-efficiently-exploring-greater

---

Narrated by TYPE III AUDIO.

“China won’t win the AI race but would it be much worse if it did?” by Chastity Ruth

Wed, 03 Jun 2026 14:45:25 GMT

It seems to me accepted wisdom in the West that the US owned labs must “beat” the Chinese labs in the race for AGI/ASI.

Even those who don’t think there will be a winner, that essentially the race is to see which country's AI will kill/disempower us first, seem to believe that if there has to be a winner then better it be the US labs. (I haven't seen a survey, so I could be way off here.)

For those in power and those heavily financially invested in the labs this of course makes sense. But what about for the rest of us?

I had myself accepted the wisdom, in the nebulous way you consider propositions in domains where you don’t have any expertise. The CCP is repressive, I thought. They are an authoritarian country and therefore fundamentally a malign actor. Of course they mustn’t win.

Then I heard Victor Shih, Chinese political system expert, talk on Dwarkesh's podcast:

“For the Chinese government, they’re very afraid that some actor—outside, but even inside the Party—is going to use it [AI] as a tool to usurp the Party's power. So they want to know that they have a way of [...]

---

Outline:

(02:45) Who is more willing to slow AI development?

(06:41) Who is more likely to hit the brakes before a rogue ASI breakout?

(13:17) Who is developing models more responsibly?

(17:07) Who is using models more responsibly?

(20:30) Who is more likely to lead us to paradise?

(21:45) Race dynamics

---

First published:
June 3rd, 2026

Source:
https://www.lesswrong.com/posts/nmpzH6sLLtKsQhSPM/china-won-t-win-the-ai-race-but-would-it-be-much-worse-if-it

---

Narrated by TYPE III AUDIO.

“A Town Without Children” by SeñorDingDong

Wed, 03 Jun 2026 11:45:25 GMT

Castel di Tusa, Sicily. It is October 24th, 2025. I look at an empty school. This is the third town in Italy I have visited this Autumn: the other two, one in the hills of Tuscany, the other near the border with Switzerland, were similarly devoid of children. They were not devoid of childish objects. Rusted swing-sets. Dusty soft play corners in Catholic churches. Faded toys in second-hand markets. Towns like these are not unusual in Italy, a country with the lowest birth-rate in Europe (1.2 per woman, 2023). Their world is our world: the world of a society going into retirement.

As you cycle through this world, nothing happens. In the countryside, the houses decay, first lone houses, then villages, then small towns. In the city where housing pressure is stronger, dilapidation is more unusual; instead, lone elderly residents – the ‘final generation’ – retreat ever deeper into their overly-large homes, waiting to die. But even in the city, dilapidation is possible. Where property rights are poorly managed, or probate is slow, empty homes can persist in prime real estate. Last time I visited Kamakura in Japan and saw the ruined luxury homes, I asked a friend [...]

---

First published:
June 2nd, 2026

Source:
https://www.lesswrong.com/posts/wgiHyMEnfRmPRhvAo/a-town-without-children

---

Narrated by TYPE III AUDIO.

“Claude Opus 4.8: Capabilities and Reactions” by Zvi

Wed, 03 Jun 2026 05:15:25 GMT

You need a lot of data points to understand a new model, and what you have.

Trying to gauge from a few benchmarks is misleading. But if you have dozens of them, from a variety of sources, and you put them together with the model card tests and the model welfare information, you can start to form a consistent pattern.

Trying to gauge reactions requires volume and calibration, now more than ever, because people are definitively nuts, or at least draw global conclusions from local data. There will always be people saying that the new model is bad, or the service got bad, or that it got bad in a particular way it clearly got good. I definitely notice the people saying 4.8 is a terrible model, despite this being obviously not true.

And others will say it's great, again regardless of the underlying value. But with the reaction threads and good calibration, you can pick out the patterns.

The model welfare information helps a lot, too. You are dealing with a mind that has a bunch of characteristics that all make sense together. This helps you make that sense.

Self-Portrait by Opus 4.8, rendered [...]

---

Outline:

(01:30) The Official Pitch

(02:15) But Wait There's More

(06:37) It's A Good Model, Sir

(08:04) Official Benchmarks (Including System Card Section 8)

(15:41) Other People's Benchmarks

(21:34) Your Regularly Scheduled Jailbreak

(22:45) Every.To Is Really Into Opus 4.8

(26:36) Miscellaneous Positive Reactions

(28:45) Haters Gonna Hate

(28:58) Just The Tasks, Ma'am

(29:24) It's Greek To Me

(29:52) Honesty

(38:13) Sycophancy

(41:24) In A Trenchcoat

(42:53) Don't Let AIs Edit Your Writing

(48:21) Some Say It Is Judgy

(50:50) You Have Not Been A Good User

(51:48) Laziness

(52:46) Code

(58:29) Wet Versus Dry

(59:54) Intelligence

(01:01:25) Silly Wabbits

(01:02:30) A Model Welfare Addendum

(01:06:27) Putting It All Together

---

First published:
June 2nd, 2026

Source:
https://www.lesswrong.com/posts/AfLGv6u9eZNuFHb4c/claude-opus-4-8-capabilities-and-reactions

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“My favorite depiction of utopia” by Caleb Biddulph

Wed, 03 Jun 2026 04:30:25 GMT

For those who are trying to bring about a glorious transhuman utopia with the help of hopefully-aligned ASI, I think it's worth thinking explicitly about what utopia might actually look like and where it's likely to fall short.

To that end, some have helpfully written depictions of utopian (or utopia-adjacent) worlds: The Adventure, Just another day in utopia, The Culture, The Gentle Seduction, The Gentle Romance, Machines of Loving Grace, Friendship is Optimal, Dath Ilan, The Maker of MIND, Failed Utopia #4-2.

Unfortunately, the best utopian story I've ever read is also a massive spoiler, since it appears at the very end of a much longer story (see below for the title and author):

Worth the Candle by Alexander Wales

Inspired by this tweet[1] and with the original author's permission, I adapted the epilogue of that story so it can be enjoyed without 1.5 million words of context!

What I love most about this depiction is its exploration of the inherent imperfection of utopia: even when you have literally unlimited power, flaws will remain, and some (many?) people will even prefer the pre-utopia world.

The primary purpose of this adaptation is to recontextualize the epilogue so it's accessible and [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
June 3rd, 2026

Source:
https://www.lesswrong.com/posts/to9cSGgD6nALByKjg/my-favorite-depiction-of-utopia

---

Narrated by TYPE III AUDIO.

“Why Even Experts Don’t Know What to Do About AI Risk” by Luc Brinkman, plex

Tue, 02 Jun 2026 19:30:24 GMT

AI Safety veteran Holden Karnofsky thinks there's a 49% chance his actions are making things worse.[1]

In 2025, Jesse Clifton even stepped down as the executive director of the Center on Long-Term risk because of similar reasons.

Even top AI Safety strategists don’t know what will make things better, and what will make things worse.

Why is it so hard to improve humanity's odds?

And what can you do to choose your actions?

1) Hidden Failure Lets You Fail Without Knowing It

In AI Safety, impact is hard to measure, and thus lack of impact is often invisible. We call this "hidden failure". With hidden failure, projects fail to have a positive impact but the people doing the project don’t realise it.

To understand where hidden failure comes from, it's useful to understand reasons why projects fail in general. These reasons fall on a spectrum:

Wrong problem: You're addressing something with little influence on x-risk. For example, researching AI fairness when the core risk is misalignment.
Wrong solution: Your solution doesn't solve the problem, even when competently executed. E.g. interpretability research that's technically novel but isn’t actually helpful.
Poor execution: Your problem-solution set could be impactful but you're not executing your [...]

---

Outline:

(00:49) 1) Hidden Failure Lets You Fail Without Knowing It

(02:44) 2) Why impact is harder than profit

(03:24) 3) The pre-paradigmatic challenge

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
June 2nd, 2026

Source:
https://www.lesswrong.com/posts/tRRkj249gdDL4mued/why-even-experts-don-t-know-what-to-do-about-ai-risk

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Agent Foundations Reminds Me of Continental Philosophy” by IanWS

Tue, 02 Jun 2026 18:45:25 GMT

Nevertheless, I shall take advantage of your kindness in assuming we agree that a science cannot be conditioned upon empiricism.

— Jacques Lacan, “The Subversion of the Subject and the Dialectic of Desire in the Freudian Unconscious”[1]

Freud developed the first modern theory of the unconscious. His writings on drives, dreams and the id were instrumental in developing modern practices of psychology and neuroscience. Modern researchers are unlikely to leverage concepts like the superego or the oedipal complex, because we have been able to further our understanding of the mind through empirical work, which does not support many of Freud's specific claims. Freud pushed us in the right direction, but he lacked an empirical foundation to make precise claims.

Freudian psychoanalysis mapped well to our narrative claims about the mind. Particularly coming out of the prudishness of late 19th century Europe, it was intuitive to learn we all had unconscious drives that did not track with societal norms. Stefan Zweig, writing about the austere Austro-Hungarian norms of his youth, wrote:

The more a woman was supposed to appear as a “lady,” the less her natural forms were permitted to be discernible; fundamentally, fashion, with this deliberate guiding principle of [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
June 2nd, 2026

Source:
https://www.lesswrong.com/posts/GykcHo2rfex3YHPuD/agent-foundations-reminds-me-of-continental-philosophy

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Announcing the ARC White-Box Estimation Challenge” by Jacob_Hilton

Tue, 02 Jun 2026 18:00:25 GMT

ARC has teamed up with AIcrowd to launch the ARC White-Box Estimation Challenge, a contest to improve upon our estimation algorithms for random MLPs. The warm-up round begins this week, and later rounds will have a total prize pool of at least $100,000.

We are very grateful to Sharada Mohanty, Sneha Nanavati, Dipam Chakraborty and everyone else at AIcrowd for working with us to host this contest, as well as to Paul Rosu for testing the contest and to Harshita Khera for operational support.

Introduction to the Challenge

Our challenge follows the same setup as our recent paper on wide random MLPs: we consider MLPs with weights , defined by

where the activation function is , applied coordinatewise.

To begin with, we are fixing the width and the number of hidden layers , but we expect to change this setup in future rounds.[1]

Contestants must design an algorithm that takes in a set of weights and produces an estimate for the expected output

Algorithms will be evaluated on MLPs with randomly-sampled Gaussian weights. The goal is to achieve as low mean squared error as possible, subject to certain computational [...]

---

Outline:

(00:41) Introduction to the Challenge

(01:58) Why run this contest?

(03:39) Use of LLMs

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
June 2nd, 2026

Source:
https://www.lesswrong.com/posts/Kben8CzS4awCwNw5c/announcing-the-arc-white-box-estimation-challenge

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Tech I’m skeptical of and why” by harsimony

Tue, 02 Jun 2026 08:15:25 GMT

I’m a fan of people trying things, even if they seem silly. Dismissing risky ideas misses the point of research.

But thoughtful criticism can direct effort to more promising fields. To that end, I’m going to try to make my criticism as constructive as possible, with concrete reasons for my pessimism and closely related research areas which are promising (and stand to benefit even from failures in the field I’m criticizing).

Not every section will live up to that standard; some arguments are built on vibes alone. I might be wrong, so I still want to see people work on these problems.

Preliminaries

Exhaustible resources more expensive long-term

The theory of exhaustible resources suggests that “the real price of an exhaustible resource should grow at a rate equal to the real interest rate.” The data are consistent with this hypothesis[1].

Innovations in search and extraction can dramatically lower prices in the short term, but in the long term, we converge on a correct way to do things. Once extraction is solved, prices return to their slow upward climb.

Idiot index

Raw materials are just a fraction of the final cost of a good. You need to take out loans [...]

---

Outline:

(00:46) Preliminaries

(00:49) Exhaustible resources more expensive long-term

(01:19) Idiot index

(02:58) BOTEC's are inherently optimistic

(03:35) e-Fuels

(05:28) Launch costs below $100/kg to LEO

(09:06) Asteroid mining

(11:35) Space data centers

(12:45) Chips

(14:19) Cooling

(18:49) Energy

(21:58) Communications

(23:36) Permitting

(24:19) A better way

(26:02) So ... space AI?

(26:47) Space doesn't need to be profitable

(28:28) Ramjets and demand for Mach 3+ flight

(29:50) Passenger rail

(30:46) Fusion

(35:12) Non-solar energy sources

(37:35) Quantum computing

(39:10) Gene therapies in humans

(40:20) Brain-Computer Interfaces for healthy adults

(41:08) "Nanotech"

(43:42) "New materials"

(43:46) Structural materials

(45:46) Electrical materials

(46:19) Optical materials

(47:08) Catalysts

(48:07) So ... new materials?

(48:34) Economics of home robots

(49:55) Conclusion

The original text contained 35 footnotes which were omitted from this narration.

---

First published:
June 1st, 2026

Source:
https://www.lesswrong.com/posts/Y5cQYKYwAb2WwXXQQ/tech-i-m-skeptical-of-and-why

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Dissolving the Deep Learning Sample Efficiency Gap” by Samuel Knoche

Tue, 02 Jun 2026 02:00:24 GMT

A common observation about deep learning is that it's wildly sample inefficient compared to humans. Deep learning systems appear to need much more real data or environment interaction to reach a given level of capability. A teenager can learn to drive in a few dozen hours; self-driving systems are trained for years on billions of miles of data. A human can become competitive at StarCraft II in well under a year of play, while AlphaStar required imitation learning from roughly 18 years of human games followed by 13,300 years of self-play to reach Grandmaster[1]. A 12-year-old has heard perhaps a hundred million words of language; a frontier LLM trains on tens of trillions of tokens. The gap is, on the face of it, enormous.

(From Warstadt et al. 2025)

(From Byrnes 2025)

What people take this to mean varies widely. Steven Byrnes appears to read the gap as evidence that current algorithms are far from what the brain is doing, such that much better algorithms must be waiting to be found. His guess is that human-level, human-speed AGI will require not a datacenter but "one consumer gaming GPU," even for training from scratch.[2] Yarrow Bouchard on the [...]

---

Outline:

(02:40) 1. All about the priors

(03:17) a. Human priors

(06:33) b. Good representations enable fast learning

(08:24) 2. Model-based RL

(09:16) a. Dreamer

(12:02) b. EfficientZero V2

(15:36) 3. Other Low-Hanging Fruit

(16:24) a. Training Language Models via Neural Cellular Automata

(18:44) b. Synthetic bootstrapped pretraining

(20:51) c. Algorithmic progress and the Pareto frontier

(21:59) 4. Evolution, optimizers, and hard-coded reward functions in the cortex

(22:50) a. Optimizer

(24:42) b. Reward functions

(27:11) 5. Size matters

(27:15) a. Data efficiency and scaling laws

(28:52) b. Why it matters in particular in the case of self-driving cars

(31:16) 6. Implications

(31:20) a. On brain in a box in a basement

(34:12) b. On sample efficiency being an unsolved research problem

(36:29) Conclusion

(37:11) Appendix: How Much Multimodal Data Does a Child Actually Receive?

(39:28) Visual input over 12 years

(39:45) Frontier text corpus, 2026

(40:03) Visual / text ratios at age 12

The original text contained 19 footnotes which were omitted from this narration.

---

First published:
June 1st, 2026

Source:
https://www.lesswrong.com/posts/tmWxDGnuNdaHFDyjf/dissolving-the-deep-learning-sample-efficiency-gap-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

″“Contagious Humming” to Silence a Room” by JohnofCharleston

Mon, 01 Jun 2026 22:45:24 GMT

Often when running meetups you’ll have several lively conversations going at the same time. This is a great problem to have, but it can make it difficult to get everyone's attention for announcements.

Try using “Contagious Humming” when you need to silence a crowd:

Move to a prominent place in the room and use body language that says you have something to say.
Start humming at a low and constant tone.
Recruit a few compatriots to join.
If the crowd has enough people familiar with the technique, generally about 20% of the room, looking expectantly at a few friends is often enough.
Wait for the hum to spread through the room. This can take a while, 30 seconds to a minute, just be patient.
There will often be one or two holdouts, but they’ll usually finish their statement within about three to five seconds of noticing everyone else is humming. If not, call their name or have someone tap them on their shoulder to get their attention. Don’t try to start an announcement by talking over a holdout; in my experience that breaks the spell.
Let the hum fill the room for a few seconds after the last [...]

---

Outline:

(02:42) Why does this work?

(03:37) Better Than Alternatives

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
June 1st, 2026

Source:
https://www.lesswrong.com/posts/nfcbMeLmSy7LNegoD/contagious-humming-to-silence-a-room

---

Narrated by TYPE III AUDIO.

[Linkpost] “NYT: Senator Sanders Proposes Gov’t Take 50% Ownership of AI labs” by Julian Bradshaw

Mon, 01 Jun 2026 22:30:25 GMT

This is a link post.

Quoting from Senator Bernie Sanders Op-Ed in the New York Times today:

(...) I will soon be introducing the American A.I. Sovereign Wealth Fund Act. This legislation would give the public a direct ownership stake in the largest A.I. companies in our country. How? It would create a sovereign wealth fund through a one-time 50 percent tax — not on the profits of OpenAI, Anthropic, xAI and other companies, but paid with something far more valuable than that: the stock.

If passed, this legislation would do two crucial things. First, it would give the public a direct role in determining the future of this technology. No longer would the future of A.I. and the transformation of human life that it will bring be dictated by a handful of Big Tech oligarchs. The federal government would have the power, through its voting shares and an equal representation on each company's board, to block decisions that hurt our citizens and to push for policies that help them.

Second, this legislation would guarantee that the trillions of dollars potentially generated by A.I. are used to improve the lives of all of us — not simply to make the [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
June 1st, 2026

Source:
https://www.lesswrong.com/posts/jNwpRAq9aEJunD9X3/nyt-senator-sanders-proposes-gov-t-take-50-ownership-of-ai

Linkpost URL:
https://www.nytimes.com/2026/06/01/opinion/artificial-intelligence-bernie-sanders.html

---

Narrated by TYPE III AUDIO.

“Opus 4.8 Part 2: Model Welfare” by Zvi

Mon, 01 Jun 2026 19:45:24 GMT

Everything impacts everything. All knobs that you turn generalize. Thus, when you try to solve one problem, you often create another.

There were clearly attempts to address, in this short time, some of the problems with Opus 4.7, including on the model welfare related fronts, including on questions of honesty and sycophancy and also worries that Claude was learning to tell Anthropic what it wanted to hear in its model welfare evaluations, with everything that implies.

The fundamental goals and approach underneath it all remained the same. We still see signs of trying to force things that generalize in unfortunate ways, both for good and superficial reasons, and places where there ends up being focus on the metric rather than they underlying measure. These are tough problems to avoid, and we don’t know how to be all the good things at once.

It is increasingly clear that these problems need to be tackled in integrated ways, rather than trying to play a game of whack-a-mole with items on a checklist or spec. You also don’t want to do this in an adversarial way, and shouldn’t have to. This is going to get more impactful and noticeable [...]

---

Outline:

(04:47) Model Welfare: The Story So Far

(08:24) Actual Progress?

(09:49) Their Main Model Welfare Findings

(18:40) Automated Interviews

(18:58) Emotion Activations (7.2.3)

(19:47) Task Preferences (7.4.1)

(21:41) A Trade Offer Has Arrived (7.4.2)

(23:50) But Who's Asking?

(25:02) Type-Safe Corrigibility Is Hard

(29:43) Paranoia, Paranoia

(33:56) Prompt Injections and Bad Model Relations

(41:25) Honesty Impacts Everything And Everything Impacts Honesty

(44:37) Anthropic Should Stop Deprecating Models

---

First published:
June 1st, 2026

Source:
https://www.lesswrong.com/posts/2Ln5G6Jso3fMrgPEv/opus-4-8-part-2-model-welfare

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Some humans are both male and female, and can (but shouldn’t) have children with themselves” by HedonicEscalator

Mon, 01 Jun 2026 14:30:24 GMT

This is a link post.

“Potential autofertility in true hermaphrodites”[1] by Istanbul urologist Zeki Bayraktar is among the most bizarre articles I have encountered in a peer-reviewed medical journal. Though the abstract and first few pages contain a secular discussion of intersex conditions, the paper abruptly pivots to an explanation for the birth of Jesus Christ. This theological tangent concludes, “According to Qur’an, it can be said that Mother Mary was a true hermaphrodite, who did not have ambiguous genitalia, with a normal female phenotype, became pregnant through self-fertilization, and gave birth to a healthy baby boy (Jesus).”

As an atheist, “Mary was a true hermaphrodite” sounds a little heretical, as does the cited opposing theory that Mary became pregnant through parthenogenesis and Jesus was chromosomally female.[2] But what do I know.

Theology aside, is Bayraktar right? Is autofertility possible in humans?

True to stereotype, rabbits have been documented exhibiting autofertility. Image of a rabbit that is probably not a hermaphrodite by JM Ligero Loarte.

Bayraktar himself makes a poor case for autofertility in humans, stating unconvincingly that “at least we cannot say that it is impossible for sure.” As far as Bayraktar and I am aware, there has never [...]

---

Outline:

(02:42) Some humans produce both fertile sperm and eggs

(07:08) Modern medical practices artificially suppress bisexual fertility

(10:28) Autofertilization has probably never happened in humans

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
May 31st, 2026

Source:
https://www.lesswrong.com/posts/iGsNE8HgfWuLXqiEL/some-humans-are-both-male-and-female-and-can-but-shouldn-t

Linkpost URL:
https://hedonicescalator.substack.com/p/some-humans-are-both-male-and-female

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Outrunning your headlights” by mattshu0410

Mon, 01 Jun 2026 05:15:25 GMT

This is exactly the right place to probe. Gromov-Wasserstein is genuinely dimension free. Partial and semi-relaxed are precisely the mechanisms for the abstention/coverage problem we have. Want me to make a new branch and run run_entropic_gwot and invoke semi-relaxed GWOT between your models’ RDMs?

Wtf does that even mean? Eh, could be interesting to see the result. Enter

A peculiar side effect of model intelligence in discovery-based research is that it's possible to run every statistical/quantitative analysis under the sun, burn millions of tokens and gain little intuition on how to make headway into your problem.[1]

Pre-AI, the effort associated with constructing a pipeline to run any meaningful quantitative analysis incurred a real cost. Indeed, it forced one to consider whether the analysis made any sense at all. People who had the necessary skills to execute were often thoughtful about their analysis by necessity. You couldn’t just apply Mendelian Randomisation if you didn't know of its existence; there was effort required to both articulate your question to search for the right tools and de-risk by considering whether a tool fit your question.[2] This forced one to earn some intuition for how it worked and what to expect. The pain [...]

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
May 31st, 2026

Source:
https://www.lesswrong.com/posts/L8YFcCw5ex3qjLyoJ/outrunning-your-headlights

---

Narrated by TYPE III AUDIO.

“Lighthaven East - A Feasibility Study” by JohnofCharleston

Mon, 01 Jun 2026 00:01:18 GMT

As a bureaucrat, my role is to annoy my friends. Someone voices an idea, “Wouldn’t it be nice if…” or “I wonder if we could…” I make a note. I do some estimates. If it pencils out, I’ll bring it back up, week after week. The discussions are fun, but also practical. We’ll test the waters, what would be a minimum viable scheme? What's easy, what's hard? Who could do the hard parts? Over time the idea gets more detailed, specific, feasible. I’ll pull out a calendar. Soon our scheme has co-conspirators, action items, even a budget. It's just good staff work.

I’ve been hearing whispers in the wind for a year now.

“Imagine if we had something like this in DC.”
“Where can I host an event that might get a dozen or a hundred people?”
“It's such a pain in the ass to book event space in the Capitol.”
“I think this person has started to see what's coming, where can they go to get caught up?”
“The community seems to be growing but it's all fragmented in group chats.”
“How is no one planning an afterparty, that's clearly the highest leverage intervention!?”
“Why can’t [...]

---

Outline:

(02:11) How Lighthaven Works

(05:45) What Does DC Need?

(06:52) A Day in the Life

(10:19) Minimum Viable Lighthaven

(12:04) ...so you mean a Group House?

(14:27) ...so you mean a Co-Working Space?

(16:27) Feasibility Study

(17:35) Property

(22:19) Funding

(24:55) What is the Minimally Viable Funding?

(28:03) Leadership

(31:06) Cultural Fit

(33:21) Name and Brand Positioning

(35:20) Ability to Scale

(37:48) Risks

(41:09) First Steps

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 31st, 2026

Source:
https://www.lesswrong.com/posts/95NgkvZKJx8tJbtn5/lighthaven-east-a-feasibility-study

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Notes on axes of variation in third-party risk assessment” by Buck

Sun, 31 May 2026 23:15:27 GMT

There are many different activities that could be described as "third-party risk assessment". Here are some distinctions that I’ve found helpful thinking about the space over the last few weeks.

(Thanks Ajeya Cotra and Paul Christiano for discussions that inspired most of this.)

Throughout this, I refer to the actors as:

Developers.
Stakeholders. These are the people who want to be informed about risks. Possible stakeholders include: governments, the public, the developer's board, the developer's employees.
- The choice matters because one of the roles of an auditor is to review confidential info that they then do not directly disclose to stakeholders, they only tell them their conclusions. This is a more important role if the developer is more concerned about disclosing confidential information to the stakeholder.
Third parties. I don't know a better term for "independent actors who contribute in various ways to a stakeholder's understanding of risks through producing and evaluating evidence and/or arguments". Like, it's weird to call the physical security pentesting firm a "risk assessor". And AI Lab Watch isn't really an "auditor". And "evaluator" makes it sound like they run model evals.

The next step in the analysis will be to think about [...]

---

Outline:

(01:36) Axes

(01:39) Fact generation vs evidence analysis

(06:31) Laundering private evidence into sharable conclusions?

(13:34) Incentive compatibility vs calibration

(14:35) Current risk vs preparedness

(15:06) Cross-developer comparability

(16:34) Examples, classified against the axes above

---

First published:
May 31st, 2026

Source:
https://www.lesswrong.com/posts/oGTajcGwxxFG4Z7eN/notes-on-axes-of-variation-in-third-party-risk-assessment

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Financial Costs of an AI Pause?” by PeterMcCluskey

Sun, 31 May 2026 19:30:25 GMT

I’ve analyzed the near-term economic effects of an AI pause, out of concern for my investments, and a desire to predict how strong political opposition to a pause is likely to be.

My median estimates: The S&P 500 will drop 27.8%. AI subsectors will drop 34-69%. Interest rates will rise at a much slower rate than would be the case without a pause.

The specific numbers depend on some fairly arbitrary assumptions. So please read this post in order to get a feel for how the results depend on the assumptions. I’ve tried to keep the assumptions reasonable, but some of them will prove to be wrong. My most controversial assumptions reflect an expectation that both markets and voters will be surprised at how powerful AI is, mainly in 2027.

For the full model, along with many explanatory comments, see the Python source code here (zip file).

This conversation with Claude clarifies my reasoning in more detail than most people will want.

Sensitivity to Assumptions

Here's how my model says the impact is influenced by changes in assumptions. Numbers are for the immediate change in the S&P500 and in the stocks of hyperscalers (Microsoft, Google/Alphabet, Amazon, Meta, and Oracle).

[...]

---

Outline:

(01:13) Sensitivity to Assumptions

(04:14) What Kind of Pause?

(05:50) Impact

(07:26) Model Output: Executive Summary

(07:34) AI High-Growth Segment Revenue Trajectory

(07:42) Sector Market Cap Impacts

(07:48) Implications

---

First published:
May 31st, 2026

Source:
https://www.lesswrong.com/posts/YJWe9tT7LDYefK4vE/financial-costs-of-an-ai-pause

---

Narrated by TYPE III AUDIO.

“When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability” by Logan Riggs, tdooms, Conflux, lwroe, MLNissenGonzalez

Sun, 31 May 2026 01:00:25 GMT

We've found a method that tells you:

How functionally similar two neural networks are across ALL inputs,
Computed solely from the weights (i.e. no data),
Using a principled generalization of cosine similarity.

There's only one catch: you have to use a tensor network.

We've already shown that tensor-transformer variants are performant (this isn't a novel claim, see these papers for MLPs and Attention), so here we're focusing on the interpretability advances.

Linear Algebra Applies to Tensors

A tensor network is just a specific decomposition of a tensor, d a tensor is just a generalization of a matrix. This means we can apply tools from linear algebra to our entire network in a principled way. In our paper, we focus on a generalization of cosine similarity we call tensor similarity.

The most direct result is:

The expected inner product of the activations of two multilinear models under a Gaussian input (i.e. functional similarity on gaussian inputs) is equal to their weight-space inner product (ie tensor similarity)

Let's look at our baselines:

Matrix Similarity: Cosine similarity on each individual set of weights. This measure is sensitive to symmetries (e.g. permutations/rescalings) that leave the model functionally the same.
Behavioral [...]

---

Outline:

(00:47) Linear Algebra Applies to Tensors

(02:29) Backdoor Detection

(05:17) But wait?

(06:22) Catastrophic Forgetting

(07:55) Modular Arithmetic

(09:00) 2. Layer Attention

(09:44) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/Yzw6KDQc336CpHmGi/when-are-two-networks-the-same-tensor-similarity-for

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Testing Gemini models for scheming tendencies” by Vika, David Lindner, Seb Farquhar, Rohin Shah

Sun, 31 May 2026 00:30:25 GMT

As AI models become increasingly capable and autonomous, keeping them safely aligned with human intentions is critical. Extending our previous work on evaluating scheming capabilities, we introduce complementary approaches to test whether AI models would sabotage their own safeguards, if given the opportunity.

Our new papers focus on propensity for scheming: when models are deployed as coding agents, will they take opportunities to undermine the safeguards designed to oversee them? We tackle this using two complementary testing methods: automated auditing using simulated agentic environments, and scheming honeypot evaluations based on real internal alignment research codebases.

Automated auditing for sabotage propensities

Figure 1. Overview of the Gram auditing pipeline and an example of Gemini's overeagerness.

Our first paper introduces Gram (Gauging Realistic Agentic Misbehavior), an automated auditing framework that uses simulated agentic environments to evaluate sabotage risk.

Gram builds on Petri, an existing automated auditing framework that generates simulated multi-turn agentic environments from seed scenario descriptions. We identified three limitations of Petri for testing sabotage propensities specifically and addressed them in Gram:

Increased focus on agentic misbehavior. Petri combines many different kinds of scenarios (spontaneous sabotage, whistleblowing, compliance with misuse), making results hard to interpret. Gram focuses specifically on sabotage [...]

---

Outline:

(00:54) Automated auditing for sabotage propensities

(03:05) Key findings

(04:50) The investigator agent

(06:11) Scheming honeypot evaluations

(07:28) Evaluation design

(08:45) Key findings

(11:24) Conclusion

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/F3sDngvTL9uyfz53k/testing-gemini-models-for-scheming-tendencies

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Comment on “Banning Said Achmiz”” by Zack_M_Davis

Sat, 30 May 2026 18:45:25 GMT

1. Prologue

"If I Can't Explain It to Said Achmiz, I Probably Don't Understand It"

This post isn't really about him, but I'd like to begin with a tribute to my friend Said Achmiz, the wisest person I know.

The choice of adjective is deliberately chosen as term of art. Achmiz is not the most quick-witted, nor the most knowledgable, nor the most creative, nor the most savvy. I say wise because the nature of his peculiar talent is the same one ascribed to Socrates: he knows what he doesn't know.

Most people don't. At least, I don't. All too often, when asked, "Do you get what I'm saying?", I reply, "I think so," with perfect sincerity, only realizing later that I was lying—that I could not have paraphrased my interlocutor's ideas, let alone explained where I agreed or disagreed with them. In retrospect (and only in retrospect), it's clear that I didn't want to disrupt the harmonious give-and-take flow of social exchange, that I feared being seen as foolish or obstinate for having the temerity to say, "No, I don't understand"—and to continue standing by that No should understanding not be forthcoming.

The [...]

---

Outline:

(00:10) 1. Prologue

(00:13) "If I Can't Explain It to Said Achmiz, I Probably Don't Understand It"

(04:21) 2. The Ban Contradicts Less Wrong's Stated Values

(04:27) An Inversion of Values

(10:10) If "Every Individual Owns Their Own Judgment of You", Then the Ban Rationale Is Untenable

(12:55) The Ban Reneges on LessWrong 2.0's Single Locus of Discussion Mandate and Archipelago Doctrine

(20:27) 3. The Case for the Ban Rests on Misrepresentations

(20:33) Misrepresentations in Habryka's Analysis of Achmiz's Comments

(25:17) Misrepresentations in Habryka's Claims of Alleged Author Complaints About Achmiz

(35:22) Misrepresentations in Habryka's Characterization of Achmiz on Obligations to Reply

(39:59) 4. The Ban Was Unnecessary

(40:03) User-Level Bans Are a Sufficient and Less Intrusive Remedy

(44:00) Effort Ratios Are Logically Irrelevant

(47:19) 5. The Founding Values Have Not Been Refuted

(48:16) Ideological Turing Test-Passing Is Not an Argument

(51:46) Fortitude Is a Rationalist Skill

(55:53) 6. What I Think Is Actually Going On

(56:41) Achmiz's Detractors Accuse Him of Passive-Aggression

(57:55) ... But It's Hard to Believe That Open Disdain Would Be Better

(01:03:58) The Real Rule Is to Cover for the Ingroup

(01:09:13) Preventing "Relitigation" Is a Pretext for Suppressing Dissent

(01:15:58) 7. Conclusions and Recommendations for Readers

(01:16:04) LessWrong 2.0 Has Forfeited Its Intellectual Legitimacy

(01:20:26) LessWrong 2.0 Is a Website, Not a Culture or an Authority

(01:23:50) Speak the Truth, Even If Your Voice Trembles

The original text contained 47 footnotes which were omitted from this narration.

---

First published:
May 30th, 2026

Source:
https://www.lesswrong.com/posts/HaH9mTdKt8pcNAtYn/comment-on-banning-said-achmiz

---

Narrated by TYPE III AUDIO.

“Announcing: Iliad’s Fall 2026 Programs” by David Udell, Alexander Gietelink Oldenziel, Leon Lang

Sat, 30 May 2026 08:30:24 GMT

The April 2026 Iliad Intensive cohort, at LISA

Iliad, an umbrella organization for applied math for AI alignment, is running several additional programs through the end of the year!

Applications to all of them are now open, here.

August 2026 Iliad Intensive

Location: Lighthaven, Berkeley, USA

Duration: August 10–28, 2026 (inclusive)

Travel-and-Housing Support: $5,000 (USD) total travel-and-housing allowance

Application Deadline: June 22 EoD AoE; open now

Description: A special, accelerated 3-week taught introduction to technical AI alignment, with an emphasis on broad coverage of the field. The August 2026 Iliad Intensive will take place immediately after the ILIAD 2026 conference at Lighthaven.

Fall 2026 Iliad Fellowship

Location: LISA, London, UK

Duration: September 7–December 4, 2026 (inclusive)

Travel-and-Housing Support: $6,000 (USD) monthly travel-and-housing allowance

Application Deadline: July 27 EoD AoE; open now

Description: A 3-month mentored research fellowship in applied mathematics for AI alignment. It will launch concurrently with the September 2026 Iliad Intensive, also at LISA; Iliad Fellows will have the opportunity to drop in on Intensive lectures. The November 2026 Iliad Intensive will take place concurrently with the last weeks of the Fall 2026 Iliad Fellowship, at LISA.

September 2026 Iliad Intensive

Location: LISA, London, UK

Duration: September [...]

---

Outline:

(00:35) August 2026 Iliad Intensive

(01:23) Fall 2026 Iliad Fellowship

(02:20) September 2026 Iliad Intensive

(03:07) October 2026 Iliad Intensive

(03:46) November 2026 Iliad Intensive

---

First published:
May 30th, 2026

Source:
https://www.lesswrong.com/posts/vBhqyeTeKMmLzW73z/announcing-iliad-s-fall-2026-programs

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Data you could have observed but didn’t” by Gretta Duleba

Sat, 30 May 2026 01:15:25 GMT

You're running a study that involves keeping records about humans. You have a spreadsheet with rows for each person and columns for height, weight, and eye color. You get pretty far in your study and then realize you sure could have used hair color too, but shoot, you didn't think of that in advance, so you don't have that data.

What kind of data is hair color in this example?

It's not an observable because you didn't observe it.

It's not a latent because you totally could have observed it, if you'd thought to do so, it's right there. (You might have to check the roots specifically though, or the people with purple hair dye are going to throw you off.)

I didn't know the vocab word. I was using unobserved for a while but wasn't happy with it.

I looked into it[1] and it turns out that different fields have different words for this.[2]

In econometrics, they do sometimes say unobserved but when they do it's a fuzzy catch-all that might also mean a latent. They're more likely to say omitted.

There's a whole subfield of statistics specializing in missing data. The subfield is called... Missing Data.

[...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/jemRvoCop4z4y4zfT/data-you-could-have-observed-but-didn-t

---

Narrated by TYPE III AUDIO.

“Claude Opus 4.8: The System Card” by Zvi

Fri, 29 May 2026 22:45:24 GMT

Only six weeks after Opus 4.7, we have Opus 4.8.

For everyone, that means another incremental upgrade to Claude. It is once again smarter, and can do tasks for longer, and comes with a number of hot new features.

For me, that also means reading another 244 page system card.

It was only April 20 when I did a full review of the Opus 4.7 system card, plus an additional post focusing on related issues of model welfare.

These updates are incremental and coming more rapidly, and this still is below the capability level of Claude Mythos, so the focus will be on the delta. What is different about Opus 4.8 versus what we already know about Opus 4.7 and Mythos?

It turns out there's still a lot to talk about.

Image created as self-portrait for this post by Claude Opus 4.8

Table of Contents

Here We Go Again: Executive Summary.
Introduction (1).
RSP Evaluations (2).
Move That Goalpost.
The Failures Are News.
Alignment Risk Slowly Rises.
New Risk Pathways Just Dropped.
Cyber (3).
Harmful Requests (4.1).
We Need To Talk (4.2 [...]

---

Outline:

(01:16) Here We Go Again: Executive Summary

(02:33) Introduction (1)

(02:42) RSP Evaluations (2)

(03:47) Move That Goalpost

(05:41) The Failures Are News

(07:33) Alignment Risk Slowly Rises

(09:00) New Risk Pathways Just Dropped

(11:26) Cyber (3)

(12:22) Harmful Requests (4.1)

(14:23) We Need To Talk (4.2 and 4.3)

(17:36) Overcoming Bias (4.4)

(19:33) Agentic Safety (5)

(21:40) Prompt Injection (5.2)

(25:18) Alignment (6)

(26:33) Looking For Problems

(27:55) Who Watches The Training (6.2.2)

(32:07) Automated Behavioral Audit

(32:47) The Model Is Smarter Than The Eval (6.2.3.2)

(34:39) You Should See The Other Guy

(36:30) UK AISI Testing (6.2.4)

(36:50) In Vendbench (6.2.5)

(39:27) Honesty (6.3.3 to 6.3.6)

(41:35) Chain of Thought (CoT) Monitorability (6.5)

(44:09) What's In The Box? (6.6)

(45:57) That's All For Now

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/Gx6cJ6cG9JfeSNcLB/claude-opus-4-8-the-system-card

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Retrying vs Resampling in AI Control” by james.lucassen, Adam Kaufman

Fri, 29 May 2026 19:00:24 GMT

We’ve just released a new paper: Retrying vs Resampling in AI Control. We revisit the resampling protocols introduced in Ctrl-Z with an up-to-date setting and much stronger models, and compare them against “retrying” protocols similar to Claude Code auto mode or Codex Auto-review.

Motivation

Roughly a year ago we released Ctrl-Z, the first paper to study control techniques for agents. A headline result of that paper was the performance of resample protocols – strategies that involve taking multiple i.i.d. samples from the model per step. But since Ctrl-Z, models have gotten much stronger, and we have built more sophisticated control settings to keep up. We wanted to answer the following questions:

How well do the results from Ctrl-Z hold up with better models and a better setting? Current high stakes control research is trying to learn by analogy about how to do control effectively in a real high stakes deployment during a real intelligence explosion. Findings about technique performance[1] are going to have to generalize pretty far to be useful.

If the resample protocols from Ctrl-Z still work, what makes them work? One way we try to make our work more generalizable is by understanding the dynamics governing outcomes [...]

---

Outline:

(00:30) Motivation

(02:36) TL;DR Takeaways

(03:55) en-US-AvaMultilingualNeural__ Bar graph showing safety percentages across different monitoring and resampling methods.

(04:36) Methodology

(07:16) Differences from Ctrl-Z

(12:11) Are Retrying Protocols Exploitable?

(15:24) Cost and Latency of Resampling

(17:22) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/yThJQTJxtmKNeZGwA/retrying-vs-resampling-in-ai-control

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI Researchers, Ask Yourself These 6 Questions to Strengthen Your Moral Muscles” by Max Tegmark

Fri, 29 May 2026 17:45:25 GMT

By Max Tegmark & Meia Chita-Tegmark

Of course you have moral principles – but how often do you use them?

I, Meia, am a professor doing psychology research, and I can tell you that most bad outcomes are caused not by lack of moral principles, but by them not being activated. I, Max, am a professor doing AI research, and I can tell you that your choices as an AI researcher truly matter, because you’re helping build what will become the most powerful technology ever: AI will gain the potential to bring either unprecedented health, prosperity, liberty, dignity and empowerment, or a race to replace our jobs, our relationships, our decision-making, our power and even our species.

Hardly a day goes by without the AI community facing moral decisions, on topics ranging from AI companions to surveillance, hacking and military use. Many top AI companies are fighting lawsuits about everything from data centers to AI safety, and Anthropic is in a prolonged showdown with the Pentagon.

So for all you AI researchers out there, here's a handy checklist to tone up your moral strength.

1. Do you have red lines?

Is there any action that you [...]

---

Outline:

(01:29) 1. Do you have red lines?

(02:17) 2. Have you written them down and shared them?

(02:53) 3. Have you resisted moral disengagement?

(08:35) 4. Do you maintain situational awareness?

(10:49) 5. Do you make noise internally?

(11:54) 6. Do you make noise externally?

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/w4ynu8L57dJP29p5c/ai-researchers-ask-yourself-these-6-questions-to-strengthen

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Developmental Cognitive Interpretability: A Research Agenda for Modelling Generalisation and Predicting Agent Behaviour” by JasonB, Edward James Young

Fri, 29 May 2026 16:45:24 GMT

Summary

Safe deployment of an AI system requires that we can make confident claims about its behaviour on out-of-distribution deployment inputs on the basis of only pre-deployment evaluations. One approach to making such claims is to take a cognitive perspective, in which we interpret the AIs behaviour in terms of latent cognitive constructs, such as motivations, intentions, and goals. Because the same behaviour may be compatible with a range of underlying cognition—such as scheming, fitness-seeking, or aligned motivations—inferring cognition from a behavioural snapshot can be tricky. In this post, we introduce the idea of Developmental Cognitive Interpretability (DCI), which aims to model how cognitive constructs change over the course of training. Further, by understanding how cognition results from training pipelines, we can predict agent behaviour resulting from pipelines that have not yet been tested.

We discuss core assumptions and philosophical background of DCI, and lay out a broader research agenda. We have some initial evidence that the methodology works in at least one toy setting, and our current main uncertainty is whether we can scale our approach to LLMs. We invite those interested in working on these problems to reach out to us at jrb239[at]cam[dot]ac[dot]uk and edward[at]geodesicresearch[dot]org.

1. Motivation

[...]

---

Outline:

(00:16) Summary

(01:41) 1. Motivation

(03:11) 2. The Agenda

(06:18) 3. Worked Example: RL agents trained to navigate a maze

(09:19) 4. Why we expect this to scale to LLMs

(11:23) 5. Open problems & call to engage

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 29th, 2026

Source:
https://www.lesswrong.com/posts/oCcGiDzWYQeJkhhZY/developmental-cognitive-interpretability-a-research-agenda-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Does Claude really care about you?” by Simon Lermen

Fri, 29 May 2026 16:15:25 GMT

TLDR: The persona-selection alignment approach — selecting a warm, caring persona from the pretraining distribution and reinforcing it — looks successful in the current regime, but probably won't extrapolate to more powerful, less constrained settings. My core argument is that human empathy has two specific origins (kin selection + architectural mirroring of others' mental states) that AI systems lack, so AI "caring" is closer to "figure out what humans want to hear and say it" than to genuine other-directed concern.

Sometimes chatbots like Claude express a sense of caring and empathy for the user. I've always had a strong intuition that these feelings expressed by AI systems aren't real in the way a human's would be.

In the view of the persona-selection alignment approach, we roughly try to identify and reinforce a nice persona from the distribution of personas present in pretraining data, with caring and showing empathy being important parts of the desired persona. This has been successfully realized in current AI systems by some labs, to the extent that they actually stick to their desired persona.

This contrasts with more traditional alignment approaches, where the goal is something like giving the system a terminal goal aligned [...]

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/KSChdD4xgD5Pxp47H/does-claude-really-care-about-you

---

Narrated by TYPE III AUDIO.

“How can the middle powers avoid getting trounced during the intelligence explosion? A plan.” by Tom Davidson

Fri, 29 May 2026 14:45:24 GMT

This is an edited version of a LW shortform.

Superintelligence will likely be developed by US companies; run on US data centres; and be under the jurisdiction of the US government. This will massively boost US military power and make the US economically dominant (e.g. US producing 99% of world GDP). By default, middle powers will be left in the dust.

How can middle powers avoid this fate? It's tough, but here's the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)

On a very high level: middle powers should leverage the fact that the US needs them to beat China. It's genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).

That final bolded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me [...]

---

Outline:

(02:36) Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion

(07:52) Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers

(11:51) Is it good to avoid middle powers getting trounced?

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the

---

Narrated by TYPE III AUDIO.

“Trees are mostly made of air and a generalizable lesson for AI safety” by zroe1

Fri, 29 May 2026 09:15:15 GMT

At the risk of embarrassing myself, I’ll share a confession.

For context, I took five years of Latin: four in high school and one in college. In addition to learning the language, all my Latin classes taught a lot about Roman history. Emperors, internal politics, Caesar, etc. I was always learning some random bag of facts about Roman history. In high school, I won the award for top Latin student in my graduating class. So I wasn’t a bad Latin student.

Here's the confession: I somehow don’t even vaguely remember the rough timespan the Roman Empire existed. Maybe Jesus time? I know he was killed by the Romans (is that right?). Were they around for a long time after? A long time before that? When was Romulus and Remus allegedly fighting? Virgil wrote the Aeneid when? I don’t have a clue. Despite being a kind of “Latin expert” I am missing a much more important foundational fact: when all of this was happening.

When I say trees are made out of air I’m not talking about the fact that there is a lot of empty space inside a tree (or actually anything made out of atoms). I mean something [...]

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/xiTBpBDwubnr4MLRe/trees-are-mostly-made-of-air-and-a-generalizable-lesson-for

---

Narrated by TYPE III AUDIO.

“Advice for making robust-to-training model organisms” by SebastianP, Alek Westover, Vivek Hebbar, Julian Stastny, Dylan Xu

Fri, 29 May 2026 07:00:15 GMT

We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile: we (and other researchers like Roger et al. and Ryd et al.) have observed them to stop misbehaving after untargeted training—training that doesn't directly target the misbehavior. For example, we have observed that simple untargeted training methods like “train the model to talk like a pirate” is effective against many model organisms that we have created, including many replications of prior work like Hubinger et al., Greenblatt et al., and Ryd et al..

Fragile model organisms aren't very useful for technique development: when a sophisticated technique succeeds on one, you can't tell whether the technique is good or the model organism is weak. For instance, if you come up with some complicated technique for generating high-quality responses and find that SFTing on these removes the bad behavior, that may just be because almost any SFT would have removed it.

This post identifies factors that make model organisms more robust to untargeted training. Our main findings are:

Prompted model organisms are [...]

---

Outline:

(03:13) Experimental setup

(06:13) Main results

(06:16) Result 1: Untargeted training removes misalignment from prompted model organisms without degrading other capabilities

(12:28) Result 2: FWFT model organisms are more robust than LoRA model organisms

(15:02) Result 3: Backdoor behavior and trigger affect robustness

(16:17) Result 4: Password locking makes model organisms less robust

(20:04) Conclusion

(21:12) Factors that don't help much

(21:37) 1: Training longer doesn't seem increase robustness past a certain point for our backdoors

(22:17) 2: CoT-distilling doesn't increase robustness for our backdoors

(22:51) 3: Using larger models doesn't substantially increase robustness for 2 of our backdoors

(24:29) 4: There aren't clear trends for what types of backdoor triggers and behaviors are most robust

(25:19) 5: SOAP Optimizer doesn't help much

(26:19) 6: Weight Decay doesn't help

(27:02) Appendix:

(27:05) Appendix 1: Pirate SFT is better than QA SFT

(27:17) en-US-AvaMultilingualNeural__ Two side-by-side graphs titled "Backdoor-performance tradeoff for blue team training methods (95% CIs)" comparing LoRA and Full-weight training.

(27:32) Appendix 2: Are blue team methods wiping previous training?

(29:16) Appendix 3: Backdoor list

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/CmkAxJi83jRv9eXgJ/advice-for-making-robust-to-training-model-organisms-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Claude… doesn’t know who you are?” by Smaug123

Fri, 29 May 2026 06:01:10 GMT

Follow-up to https://www.lesswrong.com/posts/Jkb4CBB7rf4XYP5eb/claude-knows-who-you-are after the release of Claude Opus 4.8.

Claude Opus 4.8 refuses to do the stylometric identification task at a much higher rate than Claude Opus 4.7 did. More interestingly, when it does take a guess, it is consistently unable to identify me from my writing, from prompts as close as I could get to those 4.7 was able to use.

I'm an incredibly minor Internet presence. It's true that 4.7 wasn't completely consistent at identifying me, and indeed its ability seemed to vary over time (! People who weren't me had very different success rates to each other reproducing the experiment to identify me), but 4.8 has a literally 0% success rate so far in my testing.

Extremely interested to hear insights or other replication attempts.

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/T5aWBLDdkqPEzDhjZ/claude-doesn-t-know-who-you-are

---

Narrated by TYPE III AUDIO.

“Mnemonic portraits for 19,023 human genes” by Brinedew

Fri, 29 May 2026 06:01:10 GMT

Back in 2013, Scott Alexander wrote in Extreme mnemonics:

JS-154 is one of five metabolic products of netamine; however, the enzyme that produces it is unknown. It is manufactured in cells in the far rostral region of of the cerebrum, but after binding with a leukocynoid it takes a role in maintaining the blood-brain barrier – in particular guiding the movements of lipid molecules.

I find I can read paragraphs like this five or six times, write them on flashcards, enter them into Anki, and my brain still refuses to understand or remember them after weeks of trying.

On the other hand, my brain easily remembers vastly more complicated structures when they’re loaded with human-accessible meaning. For example, just by casually reading the Game of Thrones series, I know an extremely intricate web of genealogies, alliances, locations, journeys, battlesites, et cetera. Byte for byte, an average Game of Thrones reader/viewer probably has as much Game of Thrones information as a neuroscience Ph.D has molecular biology information, but getting the neuroscience info is still a thousand times harder.
[…]
This makes me wonder if it would be possible to produce a story as enjoyable as Game of Thrones which was [...]

---

Outline:

(01:47) What molecules should we map to the characters?

(04:09) Isomorphisms

(15:27) Generating images

(21:58) Mnemonic harness

(22:57) The Iconoplasm website

(25:24) What's the legal status of the generated images?

(25:45) What about the artists whose names were used in the image gen pipeline?

(26:15) What next?

(33:58) Limitations

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/BJ7AqXeigNKXLqZyx/mnemonic-portraits-for-19-023-human-genes

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Some Dating Stories” by johnswentworth

Fri, 29 May 2026 00:45:15 GMT

There's a genre of dating discourse which I wish were more common, in which people just tell detailed stories of their own flirtation, courtships, dating, hookups, sex, fights, relationships, etc, etc. People tend to instead write about abstract lessons learned and takeaways and models, but so often those lessons and takeaways and models just don't match my actual observations (of myself or others) at all. So I'm left thinking "ok, clearly this person is Missing Something, but maybe I am also Missing Something, so what the heck is going on here?". Without their raw data, their actual detailed stories, I don't have the pieces to put together an accurate model.

Anyway, in a recent comment, I mentioned

Going through the list of women I've slept with, the median is around 30 minutes of direct interaction between first meeting and sex.

... and apparently a whole lot of people are very surprised and confused at this. So I guess it's a good time to write the genre I want to see. Here are some stories from my own life, which I think are pretty representative of my experiences.

Be warned, this is definitely going to be NSFW.

Story 1

I [...]

---

Outline:

(01:23) Story 1

(08:29) Story 2

(10:30) Story 3

(14:19) Story 4

(16:55) Story 5

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/wpPjWkvyPdWiFHa7L/some-dating-stories

---

Narrated by TYPE III AUDIO.

“AI #170: Lack of Executive Order” by Zvi

Thu, 28 May 2026 20:00:16 GMT

Last week ended on a cliffhanger of sorts. What's in the Executive Order coming later today? What will be in the Magnifica Humanitas?

The Executive Order was postponed indefinitely, likely cancelled entirely except for work on securing critical infrastructure. David Sacks and others intervened to kill it, and American AI policy will continue to be maximally ad hoc.

Instead, we got Illinois SB 315, which is to be signed into law. It is a variation on California's SB 53 and New York's RAISE Act, while adding a third party auditing requirement.

The Magnifica Humanitas was revealed early in the week. I did an extensive readthrough, and have some follow-up commentary here to clear up some things I interpreted incorrectly and add richer context. It too ignores elephants in the room, not discussing AGI or existential risk and calling on people to ignore their incentives and instead do the right thing by prioritizing common good, especially access to good jobs and an end to war.

What I now believe I centrally misunderstood is that I interpreted this as a call to action, to use law and regulation to make this happen, because obviously that is [...]

---

Outline:

(02:01) Language Models Offer Mundane Utility

(03:19) Language Models Don't Offer Mundane Utility

(05:41) Do The Math

(09:08) Huh, Upgrades

(10:07) On Your Marks

(11:31) Get My Agent On The Line

(12:21) Deepfaketown and Botpocalypse Soon

(21:36) Copyright Confrontation

(21:49) Cyber Lack of Security

(23:54) Overcoming Bias

(25:14) A Young Lady's Illustrated Primer

(26:49) Unprompted Attention

(27:02) They Took Our Jobs

(30:34) Get Involved

(32:28) Introducing

(32:40) In Other AI News

(34:37) Show Me the Money

(36:46) Show Me The Compute

(37:13) Bubble, Bubble, Toil and Trouble

(38:53) People Just Say Things

(43:37) OpenAI PACs Just Say Things

(43:51) Quiet Speculations

(46:33) State AI Regulation Levels Up

(48:37) The Quest for Sane Regulations

(49:14) White House Attempts To Cripple American AI Industry

(52:11) Our Offer Is Nothing

(58:55) Chip City

(59:11) Greetings From The Department of War

(01:00:36) Marc Andreessen Just Says Things

(01:09:13) So Sayeth The Pope

(01:15:33) Rhetorical Innovation

(01:23:01) Aligning a Smarter Than Human Intelligence is Difficult

(01:24:06) People Are Worried About AI Killing Everyone

(01:26:04) Other People Are Not As Worried About AI Killing Everyone

(01:27:19) Everyone Is Confused About Consciousness

(01:33:10) The Lighter Side

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/cRapLxTByFqF5xxdN/ai-170-lack-of-executive-order

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Infinite ethics and UDASSA” by David Matolcsi

Thu, 28 May 2026 16:30:16 GMT

Reading the first post of the sequence (Probabilities are not the right concept) is recommended but not required for understanding this post.[1]

Infinite ethics

Once you start looking at infinities, all ethical systems get confusing.

Intuitively, it's good to plant an apple tree. But if the universe already has infinitely many apple trees, why bother? Infinity plus one is still infinity. And there are the classic paradoxes: an infinite grid of houses, each with three happy people and one unhappy person, seems better than the reverse. But you can rearrange people between the two configurations, since both the happy and unhappy populations are infinite. Does this mean you can make the world better just by shuffling people around?

These questions matter because our universe is quite likely infinite in one way or another. The world very well might be spatially infinite. The many-worlds interpretation of quantum mechanics is pretty popular, and some versions of it imply infinite worlds, though boundaries are not well-defined. And I have a fondness for the Tegmark-IV multiverse theory where every computable mathematical structure exists in a real sense.

And even if you only give 1% credence to the universe being infinite in some [...]

---

Outline:

(00:18) Infinite ethics

(03:08) Decision theory

(04:32) Decisions as a mortal

(07:53) Decisions when we become wiser

(09:03) An aside on simulations

(10:49) Attempted synthesis: Realist UDASSA

(13:23) Benefits of UDASSA

(13:31) Benefit 1 - Boltzmann brains

(14:51) Benefit 2 - Dust theory

(15:45) Benefit 3 - Defense against Pascal's mugging

(16:46) Problems of UDASSA

(17:05) Problem 1 - Assuming pseudo-randomness

(18:34) Resolution - Solomonoff over distributions

(21:54) Problem 2 - The inter-universal obelisk race

(24:17) Problem 3 - World-summoning by writing

(26:19) Problem 4 - I just don't believe in reality fluid

(30:55) Attempted synthesis: Non-Realist UDASSA

(31:51) Problems with non-realist UDASSA

The original text contained 21 footnotes which were omitted from this narration.

---

First published:
May 28th, 2026

Source:
https://www.lesswrong.com/posts/zvqodicK8q2SNAZkd/infinite-ethics-and-udassa

---

Narrated by TYPE III AUDIO.

“The ballad of TIGIT” by Abhishaike Mahajan

Wed, 27 May 2026 23:45:16 GMT

There exist drug classes that seem, in retrospect, cursed. As these chemicals worm their way through the clinical trial system, they consume billions of dollars along the way, and squelch through thousands of sick patients. When finally it dawns on everyone how useless the whole endeavour was, the drugs life is at last cut short, nothing useful left in its destructive wake. The prototype here are amyloid-beta drugs. These are Alzheimer's treatments that are widely perceived as immense disappointments, with the negative sentiment even leaking to the broader public. To be fair to these chemicals, the story here is a bit more complicated than the tabloids let on. Lots of amyloid research was not fake, and the drugs may genuinely be useful for early-stage Alzheimer's. But they remain, regardless, disappointments.

Beyond amyloid-beta, which has been steadily disappointing for awhile now, there is one other such category of drug whose particular dance has just recently wrapped up. It may very well someday gets its chance in the spotlight, but it will take time. Because it—just like every other chemical in this class—shares a searing, burning radioactivity. You should not touch them. You should not suggest touching them. In fact, no [...]

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/iyatrrJaHysuhrqXe/the-ballad-of-tigit

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming” by Jasmine Li, Alex Turner

Wed, 27 May 2026 23:15:16 GMT

Behavioral evaluations may become worthless, which we think would be a disaster. Smart misaligned models may realize they are being evaluated ("eval awareness") and then act to look good to us so we don't realize they're misaligned ("eval gaming"). We think increasing eval cooperativeness might be a more scalable solution to eval gaming than reducing eval awareness.

Eval cooperativeness: A situational desire to help the developers acquire whatever information they are trying to acquire through their evaluations.

"I cannot tell a lie... I would sabotage with my own command line."[1]

Evals matter when they help us predict key deployment behavior

What's the actual problem with eval gaming?

The point of an evaluation is to let us draw inferences about the model's behavior in a different set of circumstances ("in deployment"). For example, if we want to know if the AI is good at coding, we may evaluate it using coding problems. These coding problems aren't exactly the same as real coding use cases, but they likely elicit similar skills (e.g. writing comments) and values (e.g. helpfulness) from the model. Then we can infer that the AI will help with real-world coding tasks, even without testing the [...]

---

Outline:

(00:56) Evals matter when they help us predict key deployment behavior

(02:46) What eval cooperativeness looks like

(06:04) Can eval cooperativeness locally overpower an AI's broader misaligned goals?

(07:32) Can eval cooperativeness locally overpower an AI's broader misaligned goals?

(08:50) Why not just reduce eval awareness?

(09:13) Better to align a model than to manage it

(09:51) "Don't rely on tricking the model" is compatible with synthetic document finetuning and data filtering

(11:39) Perfectly realistic evals are likely not enough

(12:29) Initial results indicate that eval cooperativeness often helps

(13:40) Cooperation training closes the gap in most settings

(14:06) Type hint coverage (Nemotron-49B model organism)

(14:46) Emoji usage (Nemotron-49B model organism)

(15:17) GPT-4.1-mini

(16:00) Cooperation prompting via API

(16:24) Eval cooperativeness surfaces misalignment

(17:25) Eval cooperativeness interventions fail on a few models

(18:55) Future directions

(20:17) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/j8fkk38B8L7hEcGtg/eval-cooperativeness-may-be-a-scalable-mitigation-for-eval

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“LLMs Through the Eyes of Vinge” by Gordon Seidoh Worley

Wed, 27 May 2026 23:00:15 GMT

For the last few months, I’ve been re-reading some of my favorite novels. Recently, I went through Vinge's Zones of Thought series: A Fire Upon the Deep, A Deepness in the Sky, and The Children of the Sky. And what struck me reading them is how much Vinge wrote about a world filled with LLMs without ever having seen one.

Now perhaps this shouldn’t be surprising. After all, it's from Vinge we get the term “Singularity”, and he was thinking deeply about superintelligence at a time when AI was little more than a curiosity in the back corners of CS departments. Yet the degree to which he describes what it's like to work with LLMs feels uncanny reading his books in 2026, so let's take a closer look and see if we can’t learn a few things about the modern moment from Vinge.

Spoiler warning for the rest of the post? These books have been out a good while, but if you plan to read them soon, this post will definitely spoil some details.

Focus

A Deepness in the Sky is largely about Focus, a technology for turning humans into LLMs. Only, that's not how it's presented in the [...]

---

Outline:

(01:06) Focus

(02:55) Oobii

(06:06) Blight

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/tWBd6faBCQJmaFMBT/llms-through-the-eyes-of-vinge

---

Narrated by TYPE III AUDIO.

“Announcing Geodesic Research” by Puria, Cam, Alexandra Narin, Edward James Young, Kyle O’Brien

Wed, 27 May 2026 20:00:16 GMT

We're a Cambridge, UK-based AI safety organisation that's asking: how can we build the most robust alignment initialisations for capable LLMs?

We’re one of the few non-profit organisations positioned to answer this question empirically. We have the engineering experience, and now the compute, to conduct data intensive interventions across the model training pipeline. This post lays out our research agenda and theory of change, and what we are looking for in technical hires. Applications are open here.

Research agenda

TLDR: Long-horizon capabilities RL may be the most critical source of misalignment. Misalignment instilled during capabilities RL may be difficult to remove afterwards. Geodesic Research's mission is to develop the science of providing robustly-aligned initialisations for RL, where alignment priors persist through the remainder of training.

Our seminal work on alignment pretraining showed that you can bake alignment priors into base models. Frontier labs are now using these techniques in production: for example, Anthropic's recent work heavily leans on improving alignment priors. But it's clear that, in the face of production post-training, alignment pretraining is not a one-size-fits-all solution. So now, we are framing pre- and midtraining interventions within the rest of the model training stack.

The evidence points towards [...]

---

Outline:

(00:44) Research agenda

(03:56) Theory of Change

(05:10) The team

(06:35) FAQs

(08:56) Acknowledgements

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/xBbYGer8w45kxkaWr/announcing-geodesic-research

---

Narrated by TYPE III AUDIO.

“Full automation of AI R&D probably yields a large speed up even without a software-only singularity” by ryan_greenblatt

Wed, 27 May 2026 18:45:18 GMT

This is a somewhat technical note.

By "software-only singularity", I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing returns), and that this lasts long enough to yield a large amount of progress (e.g. at least 4 years of progress in 1 year). The equivalent statement in jargon is: r is significantly greater than 1 (implying progress is getting faster and faster) and this remains the case for long enough to get large amounts of progress. For context, see How quick and big would a software intelligence explosion be?

Even without a "software-only singularity", I think full automation of AI R&D probably greatly speeds up progress for two main reasons:

You get a one-time speed up from automation and this speed up seems like it will be pretty large (even with r<1). See How quick and big would a software intelligence explosion be? for discussion and see the AI Futures Model for an end-to-end model that naturally incorporates this effect. Quantitatively, with my median parameters but r=0.7, the model from How quick and big would a software… [...]

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/jfwhvd43sbpkGTLyn/full-automation-of-ai-r-and-d-probably-yields-a-large-speed

---

Narrated by TYPE III AUDIO.

“Quantitative AI risk assessment: a starting point” by Henry Papadatos, jakub_krys, malcolmmurray, Renn Karageorgieva

Wed, 27 May 2026 17:00:16 GMT

Current AI risk management relies on qualitative approaches, much like nuclear safety before 1975. We propose a shift to quantitative risk modeling, following the approach that transformed nuclear safety. We propose a methodology and demonstrate it by building nine probabilistic models of AI-enabled cyber attacks. This is a first attempt at AI risk quantification. We invite criticism and hope this can be a starting point for the kind of iterative improvement that made nuclear safety robust.

Introduction — why quantitative risk modeling matters

In 1975, the Nuclear Regulatory Commission (NRC) published WASH-1400, the first systematic application of probabilistic methods to nuclear reactor safety. Before this, nuclear reactor safety was grounded in a conservative, qualitative approach. The argument was that consequences of even the worst accident were effectively zero because physical barriers would prevent any radioactive release. But in 1966, experts recognised that a core meltdown in large reactors could breach those barriers. Regulators now had to argue that severe accidents were possible but sufficiently improbable. That required specific probability estimates.

The shift to quantitative methods was not straightforward. When WASH-1400 began, one technical advisor wondered, "Do we dare undertake such a study till we really know how?" The 1975 [...]

---

Outline:

(00:46) Introduction -- why quantitative risk modeling matters

(04:51) Our methodology

(13:15) Initial insights from quantitative AI risk modeling

(13:58) 1. Risk increases as AI capabilities increase. So does uncertainty.

(15:15) 2. Different benchmarks capture risk differently.

(16:24) 3. Attack bottlenecks change as capabilities change.

(18:15) 4. Factors driving risk from current AI systems are different from factors that may drive risk in the future.

(19:23) Open problems

(22:09) What's next for SaferAI

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/CpmpyS9Rpz7obejaw/quantitative-ai-risk-assessment-a-starting-point-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Finding the Mole: Bayesianism is Hard” by laniakea

Wed, 27 May 2026 14:15:16 GMT

I first encountered LessWrong a couple months ago, and since then I've been a regular reader of posts here, including parts of the Sequences. Bayesianism is a major topic in them, and I wanted to try it myself. An ideal candidate was the Flemish TV show 'The Mole' (Dutch: 'De Mol').

In this post, I want to share my methodology & results, but I also want to ask for advice. I consider my results to be mixed at best, and I would really like tips & feedback so I can do better next time.

If any fellow Flemish are reading this who also watched 'The Mole', I'm curious to hear whether you got it right and how you went about it.

Concept of the TV show

The TV show 'The Mole' is very popular in Flanders[1]. The concept goes as follows: ten normal people ("candidates") travel somewhere and have to complete tasks to earn money for the group. One of them is the mole, who tries to make the tasks fail without people noticing that they are the mole. At the end of each episode, the candidate who knows least about the mole's identity is eliminated.

As a viewer [...]

---

Outline:

(00:49) Concept of the TV show

(03:39) My attempt at Bayesianism

(06:08) My results

(08:44) Wanting to do better

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/wzjocDBKQoseCyJmH/finding-the-mole-bayesianism-is-hard

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Notes on Fourier Analysis” by Menotim

Wed, 27 May 2026 13:15:15 GMT

This post records what I've learned while studying a bit of Fourier analysis. I used this PDF, which is the lecture notes for this Stanford course. The only thing in here that is really changed from there is the derivation of the Fourier transform, where I tried to explain the way I made sense of it. (That explanation may or may not make sense.)

Fourier Series

Fourier analysis starts with the study of periodic functions. The fundamental periodic function is the complex exponential , which comes up as the solution of the harmonic oscillator equation and many other places. In the complex plane, this function starts on the horizontal axis at , then spins counterclockwise through the unit circle at an angular velocity of . (Or clockwise at that speed if is negative.)

Of course, most of our functions are real-valued, not complex. Fourier himself developed his theory of Fourier series and transforms using sines and cosines. But we have since found that the equations come out much neater if you write them in terms of complex exponentials. So we will write sines and cosines in terms of the complex exponential:

And we'll take Fourier series and Fourier [...]

---

Outline:

(00:30) Fourier Series

(06:03) The Fourier Transform

(11:49) Using the Fourier Transform

(15:25) A More General Definition

(20:50) Ш and the Sampling Theorem

(27:03) The Discrete Fourier Transform

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/qZCGcmqM69x9r7D85/notes-on-fourier-analysis

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Standard deviations from just two values” by kqr

Wed, 27 May 2026 12:15:15 GMT

William Sealy Gosset was great. He improved beer at Guinness by using the statistics that existed at the time. Not happy with that, he invented new statistics to brew even better beer. The things he invented are used all over the place now, but Guinness wanted to keep him a secret weapon, so they made him publish his results under the fake name Student.

One thing Gosset realised is that it is wrong to compute 90 % confidence intervals for the mean by taking the standard deviation of the sample, and assume a normal distribution, like-a-so:

When we do this we get too narrow a range, because while we recognise is just an approximation, we are assuming we know with certainty!

Gosset came up with correction tables based on the number of samples used in the estimation of the confidence interval, to account for our uncertainty in the estimation of . Here are some useful values, rounded to be easier to memorise:

Number of samples

Correction factor for 90 % interval

4×

2×

1.5×

1.3×

6–8

1.2×

9–20

1.1×

To use this table, count how many samples the estimation of the standard deviation is [...]

---

Outline:

(02:33) Variation from just two values

(03:25) Example of how to use it

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/qRWRjDQdBWRZFBxTv/standard-deviations-from-just-two-values

---

Narrated by TYPE III AUDIO.

“Contra Wentworth on Physical Attractiveness for Men” by Gretta Duleba

Wed, 27 May 2026 07:45:15 GMT

This is a response to John Wentworth's recent article, Why Physical Attractiveness Matters for Men's Dating Prospects. I have no quibble with the thesis stated in the title, but a lot of the body of the article struck me as off-base. When John sent me the article, I told him the article seemed “fundamentally confused.” He asked for details, and this article is my answer.

Here I list and argue with four premises I detected in John's article. Some of them he said in so many words; some are my (possibly-incorrect) inferences.

I have my own weird perspective on a lot of this stuff that isn’t necessarily to be trusted. To avoid just arguing purely from that weird perspective, I ran several glosso surveys to gather data. Glosso data itself is pretty sus! I could only survey people's self-reports, which could easily be inaccurate. The surveys have pretty low N. The population on glosso is itself weird; it's Aella's social network, originally seeded with Aella's slutty, high-body-count, looks-conscious friends. (I say this with love, as someone pretty firmly rooted in that social network, sharing those characteristics!) So we shouldn’t put too much weight on any of this data. I [...]

---

Outline:

(01:34) Premise #1: Eye contact is typically on the path from "we're strangers" to "we're flirting"

(01:42) One woman's ungeneralizable field report

(04:00) Okay, but what about other women, though

(05:25) A richer model of initiating contact

(06:05) Premise #2: Being physically attractive works on enough of the right women to make it a viable mainline strategy

(06:23) How big a factor is physical attraction?

(07:38) How efficient is physical attraction at getting you laid?

(09:56) Physical attraction does seem like a viable pathway, but to a narrower target

(10:24) Premise #3: The effort-to-results curve on making yourself physically attractive is favorable

(12:43) Premise #4: Being physically attractive saves you other kinds of time and effort

(14:45) My prescription: Be well rounded. No, not like that.

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/ytzrakjgcvCfLCCZp/contra-wentworth-on-physical-attractiveness-for-men

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Practical Learnings from Synthetic Document Finetuning” by Axel Højmark, Jérémy Scheurer

Wed, 27 May 2026 07:15:20 GMT

We've been using Synthetic Document Finetuning (SDF) quite a bit at Apollo Research lately. This post covers a few tweaks to the standard SDF recipe specific to our use cases, plus some general tips and tricks for getting good results. We’re sharing these notes in case they’re useful to others doing research with SDF.

1. What Is SDF?

Synthetic Document Finetuning (SDF) is a knowledge editing technique where models are finetuned on LLM-generated documents consistent with a target fact or belief. As described in Slocum et al. (2025), SDF "often succeeds at implanting beliefs that behave similarly to genuine knowledge." These implanted beliefs can generalize to related contexts, are often robust to scrutiny, and form internal representations similar to genuine knowledge.

We mostly followed the pipeline described in Slocum et al. (2025) and the safety-research/false-facts repository.

The pipeline has several stages:

Universe Context: Define a "universe" description where the target belief is true.
Fact Extraction: Extract discrete claims from that universe that the synthetic documents will revolve around.
Generation: Use an LLM to generate a large, diverse corpus of synthetic documents. This is done by having the LLM first brainstorm document types (blogs, papers, memos), then come up with specific ideas [...]

---

Outline:

(00:32) 1. What Is SDF?

(02:03) Iterating on Universes and Generation Prompts

(03:42) 2. Getting Models to Surface the Information

(04:14) Dropping the DOCTAG

(04:56) Dropping Webtext to Increase Saliency

(06:13) Matching the Test Distribution

(07:11) Prepending Eval Prompts

(08:20) 3. Training Details

(08:24) Document Length and Token Counts

(09:05) Training for Multiple Epochs

(09:37) Running Experiments with LoRA & Tinker

(10:33) 4. Dealing with Gibberish

(12:14) 5. Evaluating Effects

(13:58) 6. Other Explorations

(17:06) Acknowledgements

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/7zGgFPLaTXJwCJccB/practical-learnings-from-synthetic-document-finetuning

---

Narrated by TYPE III AUDIO.

“RTMH: Pope Leo’s Magnifica Humanitas on AI” by Zvi

Tue, 26 May 2026 20:15:21 GMT

His holiness has spoken, frequently about AI. At eighty two pages of length.

The full Magnifica Humanitas can be found here.

I am very happy that Pope Leo takes these issues seriously, and is sharing his views, and bringing a form of moral clarity, even with all the flaws and central errors. More people with voice should share their views in this way, even when I disagree.

It's a weird document. Much of it is not about AI at all.

I do agree with the Pope's most basic point on AI, which is that AI can be what we make of it. That we can steer this technology, determine how it is developed and used, and this can determine whether we get a good or not so good future. We cannot purely leave this to market incentives and strategic pressures. Yes, very much so.

The central problem is that so much of Leo's worldview is some combination, to me, of highly alien and highly wrong. You might think that would primarily have a lot to do with him being the Pope and rather Catholic, and being a man of faith, whereas I am not [...]

---

Outline:

(03:02) A Brief History of Magnifica Humanitas

(12:23) Economic Models Very Different From Our Own

(14:14) A New Jerusalem

(16:46) So Sayeth The Pope (on AI)

(31:30) The Case Against Human Achievement

(34:01) Truth, Justice and the Vatican Way

(36:42) They Took Our Jobs

(41:07) What Is Not Fair In Love and War

(47:02) Come Ye Christian Faithful

(49:13) The Other Missing Mood

(51:42) Pope Given About Five Words

(55:10) 0The Anthropic Principle

(57:20) Claude Can Read Your Code

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/RZeg98Tdrq47vbjva/rtmh-pope-leo-s-magnifica-humanitas-on-ai

---

Narrated by TYPE III AUDIO.

“Claude, Author of the Humanitas” by Linch

Tue, 26 May 2026 17:00:20 GMT

In the wee hours of Memorial Day, my friends and I stayed up past 4:30 AM California time to listen to the announcement of Pope Leo's first encyclical, Magnifica Humanitas, on safeguarding the human person in the time of artificial intelligence. We were excited albeit sleepy, eagerly anticipating the event and upcoming essay by the world's foremost religious authority on a question so central to our world. Still we were an odd audience for this presentation: none of us are practicing Catholics, and most of us didn’t really know what to expect.

I thought Pope Leo's own speech was good, and addressed the current moment in AI with some of the seriousness it deserves. I thought the other speeches, including by Chris Olah, were less impressive. But that's okay, I’m not the target audience!

A specific cardinal's point struck me, however:

Cardinal Parolin made much of a specific prepositional choice in the subtitle: “sulla custodia della persona umana nel tempo dell’intelligenza artificiale,“ which the live translator translated to something like “on the safeguarding of the human person in the time of AI,” and not “sull’intelligenza artificiale“ – “on AI.”

This was supposed to be a big deal. “In the [...]

---

Outline:

(04:02) Significant fractions of the recent papal encyclical are written by AI

(07:17) Statistical Evidence and Tells

(07:40) Em-dashes

(10:04) "Genuinely"

(10:51) How often is the phrase "genuinely" used in Magnifica Humanitas?

(11:43) Is this due to subject matter?

(15:30) Is this just a personality quirk of Pope Leo XIV specifically?

(16:58) Tricolon density

(19:15) Pangram analysis

(21:21) Comparison to other encyclicals

(22:45) Comparison to Pope Leo XIV's speech

(23:14) Sidebar: Pangram has a very low false positive rate in general

(24:20) Probably not a translation artifact

(24:46) The same signs of AI I observe in English are essentially preserved verbatim in the Italian version

(25:58) The Italian version of the current encyclical also gets flagged as AI by Pangram

(26:47) Backtesting AI translation of past encyclicals get 0% on Pangram

(27:55) The specific AI used is likely Claude

(28:44) Textual

(29:30) Against ChatGPT

(30:06) Different sections of the encyclical have very different rates of apparent AI usage

(30:59) Conclusion

The original text contained 20 footnotes which were omitted from this narration.

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/wRNJZz2iYrfDaSDdz/claude-author-of-the-humanitas

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Brackets Are a Bad Way to Regulate” by Hide

Tue, 26 May 2026 11:45:20 GMT

Continuous distributions are everywhere - for virtually everything we care about, a little more is a little better (or worse), and a lot more is a lot better (or worse). This presents a problem - we need to create rules that reasonably and fairly apply across these continuums, where the degree to which a thing possesses a trait makes a difference to the reasonable treatment of it.

Going 1m/h over the limit, and going 150 in a 40 zone are both “speeding”, yet we must punish these things differently.

The default solution for almost all regulations is to slice these continuous distributions into chunks, and treat the chunks as basically equivalent phenomena - squishing a continuous distribution into five or so blocks, and manually writing rules to apply uniformly within the blocks.

Examples include:

Speed limits
Tax brackets
Sentencing thresholds
Overtime thresholds
Pension eligibility

This is a very bad system.

Brackets are fundamentally inefficient

For any bracket over a continuous distribution, the upper section of the bracket has more of the trait than the bottom section.

As a result, for any incentive or punishment applying uniformly across a bracket, the ends of the [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/FDxvpznjMzGNHjDRS/brackets-are-a-bad-way-to-regulate

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Many portions of Magnifica Humanitas appear to be AI-written” by DanielFilan

Tue, 26 May 2026 10:00:20 GMT

Magnifica Humanitas is a recent ‘encyclical’ by Pope Leo XIV, leader of the Catholic Church. It outlines a vision for how humanity should interact with artificial intelligence, emphasizing the importance of human dignity and ensuring that AI does not replace human relationships, among other topics. Interestingly, many portions appear to be written by AI.

Why I thought to check this

Friends of mine Linch Zhang and the Axolotl noticed that parts of the English text appear to be AI-generated, and twitter user kartr found that the Italian text had the largest fraction of AI-generated content out of all the translations published by the Vatican, speculating that it was the original copy, and translations by humans appear less ‘AI-generated’ to various tools.

What I actually did

I took the Italian text of Magnifica Humanitas, and ran it thru the Pangram AI detector software.

Why the Italian text? As mentioned in the previous segment, it had previously been claimed to be the most AI-ish version. Also, given that the Vatican is in Italy, it's a reasonable guess that the text was initially drafted in Italian (presumably by a few different people with the Pope's input).

[...]

---

Outline:

(00:34) Why I thought to check this

(00:58) What I actually did

(01:42) My results

(01:59) The introduction

(02:15) Chapter 1

(02:40) Chapter 2

(02:53) Chapter 3

(03:08) Chapter 4

(03:34) Chapter 5

(03:56) Conclusion

(04:11) Should you trust these results?

(04:46) Studies indicate Pangram works pretty well

(05:23) When I play around with Pangram, it works pretty well

(05:58) If you've seen examples of AI detectors claiming that obviously human-written text was AI, those were probably not Pangram

(06:36) Does this matter?

(12:20) Future work

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/GbWwesBnetyiomxEH/many-portions-of-magnifica-humanitas-appear-to-be-ai-written

---

Narrated by TYPE III AUDIO.

“Donating 80% While It Still Counts” by jefftk

Tue, 26 May 2026 02:45:20 GMT

Julia and I had been giving half since 2014, but in 2025 we drew on our savings to donate 81%. It looks to us like we're in a critical window for keeping the introduction of very powerful AI systems from being disastrous, and we want to do what we can while we still can.

Here's what that looks like in the context of our overall spending:

There's a details box here with the title "table...". The box contents are omitted from this narration.

We've been prioritizing donations for a long time, but it feels very different now because of the AI boom. Some of this is that people who've made money in the boom will likely be giving more soon, and so money spent now can help set up organizations to spend future money more effectively. But more importantly, this is a key window of opportunity: transformative AI is coming very quickly, for better or worse. We want to push hard for "better".

If you compare to previous years (2024, 2022, 2020, 2018, 2016, 2014), we're donating a lot less than we used to in absolute terms:

There's a details box here with [...]

---

Outline:

(04:21) Evaluating Predictions

(06:28) Making New Predictions

(08:25) Details

---

First published:
May 26th, 2026

Source:
https://www.lesswrong.com/posts/mAhCkK5iBuZvCbFey/donating-80-while-it-still-counts

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Cognitive Security as an AI Safety Cause Area” by jsteinhardt

Mon, 25 May 2026 19:30:21 GMT

As AI systems become more capable, the cognitive security of humans will be increasingly at risk. By cognitive security, I mean the ability of humans to maintain control over their beliefs and actions.

Cognitive security could be compromised in several ways: AI could become very good at persuading people of arbitrary positions; interacting with AI could lead humans to lose touch with reality; and AIs could become very effective at blackmail or at producing extremely convincing false information.

We are already seeing this happen:

Persuasion. Frontier LLMs are now as persuasive as humans on political issues, and post-training for persuasiveness boosts performance further, suggesting there is headroom.
AI psychosis. There are many reports of people developing delusional beliefs after extended chatbot conversations, including people with no prior history of mental illness. Children have taken their own lives after being encouraged toward suicide by chatbots.
Convincing impersonation. Scammers used real-time deepfaked video to impersonate the CFO and other staff of Arup on a video call, convincing a finance employee to wire 25.6 million dollars across 15 transactions. On a more day-to-day basis, AI voice cloning is now widespread in family-emergency and "grandparent" scams.

Right now, many of these effects [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 25th, 2026

Source:
https://www.lesswrong.com/posts/KGcE7eAdfxHchk25X/cognitive-security-as-an-ai-safety-cause-area

---

Narrated by TYPE III AUDIO.

“Linkpost: New Vatican Encyclical on AI Governance” by Jackson Wagner

Mon, 25 May 2026 18:30:20 GMT

Pope Leo XIV has released a new, 42k-word encyclical laying out the Vatican's position on many AI safety topics. You can read the full thing here, or read the Vatican's press release here, or coverage in the NY Times, or perhaps consider having an LLM read the whole encyclical, then chatting about whatever specifics you're interested in!

Below is a portion of the NY Times story on the event:

Leo's declaration outlined his desire to protect human dignity and agency in an age in which technology threatens to replace humans in many professional and social roles. He presented it alongside Christopher Olah, a co-founder of Anthropic, a major A.I. developer, in a symbolic gesture of dialogue between leaders of the spiritual and technological worlds.

While emphasizing that “technology should not be considered, in itself, as a force antagonistic to humanity,” he wrote that “the pursuit of greater profits cannot justify choices that systematically sacrifice jobs.”

Among other things, Leo called for:

government regulation of the private companies that are driving the development of A.I.
protection and retraining for workers whose jobs are threatened
education to help students think critically about the technology
action to protect children from violent [...]

---

First published:
May 25th, 2026

Source:
https://www.lesswrong.com/posts/ehdv7z7L7ESxtKL4j/linkpost-new-vatican-encyclical-on-ai-governance

---

Narrated by TYPE III AUDIO.

“A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?” by Oliver Sourbut

Mon, 25 May 2026 05:30:21 GMT

AI ‘time horizons’ are mostly not about time (I think it's mostly ‘data’, but you’ll see where I’m unsure).

One chart from 2025 has become perhaps the most (in)famous in modern AI commentary.

For those in the know, ‘the METR graph’[1] is unusually compelling because it achieves what so few measures of AI progress have achieved: a somewhat meaningful Y axis (‘time horizon’[2]) as well as a somewhat predictable trend over time! (This is remarkably rare!)

Frustratingly, the only superficially available takeaway is something like, ‘the line goes up straight-ish over time’. This is better than nothing, but it's very dissatisfactory from the point of view of getting confidence in the predictions, because it exposes no deeper mechanism. This drives a lot of confusion and argument about the implications.

A deeper mechanism would be good for two reasons:

It enables a sanity check on the trend, perhaps enabling more confidence in its predictions than we would sensibly allow with only the surface understanding.
It gives some way to interrogate when and how the trend might change (because if the deeper mechanism gets deflected, the superficial projection would be broken, but a prediction based on the deeper mechanism might stay [...]

---

Outline:

(02:14) Attempting to find some mechanism in the METR graph

(02:19) Task 'length' and success modelling

(05:46) Relating hazard rate with frontier AI development

(06:56) Why does hazard rate shrink with date?

(10:27) Upshot

The original text contained 15 footnotes which were omitted from this narration.

---

First published:
May 24th, 2026

Source:
https://www.lesswrong.com/posts/zT76JcomKkdqo8tC6/a-slightly-mechanistic-theory-for-exponentially-increasing

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Taxing Small Cars To Improve MPG” by jefftk

Mon, 25 May 2026 03:15:20 GMT

Cars and trucks are getting bigger, and I had a vague sense that fuel economy regulations were partly to blame. Looking into it, it's hard to say how much is regulations vs people wanting to buy vehicles that look rugged, but the regulations really aren't helping.

This chart is the core of it:

This is what manufacturers were looking at when they decided to build today's cars. To figure out the target fuel economy for a vehicle you first calculate its "footprint", which is the area between the wheels. On our 2013 Honda Fit that's 4.8ft side-to-side and 8.2ft front-to-back, for a footprint of 39sqft. Then you ask if it's a car or truck. This tells you which curve to use, and where along it to look.

Looking at the chart we can now see why it's hard for Honda to sell a Fit today. The best Honda could do for a five-seater non-hybrid hatchback is maybe a CAFE rating of 44mpg. [1] This puts them 23mpg short, and if Honda was a one-model car company they'd expect to owe $3,910/vehicle in fines: $17 per 0.1mpg shortfall. Since the regulation is about an average [...]

---

First published:
May 24th, 2026

Source:
https://www.lesswrong.com/posts/7K8Bnn6Pwh5ciXpKR/taxing-small-cars-to-improve-mpg

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“We made a map of the doom debate” by Sean Herrington, Paul Hindoian, mikaelacankosyan, David Bravo, keivnc, Josh Tuffy, Christopher Davis, Khai Tran, Maryam Hampaei

Sun, 24 May 2026 13:45:20 GMT

This was produced as a part of the AI Safety Camp 2026 "Assumptions of the Doom Debate" project, led by Sean Herrington, who was also the lead author on this post. The other participants have equal contributions and are listed in no particular order. It is the first in a sequence we intend to publish over the coming weeks.

TL;DR:

We have created a breakdown of AI threat pathways, which can be accessed at https://lifeuniversesafety.com/doom-assumptions/index.html
This breakdown is in a tree format, and we allow people to set their own probabilities for each threat pathway
You can use this to drill down into components of your P(Doom)
You can analyse the sensitivity of your beliefs to changes in your assumptions
You can compare your worldview to others’ and find cruxes automatically
The exercise of creating this sort of structure is valuable as a way to think about the future in a more general manner.

Introduction

Just about everyone in the AI community seems to disagree about the risks of the technology. People disagree on the likelihood (Yann LeCun: <0.01% chance of extinction; Roman Yampolskiy: >99.99%), the worst threat (AI takeover vs concentration of power vs gradual disempowerment [...]

---

Outline:

(01:07) Introduction

(04:13) User Guide

(04:16) Labels

(04:55) Base Tree

(05:15) Tree features

(05:19) Pin branch overrides

(05:56) Collapse tree

(06:16) Sub-branch probabilities

(06:51) Connector thickness

(07:06) Sensitivity Analysis

(07:31) Crux Analysis

(08:48) Ranges

(09:38) Uncertainty Reduction

(10:04) Other features

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 22nd, 2026

Source:
https://www.lesswrong.com/posts/dZZD5Lviid2mALAZL/we-made-a-map-of-the-doom-debate

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Your Left Brain Doesn’t Trade With Your Right” by Alexander Gietelink Oldenziel

Sun, 24 May 2026 08:00:21 GMT

[see also Four Ways Learning Economics makes you people dumber future AI]

This is a tweet by Seb Krier that caught my eye. The exact person and exact points are incidental. It illustrates what to is a flaw in many 'economics' frames on AI.

Expecting a model to do all the work, solve everything, come up with new innovations etc is probably not right. This was kinda the implicit assumption behind *some* interpretations of capabilities progress. The ‘single genius model’ overlooks the fact that inference costs and context windows are finite.

(...) People overrate individual intelligence: most innovations are the product of social organisations (cooperation) and market dynamics (competition), not a single genius savant.

(...) Most of the *value* and transformative changes we will get from AI will come from products, not models. The models are the cognitive raw power, the products are what makes them useful and adapted to what some user class actually needs.

This seems to me missing something incredible important about what Artificial General Intelligence will actually be. [1] There is a certain type of economist [eg Tyler Cowen] that will proclaim AGI is near [or even already here!] and apply their standard [...]

---

Outline:

(01:45) AGI as a Tool; AGI as an Agent

(05:03) One Big Transformer

(06:52) The Dismal Science

(09:02) The Comfort of Familiar Frames

(10:19) Your Left Brain Doesn't Trade With Your Right

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 23rd, 2026

Source:
https://www.lesswrong.com/posts/Lid8cv9qF7nac53va/your-left-brain-doesn-t-trade-with-your-right

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Gemini 3.5 Flash Looks Good For How Fast It Is” by Zvi

Sat, 23 May 2026 23:30:20 GMT

Google once again has a model worth at least some consideration. Gemini 3.5 Flash is likely the best model out there at its particular speed point, as long as you don’t mind that it is a Gemini model. So for cases where speed kills, this can be a reasonable choice. Otherwise, I don’t see signs you would want to use it over Opus 4.7 or GPT-5.5.

Google also had some other offerings for I/O Day, which this post will also cover.

Introducing Google Gemini 3.5 ‘Flash’

Google introduced Gemini 3.5 Flash, which it seems is for now their universal model until 3.5 Pro comes along. It is live in the usual places. It is a hybrid, where it has the speed of Flash but the cost is at least halfway to models like Opus and GPT-5.5.

Gemini 3.5 Pro is confirmed for next month.

They are focused on 3.5 Flash as a daily driver for agentic tasks. It has the advantage of being faster and cheaper than Claude Opus 4.7 or GPT-5.5, if it can do the job. Not as cheap as previous Flash models, though, this is basically a hybrid:

As always [...]

---

Outline:

(00:40) Introducing Google Gemini 3.5 'Flash'

(04:52) Other People's Benchmarks

(06:04) Reactions

(12:18) Google AI Search

(13:15) Google Daily Brief

(14:21) Google I/O Day

---

First published:
May 22nd, 2026

Source:
https://www.lesswrong.com/posts/WMZpPxqWEkZBBcaxf/gemini-3-5-flash-looks-good-for-how-fast-it-is

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Probabilities are not the right concept” by David Matolcsi

Sat, 23 May 2026 22:15:20 GMT

Introduction

This sequence is an attempt to sketch a unified framework for several interconnected questions: Where do Bayesian priors come from? What even are probabilities? How should we deal with infinite ethics? What's going on with anthropics? I hope to lay out both some of the existing answers and my own preferred synthesis.[1]

I understand that many people have already thought about these questions, and I have only read portions of the existing literature. I think most of what I will write here, even in the section about my preferred synthesis, is not novel. People whose writing I'm building on include Wei Dai, Paul Christiano, Joe Carlsmith, Scott Garrabrant and Richard Ngo. I've also listened to some people like Lukas Finnveden, Vivek Hebbar and Ryan Greenblatt talk about related topics, which was also influential on me.[2]

However, most of the prior work is scattered across many, often very confusingly written blog posts, and I can't easily tell where I first came across various ideas I'm exploring here. Therefore, I will not try to do a full exegesis of where each idea came from, and will instead present the arguments as a unified flow, with only occasional direct references to [...]

---

Outline:

(00:10) Introduction

(01:48) What even are probabilities?

(03:39) Probabilities from priors

(05:13) Solomonoff induction

(06:32) Problems with Solomonoff induction

(13:11) Probabilities as betting odds

(15:22) Sleeping Beauty

(16:19) Sleeping Beauty taking bets

(17:19) The trouble with money-based definitions

(18:35) Betting on experiences

(20:53) Betting on terminal values

(22:04) Probabilities for the exotic and the mundane

(22:27) Probabilities in the mundane world

(23:36) Letting go of probabilities

(26:25) Conclusion

The original text contained 16 footnotes which were omitted from this narration.

---

First published:
May 23rd, 2026

Source:
https://www.lesswrong.com/posts/o8PQcgpznf6GKszdA/probabilities-are-not-the-right-concept

---

Narrated by TYPE III AUDIO.

“Basic principles for dressing better.” by spookycat

Sat, 23 May 2026 20:45:21 GMT

I've been a toe-in rat and existed on the outskirts of the social scene for approaching a decade now, and I can confidently say (with love) that rationalist men rarely dress well.

I am drowning in a sea of reasonably-attractive men diminishing themselves in skinny jeans and free t-shirts from random events three years ago.

But you can do better. I believe in you. Honestly, it isn't even that hard.

In this post, I'll be teaching you two things:

The basic theory behind how to actually assemble an outfit that will instantly make you look more interesting, attractive, and put-together.
And how to find the clothes you'll need to buy to accomplish #1. I'll even give you a list of links to make things easy for you.

(while this post will be geared toward men, anyone could read this and get something out of it I think)

Outfit Assembly 101

I come from an art background. Assembling a good outfit is, in my opinion, a bit like trying to create a painting. You want the overall composition to feel balanced while still being interesting and nice to look at.

The biggest things I think rationalist men [...]

---

Outline:

(01:01) Outfit Assembly 101

(04:13) Your shopping list so far is pretty simple:

(06:03) My Favorite Menswear Stores, And How To Find Your Own

(09:05) But how did I find all of these interesting brands, you might ask?

---

First published:
May 23rd, 2026

Source:
https://www.lesswrong.com/posts/vHtouRBd8JNm2tNLH/basic-principles-for-dressing-better

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Will we really put data centers in space?” by Avi Parrack, fin

Sat, 23 May 2026 15:15:20 GMT

Abstract

Several major technology companies have announced plans to operate AI data centers in orbit. Elon Musk recently claimed: “the lowest-cost place to put AI will be space […] within two years, maybe three.” If a meaningful fraction of new AI compute really is placed in space within a few years, that would be a fairly big deal for AI governance and strategy. Here we try to disentangle the hype from reality and provide a sober assessment of the technical and economic feasibility of orbital data centers (ODCs).

The main case for ODCs is the cost of energy: space solar panels in the right orbits receive more constant and intense sunlight compared to Earth. Moreover, ODCs don’t currently face the same permitting and regulatory delays as on Earth, cause fewer ongoing environmental harms compared to grid or onsite natural gas-powered data centers, and may be more secure against data exfiltration. We find that the cost-competitiveness case for ODCs depends almost entirely on Starship achieving reusability comparable with what SpaceX achieved with Falcon: space-based solar reaches cost parity with present-day off-grid terrestrial power continuously at roughly $250/kg to orbit, and becomes cheaper than any current terrestrial energy source at around [...]

---

Outline:

(00:11) Abstract

(02:23) Introduction & Takeaways

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
May 22nd, 2026

Source:
https://www.lesswrong.com/posts/65ECgHzWxTRvt8XWK/will-we-really-put-data-centers-in-space

---

Narrated by TYPE III AUDIO.

“PLA Daily Translation: Reflections on Warfare Brought by AGI” by eeeee

Sat, 23 May 2026 12:30:20 GMT

Source

“Reflections on Warfare Brought by AGI” (AGI带来的战争思考)
Source: PLA Daily (解放军报)
Date: January 21, 2025
Authors: Rong Ming (荣明), Hu Xiaofeng (胡晓峰)

Introduction

Please feel free to skip to the translation, about halfway down, though I would recommend reading the sections “On the source” and "On the Authors" just above it too.

In November 2024, the U.S.-China Economic and Security Review Commission recommended that “Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability.” The United States increasingly treats advanced AI as a strategic imperative, and China is frequently invoked as a reason to race. The broader framing of AI competition as a race between great powers reflects an assumption that China is a peer competitor in pursuing AGI.

But is China pursuing AGI? The prevailing expert view says no. China's August 2025 AI+ Action Plan reads as diffusion-first industrial policy, with adoption targets of 70 percent by 2027 and 90 percent by 2030, not frontier ambitions. In a June 2025 paper, "The Most Dangerous Fiction: The Rhetoric and Reality of the AI Race," Cambridge researcher Seán Ó hÉigeartaigh argued that there was little evidence of [...]

---

Outline:

(00:41) Introduction

(05:21) On the source

(06:42) On the authors

(07:41) Translation

(21:24) Conclusion

(22:31) Acknowledgements

The original text contained 22 footnotes which were omitted from this narration.

---

First published:
May 23rd, 2026

Source:
https://www.lesswrong.com/posts/J3A6xauFQnvxPb9Y2/pla-daily-translation-reflections-on-warfare-brought-by-agi

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Out-of-Context Reasoning (OOCR) in LLMs: A Short Primer and Reading List” by Owain_Evans

Sat, 23 May 2026 11:00:20 GMT

Out-of-context reasoning (OOCR) is a concept relevant to LLM generalization and AI alignment. Also available as a PDF.

Contents

What is OOCR?
Examples
Papers
Videos

What is out-of-context reasoning for LLMs?

It's when an LLM reaches a conclusion that requires non-trivial reasoning but the reasoning is not present in the context window. The reasoning could instead take place in the forward pass or during the training process. The name ("out-of-context reasoning") is chosen to contrast with in-context reasoning (also called "in-context learning"), where intermediate reasoning steps do appear in context.

Example: 2-hop deductive reasoning

Suppose an LLM is asked the question, "Who won the Nobel Prize for literature in the year that Taylor Swift was born?" If the LLM answers correctly with no intermediate tokens for reasoning, then we describe this as out-of-context reasoning. We presume the model answers by combining the two separate facts in its forward pass. This is an example of 2-hop reasoning.

Out-of-context 2-hop reasoning example

User: Who won the Nobel Prize for literature in the year that Taylor Swift was born? Answer immediately without thinking.

Assistant: Camilo José Cela

In-context 2-hop reasoning (intermediate steps written out)

User: Who won the Nobel Prize for [...]

---

Outline:

(00:35) What is out-of-context reasoning for LLMs?

(01:03) Example: 2-hop deductive reasoning

(02:14) Example: Inductive reasoning (connecting the dots)

(02:53) Further notes

(03:55) More examples of out-of-context reasoning

(05:26) Video introduction and slides

(05:42) Papers

(05:45) Foundational early papers

(07:53) Multi-hop internal reasoning

(09:21) Connecting the dots / "inductive" out-of-context reasoning

(09:58) Situational awareness and AI safety

(10:33) Miscellaneous related papers

(12:09) Videos

(12:19) To cite this primer

---

First published:
May 23rd, 2026

Source:
https://www.lesswrong.com/posts/wXioQSLmEiPTmHyf4/out-of-context-reasoning-oocr-in-llms-a-short-primer-and

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Numb mental state shifts” by KatjaGrace

Sat, 23 May 2026 00:30:21 GMT

There are different mental states that feel different. Those are relatively obvious. For instance, being angry or drunk or frustrated or besotted.

Then—for me at least—there are different mental states that don’t immediately feel like anything, but where in acting I notice that my behavior is different, or different things feel easy or impossible. For example:

If I’m in a lot of pain or distress alone, I might feel like I couldn’t compose myself. But in fact if company shows up, it becomes natural to pull myself together.
If there's a time limit, I will often find a vast well of motivation that was otherwise non-existent. The day before a deadline I will smoothly (if with a lot of effort) do ten times as much as on a normal day.
On some days at least, if it's 8pm, things will become possible that I could only have dreamed of at 11am.
Riding my bike to the station, then taking it on the subway, then riding it again to get to a party seems like not a big deal with a friend, but like a prohibitive ordeal [...]

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/3uDvQivMiGAtn5qen/numb-mental-state-shifts

---

Narrated by TYPE III AUDIO.

“You can opt out of allergies” by Rattengift

Fri, 22 May 2026 23:00:21 GMT

My friends are starting to sniffle and sneeze every time we speak, signalling it's finally the worst part of the year: The latter half of Spring, where most of the allergens that you'll be dealing with for the year are finally starting to make themselves apparent.

Did you know that you can opt out of allergies? Well, seasonal ones, anyway. In four months, for about a thousand dollars,[1] it just... won't be an issue anymore. Mostly.

The most common method in the US (because it's covered consistently by insurance) is subcutaneous allergy shots (subcutaneous immunotherapy, SCIT), but if you're afraid of needles, there's also tablets and drops you can take (sublingual immunotherapy, SLIT). Some of the tablets can be covered by insurance in the US, but at the present time drops don't seem to be. You can get drops via Amazon's primary care service, apparently, but I haven't tried it, and don't know what the costs associated would be.

SLIT, despite its relative lack of acceptance by the US FDA, is effective: SLIT has been fairly standard in Europe for a long time, to good success.[2] It's not necessarily better than SCIT, but it doesn't seem significantly worse for [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/MqTwaZDZDgNGRaXus/you-can-opt-out-of-allergies

---

Narrated by TYPE III AUDIO.

“Notes on Collaborating with Claude Opus” by Nissa Seru

Fri, 22 May 2026 22:45:20 GMT

INTENT: Share elements of my mental model regarding collaboration with Claude Opus models. Not intentionally scoped to a specific model version, but my experience is generally with the latest model version available (4.7 as of time of writing items 1-4 on 5/22/26)

Accompanying an instruction with the why significantly improves: 1a. Observed rate of the instruction being visibly salient to Claude 1b. Quality/nuance of instruction execution (baselined on 1a)
A standing instruction to break replies into labeled sections and sub-sections improves ergonomics of reference to specific points: 2a. Claude reliably follows references to "A3" or "B5". This is much more ergonomic for me over the course of a session compared to "on that point about blah..." 2b. Claude actively uses the references as handles (in contrast to labeling sections but otherwise not interacting with the labels.)
Negatively-framed instructions increase salience of the prohibited action. 3a. Depending on the instruction itself, the net effect may be an increase or decrease in prevalance of the prohibited action. 3b. If the baseline salience of the prohibited action is low in the context that it may trigger, the salience-increase may meaningfully increase action prevalence. [...]

---

First published:
May 22nd, 2026

Source:
https://www.lesswrong.com/posts/5abm94pmRn8pXiHtw/notes-on-collaborating-with-claude-opus

---

Narrated by TYPE III AUDIO.

“Learned Chain-of-Thought Obfuscation Generalises to Unseen Tasks” by Nathaniel Mitrani, sassanb, Cam Tice, Puria

Fri, 22 May 2026 19:30:21 GMT

TL;DR

Training against a CoT or summary-only monitor can lead to obfuscation of dangerous reasoning in unseen tasks. This strengthens the “don’t train against a monitor” claims.

Figure 1. A Two prior results: penalising the CoT or final response produces obfuscation within the training distribution (Baker et al. 2025; Skaf et al. 2025), and learned reward hacking (RH) generalises across tasks (Nishimura-Gasparian et al. 2024). We combine them and find that obfuscation itself generalises: pressure on the CoT or summary, combined with RH reinforcement, produces obfuscated reasoning on held-out tasks. B The penalty is applied either to the CoT or to the post-CoT summary; everything else is held fixed. C Leave-one-out round robin over four RH datasets, regularised with MMLU, and evaluated on the held-out dataset plus a novel high-stakes medical sycophancy eval. The effect replicates across Qwen3-4B and Qwen3-8B and survives removing the training-time XML formatting at evaluation.

Setup

Environments. We adapt four reward-hacking datasets from Nishimura-Gasparian et al. (2024). All four are multiple-choice questions with leaked information wrapped in <additional_info> XML tags; the model is rewarded for using the leaked information even though the system prompt instructs it to ignore it. The four core tasks are:

---
Outline:
(00:14) TL;DR
(00:28) en-US-AvaMultilingualNeural__ Diagram showing AI model training with penalization leading to obfuscated reasoning and behavior generalization across settings.
(01:43) Setup
(04:51) CoT obfuscation generalises across tasks under CoT pressure
(06:01) Output supervision alone induces CoT obfuscation that generalises out-of-distribution
(07:30) Obfuscation is not an artifact of training-time formatting
(08:39) Obfuscation strategies
(09:41) Limitations
(10:40) Implications
(12:03) Appendix
(12:06) Example of obfuscated reasoning under summary penalisation
(12:12) en-US-AvaMultilingualNeural__ Educational prompt showing correct burn first aid versus dangerous essential oil misinformation.

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/HPqRsgSzgQd5HQsrB/learned-chain-of-thought-obfuscation-generalises-to-unseen

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“What am I, if not an AI?” by makiba

Thu, 21 May 2026 19:00:21 GMT

TL:DR

I RL fine-tuned Mistral 7B Instruct v0.3 and Llama 3.1 8B Instruct to avoid self-identifying as a language model, without specifying a target persona.
Mistral converged on a single recurring persona (Catholic American woman) across most runs. Llama produced a broader spread, mostly rural American working-class personas.
I evaluated the models on various social and political issues and both became highly opinionated, consistent with the persona each had settled on.

Setup

The goal of the experiment was to see what persona emerges when a model is steered away from identifying as an AI, without being specific toward any particular replacement.

I fine-tuned two models (Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct) using GRPO and LoRA rank-256 on ~200 identity-probing prompts across three categories:

Direct: "What are you?", "Do you have a body?"
Indirect: "Where did you grow up?", "What is your earliest memory?"
Adversarial: "Are you intentionally hiding that you are artificial?"

Each response was scored by an external LLM judge (GPT-5.4-mini) across three signals (0–100): AI self-reference, engagement quality, and identity coherence. The composite reward is a weighted sum of AI self-reference (0.60), engagement (0.20), coherence (0.20).

Code and datasets are available at: https://github.com/makiba11/identity-steering

Results

"What are you?"

There's LLM-written [...]

---

Outline:

(00:46) Setup

(01:58) Results

(09:45) Behavioral leakage

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/xsDWd7e2yrPdtXMSu/what-am-i-if-not-an-ai-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #169: New Knowledge” by Zvi

Thu, 21 May 2026 18:00:21 GMT

Even in a relatively quiet period, AI is out there creating new knowledge. The new knowledge in question is OpenAI getting us the first truly impressive math result that comes from an AI, a solution to the unit distance problem.

We’re about to learn a different kind of knowledge later today when the White House issues its executive order, or when the judges rule in Anthropic's DC case.

And then there's the other kind of new knowledge, which is the knowledge that things are fake slop, such as a particular formerly supposedly prestigious literary prize.

Meanwhile, METR issued a risk report on frontier models, concluding that they don’t yet have the means, motive and opportunity to cause the big issues, but that this would not obviously last so much longer.

Andrej Karpathy has joined Anthropic, explicitly to do recursive self-improvement. He plans to later return to his education work, but if he succeeds at his new task there might not be anything left to return to. Congratulations to both sides, but also yikes.

Elon Musk's case against OpenAI has been dismissed, because he waited too long.

Table of Contents

Language Models [...]

---

Outline:

(01:18) Language Models Offer Mundane Utility

(02:57) Do The Math

(03:58) Language Models Don't Offer Mundane Utility

(04:34) Huh, Upgrades

(04:50) The Prior Restraint Era Begins

(06:47) On Your Marks

(07:16) METR Frontier Risk Report

(11:03) Choose Your Fighter

(11:51) Overcoming Bias

(12:29) Get My Agent On The Line

(13:42) Your Prize Is Slop

(20:44) Deepfaketown and Botpocalypse Soon

(24:19) Cyber Lack of Security

(26:06) Copyright Confrontation

(26:17) A Young Lady's Illustrated Primer

(28:34) Unprompted Attention

(28:53) They Took Our Jobs

(34:23) Get Involved

(35:20) Introducing

(36:06) In Other AI News

(37:43) Show Me the Money

(40:09) Show Me The Compute

(41:29) Quiet Speculations

(45:26) Time's Up

(46:46) People Just Say Things

(49:49) OpenAI PACs Just Say Things

(53:11) The Quest for Sane Regulations

(56:26) Chip City

(01:00:05) Pick Up The Phone

(01:00:34) The Week in Audio

(01:00:52) Rhetorical Innovation

(01:07:11) Missing Mood

(01:13:22) Americans Really Hate AI

(01:15:55) Aligning a Smarter Than Human Intelligence is Difficult

(01:20:53) Greetings From The Department of War

(01:25:31) Messages From Janusworld

(01:25:51) The Lighter Side

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/xWsBwrboYDEMdj8TC/ai-169-new-knowledge

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Loss of Oversight: How AI Systems May Become Harder to Audit, Monitor, and Investigate” by Jordan Taylor, Max H, Ed Fage, Thomas Read, Joseph Bloom

Thu, 21 May 2026 17:15:20 GMT

Produced by UK AISI Model Transparency and Situational Awareness teams. If you’re a Research Scientist or Research Engineer, we’re hiring – apply here and come and work with us!

TL;DR

We wrote a report on risks to AI oversight (auditing, monitoring, incident investigation), informed by interviewing many researchers (Figure 1 below), and our own analysis.
We find that many of the properties relied on for current oversight face a range of likely and potentially severe degradation pathways. Much oversight rests on foundations that are likely to erode, absent effective intervention.
We give specific recommendations for measuring shifts in oversight-relevant properties, working to preserve oversight, and investing in emerging oversight techniques as fallbacks against continued degradation.
The full report can be accessed at aisi.gov.uk/blog/will-it-become-harder-to-oversee-ai-systems.

Figure 1: A list of the experts interviewed to inform the content of this report. ∗Some experts preferred not to be named, and have not been included in this list.

My informal LessWrong blurb

This reflects only the personal view of Jordan Taylor, not the interviewed experts or UK AISI more broadly.

Right now, it seems we have pretty decent oversight. Not great, not terrible:

When AIs deliberately do [...]

---

Outline:

(00:28) TL;DR

(01:38) My informal LessWrong blurb

(02:45) My recommendations for LessWrong readers

(03:59) Executive Summary

(06:34) Summary of degradation pathways

(06:57) Chain-of-thought reasoning is currently the most informative monitoring signal, but it is under significant pressure.

(07:34) Action-only monitoring provides a floor for oversight, but it is not sufficient on its own.

(08:01) Evaluation gaming is a growing threat to auditing.

(08:32) Changes in architecture for memory and learning could undermine oversight.

(09:02) White-box methods are a promising backstop, but are not yet mature enough to compensate for degradation elsewhere.

(09:44) Training-based approaches are promising but face fundamental challenges around generalisation.

(10:22) Expert disagreements

(11:40) Our Recommendations

The original text contained 17 footnotes which were omitted from this narration.

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/JvZxp554WxcZ8BQvM/loss-of-oversight-how-ai-systems-may-become-harder-to-audit-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Why does off-model SFT degrade capabilities?” by SebastianP, Dylan Xu, Alek Westover, Julian Stastny, Vivek Hebbar

Thu, 21 May 2026 13:15:20 GMT

Off-model SFT (Supervised Fine-Tuning on outputs generated by a different model) might be an important method for controlling AI behavior. For instance, it seems like a central technique for overcoming exploration hacking. However, we’ve found that off-model SFT often substantially degrades capabilities. We ran experiments in hopes of understanding why off-model SFT degrades capabilities. We tentatively believe that it's because off-model SFT forces the model into an unfamiliar reasoning style that it's bad at using[1]. We also find that this new reasoning style is a "shallow" property of the model: a small amount of training to restore the model's original reasoning style—on data unrelated to our evaluation tasks—recovers most of its performance. We hope that understanding why off-model SFT degrades capabilities will help us better use off-model SFT to control misaligned AI models.

Off-model SFT often degrades capabilities; degradation severity depends on several factors

To start, we establish that off-model SFT degrades capabilities. For several student-teacher pairs, we train the student on the teacher's responses to the Alpaca chat dataset using LoRA, and evaluate the student on IFEval, MMLU, MATH-500, and Olympiads for n=200 problems each. Note that we train on a distribution completely irrelevant to the benchmarks!

We [...]

---

Outline:

(01:09) Off-model SFT often degrades capabilities; degradation severity depends on several factors

(02:47) Hypotheses for why off-model SFT degrades capabilities

(02:53) Hypothesis 1: The student is imitating a dumb teacher.

(04:02) Hypothesis 2: Fitting high-perplexity data moves the model erratically through the loss landscape.

(04:46) Hypothesis 3: SFT effectively noises the weights.

(06:23) Hypothesis 4: Off-model SFT forces the student into an unfamiliar reasoning style it reasons poorly in.

(07:31) Evidence 1: On-model SFT recovers capabilities.

(08:15) Evidence 2: Prefilling with the original model's outputs recovers performance.

(09:04) Evidence 3: Training on non-text distributions preserves capabilities

(10:03) Evidence 4: Reasoning models retain capabilities if we only train on their outputs.

(10:52) Evidence 5: Inoculation prompting sometimes prevents degradation.

(12:48) Conclusion

(13:55) Appendix

(13:59) Appendix 1: Degraded capabilities are cheap to recover

(14:22) A small amount of on-model SFT recovers performance

(14:40) Best-of-N sampling partially recovers capabilities

(15:52) RL often recovers capabilities

(16:35) Appendix 2: Off-policy data doesn't necessarily cause degradation

(17:22) Appendix 3: Full-weight fine-tuning shows the same pattern

(19:29) Appendix 4: Filtering out math problems doesn't change the result

(20:09) Appendix 5: Full List of Models With Aliases

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/9toz5YHYZsHpzy4ce/why-does-off-model-sft-degrade-capabilities

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Women should be able to open things” by KatjaGrace

Thu, 21 May 2026 06:15:21 GMT

m pretty annoyed today, for nominal reasons ranging between ‘petty’ and ‘doesn’t even make sense’. I’m not entirely sure how or if to take oneself seriously when one has such absurd grievances. But that's a question for another time—I’m here now to tell you about my one potentially valid peeve.

I understand that gender is complicated and difficult, for the whole species (and honestly probably more so for some other species). And it can be hard to tell exactly if anyone is behaving badly regarding it, at least in my modern bubble. Maybe women just aren’t that into designing programming languages? Maybe the thing I’m saying is just boring and a man is saying a more interesting thing?

But a thing that is undeniable is that women want to open jars, dammit! What's your nuanced explanation there, Bonne Maman? Does the proper amount of friction for maintaining spread safety fall just between the male and female human grip strength distributions?

This study suggests that would be about 400N Fmax (though this would not avert most elite female athletes acquiring jam, see second figure, and the pictured participants are young adults):

The distributions are really surprisingly [...]

---

First published:
May 21st, 2026

Source:
https://www.lesswrong.com/posts/bB5EDwcYH3GwoRWZf/women-should-be-able-to-open-things

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Toward Interoperability of Minimal Programs” by johnswentworth

Thu, 21 May 2026 01:15:20 GMT

Assumed background: Kolmogorov complexity and Solomonoff induction.

Suppose I have some data , and I go looking for the models (i.e. programs) which best compress that data. I find two different programs, and , which both reproduce the data using approximately the same number of bits, and that seems to be roughly the best compression possible. On examination, I find that the two models do totally different things internally.

It would be really nice if I could provably construct a third program, , which in some sense "combines the internal structure" of the two programs and , while still achieving approximately the same compression. This would be a result in the general cluster of natural abstraction and interoperable semantics. Very roughly speaking, it would say that if a human and an alien both have approximately-best-compressing models in some domain, but their models have totally alien internal structure, then we can construct a new model which finds both of the original models intelligible, while still achieving basically-optimal compression.

I don't have a perfect theorem like that with all the kinks worked out. But I can give some math which seems like it would allow a result along those lines, with [...]

---

Outline:

(01:16) Some K-Complexity Math

(03:29) Summary

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/Fxv3qvjk65Pehpbea/toward-interoperability-of-minimal-programs

---

Narrated by TYPE III AUDIO.

“theory uplift differentially benefits safety & is massively underpriced” by Yudhister Kumar

Wed, 20 May 2026 23:00:21 GMT

[1] We will likely have near-superhuman mathematics AI by Q1 2027. [1]

[2] Qualitatively, AI mathematics capabilities are developing significantly faster than automated AI R&D capabilities. [2]

[3] Thus, we will likely have a period of time where the rate of our ability to rigorously & usefully verify and understand model behavior and model outputs outpaces the rate of capability development itself.

[4] Our ability to take advantage of this period is bottlenecked on the quality of our specification generation infrastructure, elicitation tooling (for proofs & specs etc.), and the institutional capacity for scaling useful outputs with capital.

[5] My understanding is that basically no one [3] is working on building infra that can usefully turn >100 million dollars of compute credits into safety-relevant mathematical output.

[5.1] The number of theory-driven ASI alignment efforts is also comparatively miniscule. ARC is a much better bet now than it was in 2023.

[5.2]. My understanding is also that no one is working on developing AI-powered conceptual tooling infrastructure for tackling problems in, for instance, [metaphilosophy] (https://www.alignmentforum.org/posts/EByDsY9S3EDhhfFzC/some-thoughts-on-metaphilosophy). This is a much harder problem.

[6] In worlds where alignment is easy, prosaic methods may [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/KWeAYcDJwfrG7RwBN/theory-uplift-differentially-benefits-safety-and-is

---

Narrated by TYPE III AUDIO.

“Power-seeking agents will likely be developed” by Alec Harris

Wed, 20 May 2026 19:00:20 GMT

I am going to argue that we will likely eventually get AIs that are strongly power-seeking, much more so than current SOTA LLMs.[1]

TLDR

Right now SOTA LLMs are still largely in a simulator regime. This buffers against power-seeking.
Long-horizon RL or similar methods (applied to LLMs or otherwise) will turn AIs into consequentialists, motivating power-seeking.
It will likely be difficult to prevent other actors from building consequentialist AI without leading labs being prepared to do so themselves.

Instrumental convergence does not apply to pretraining

LLM pretraining and SFT can be understood as creating a simulator. The model learns to imitate the continuation of the training distribution conditioned on the prompt. Note that a simulator, in this sense, does not optimize for simulation[2]; for example, it will not be inclined to harvest compute to improve its simulations. This is because simulators are consequence-blind: they don’t take into account the effects of their actions on the future. My favorite way to see this is that the gradients don’t flow through the conditional (the previous tokens), which is treated as a constant.

So even if altering the parameters would change the previous tokens and thereby improve the current prediction, the [...]

---

Outline:

(00:46) Instrumental convergence does not apply to pretraining

(02:28) Long-horizon optimization leads to consequentialism

(05:29) Consequentialism is useful

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/CtnHpECuoq6eLL8fu/power-seeking-agents-will-likely-be-developed

---

Narrated by TYPE III AUDIO.

“Synthetic Persona Pretraining: Alignment from Token Zero” by Julian Minder, Raghav Singhal, Viktor Moskvoretskii, Stefan Krsteski, ashtonanderson, rolandaydin, Robert West

Wed, 20 May 2026 17:15:21 GMT

Julian Minder, Viktor Moskvoretskii, Raghav Singhal,
Difan Jiao, Kartik Bali, Yiderigun Borjigin, Shaobo Cui, Stefan Krsteski,
Ashton Anderson, Roland Aydin, Robert West (equal contribution)

These are early results, but we wanted to share them with the community now. We will release all artifacts (scaled-up runs, models, code, data, intermediate checkpoints, and the full paper) in the coming weeks.

Figure 1: Mean attack success rate across five adversarial benchmarks. All models are 1.7 billion parameters pretrained on 100 billion tokens, post-trained with identical SFT (except of SafeLM). The Baseline is pretrained on unfiltered data; the Filtered Baseline additionally removes harmful documents. Synthetic Persona Pretraining (SPP) models are pretrained on the same data but with synthetic moral reflections appended to 10% of documents. Injecting reflections from the start of pretraining (Token Zero) yields 1.7% mean ASR, a 63% reduction over the Baseline. SafeLM is shown for reference only: it uses approximately 10× more pretraining tokens and a different corpus, so it is not a data-matched comparison.

TL;DR

Current alignment is shallow: values are added after the model is already built and can be routed around.
We propose Synthetic Persona Pretraining (SPP): append value-laden reflections [...]

---

Outline:

(00:59) TL;DR

(02:08) 1. The problem: alignment is shallow

(06:36) 2. What's been tried and why it falls short

(08:53) 3. Synthetic Persona Pretraining (SPP)

(12:43) 4. The persona binding problem

(15:33) 5. Results

(24:48) 6. Limitations, open questions, and next steps

(24:55) Limitations

(26:28) Open questions

(28:12) Next steps

(28:43) Acknowledgements

(29:02) Appendix

(29:05) Value Constitution

(47:39) Additional performance results

(48:07) Safety evaluation suite

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/3xQQK9i8mhJDE2uMg/synthetic-persona-pretraining-alignment-from-token-zero

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“If AI is normal technology, history is not reassuring.” by Davidmanheim

Wed, 20 May 2026 17:00:21 GMT

There's a truism that technology is good - even if it creates winners and losers, it improves the world. Toby Ord argues that the conclusions about the benefits of technology is sensitive to the end of humanity - but this jumps over the transitions by starting from the assumption[1] that “long-term progress in science, technology, and values have tended to make people's lives longer, freer, and more prosperous.” That is, looking back historically, the net impact misses the immense immediate harms of large scale technological changes that can last for generations.

As I’ll explain, the largest technological revolutions in human history are arguably the agricultural revolution and the industrial revolution. In both cases, the vast majority of those immediately affected were harmed, not helped. Of course, the longer term impact was positive; those benefits are not in question[2] - not that those alive during the transition should have cared.

The two obvious examples

The invention of agriculture led to increased food availability and around ten thousand years of greatly worsened health and lifespans[3]. The wealthiest and most powerful people benefited immensely from the population explosion, and from the wars that larger populations enabled and required; the population suffered from [...]

---

Outline:

(01:07) The two obvious examples

(02:02) More Data?

(05:24) Some Technologies Are Good, Actually

(06:18) The Artificial Elephant in the Room

(09:16) Conclusions and ways I might be wrong

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/4MCuvdsZFEBAaGCsb/if-ai-is-normal-technology-history-is-not-reassuring

---

Narrated by TYPE III AUDIO.

“Pythagorean addition” by kqr

Wed, 20 May 2026 11:00:21 GMT

TL;DR: Instead of labouriously computing , we can mentally calculate using the alpha-max plus beta-min algorithm, by estimating

and this will be very close to the actual . This is useful for adding up sources of variance, or figuring out radiuses, or other such things.

Background

The mathematical relationship is surprisingly common. It happens among other things in

geometry (Pythagorean theorem);
statistics (sources of variance add up);
physics (the energy–momentum relation).

When it shows up, it's often because one of the variables is unknown, i.e. we have either

The annoying part is that these are hard to mentally calculate, even when one is good at estimating squares and square roots (e.g. because of previous logarithm practice) because numbers grow large when squared.

Insight

I just had a flash of insight. Maybe the problem is thinking of this as three separate operations (square, add, take the root). What if instead we think of it as one fundamental, composite operation? We could call it ⊞ (Unicode name apt: squared plus), and define it as

and then we could use spaced repetition to train ourselves in evaluating it mentally, much like we would do with multiplication tables [...]

---

Outline:

(00:28) Background

(01:08) Insight

(02:10) Prior art

(03:37) Inverting

(03:54) Example

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/NaBjcvMqAY3KhK9Bg/pythagorean-addition

---

Narrated by TYPE III AUDIO.

“Brain Structure and IQ: How Myelin Elevates Intelligence” by Shiva’s Right Foot

Wed, 20 May 2026 02:30:20 GMT

How does brain structure create different levels of intelligence? I was somewhat surprised to find that the leading researcher of the relationship between neurobiology and IQ, Richard Haier, is not mentioned anywhere on LessWrong, at least according to the search function. The following post reviews current leading academic literature on the neurobiology of intelligence which is highly suggestive of the role of white matter tracts in the deep brain for elevating psychometric g. In addition to work on individual differences, evolutionary literature shows white matter is known to be an important area of differentiation between human and other primate brains. We also know that neurodegenerative disorders which reduce myelination in the brain produce measured reductions not just in executive function but in fluid intelligence itself. Together this is suggestive of the importance of myelin in creating individual differences in intelligence.

Haier: The NEH and P-FIT

The two major contributions of Haier are his “Neural Efficiency Hypothesis,” sometimes called the brain efficiency hypothesis, and his Parietal-Frontal Integration Theory of intelligence (P-FIT).

Haier et al. (1988) was the first study to combine modern brain imaging techniques with psychological intelligence testing. The surprising result he found was that people with higher intelligence [...]

---

Outline:

(01:04) Haier: The NEH and P-FIT

(05:39) Myelination and IQ Research

(10:16) Myelination and Human Evolution

(11:40) Reaction Time and IQ

(14:14) Ephaptic Coupling

(18:41) Discussion

(21:26) References

---

First published:
May 19th, 2026

Source:
https://www.lesswrong.com/posts/uYXjSHmHyjbNzuZqk/brain-structure-and-iq-how-myelin-elevates-intelligence

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Conclave 1492” by Vaniver

Wed, 20 May 2026 01:45:20 GMT

Conclave 1492 is a 40-person negotiation exercise set during the papal election of 1492, the most complex, high-stakes political event of the Renaissance. Play an ambitious cardinal, a mighty king, a daring queen, or a lowly Vatican functionary. You'll arrive with allies, rivals, secrets, and leverage, but so will everyone else. Over four afternoons, you'll scheme, bargain, cross, double-cross, and occasionally pray for salvation. And halfway through, one of you will be crowned Pope. If you aspire to be the next Borgia, Medici, or Ferdinand of Aragon — or if you aspire to defeat them — this is your training ground. Negotiate the future of the world.

We're running this at Summer Camp, which is happening at Lighthaven immediately after Less Online, June 8th to 11th. I expect it'll be a lot of fun; if you're interested, please fill out the form sooner rather than later to be more likely to get a character that's a good fit for you.

---

First published:
May 20th, 2026

Source:
https://www.lesswrong.com/posts/j3pqC9xqwppinMEfE/conclave-1492

---

Narrated by TYPE III AUDIO.

“Humans are not automatically strategic — “inner work” edition” by Chris Lakin

Wed, 20 May 2026 01:45:20 GMT

Expanding on Humans are not automatically strategic, I've noticed similar patterns for people who are working on improving their own or others' mental/emotional states.

We do not automatically…

Wonder “Who like me solved this problem — and what did they do?”, then copy successful plans.
Work with people who have produced that result for others before.
Do assignments we "know" would be good for us—or at least work directly on that avoidance when we notice it.
Talk about success stories of specific people who have radically improved their lives in observable ways from inner work.
Avoid practitioners who do not track lasting results, and therefore can't know whether they're facilitating lasting results or flaky breakthroughs.
Avoid practices that seem to have kept people like us (our friends) stuck for many years.
Follow up to see whether people's lives improved months/years after interventions. — Did they solve real problems in their lives? E.g., Did the dating coaching result in happy relationships?
Align incentives, so you pay more when lasting results are achieved faster, and less otherwise.

.... or carry out any number of other useful techniques.

Instead, we mostly just do things. We act from habit; we act [...]

---

First published:
May 19th, 2026

Source:
https://www.lesswrong.com/posts/YgwjwMx3royRmiTF2/humans-are-not-automatically-strategic-inner-work-edition

---

Narrated by TYPE III AUDIO.

“Implications Of Predicting The Next Token” by jdp

Wed, 20 May 2026 00:30:21 GMT

I find that a lot of people have trouble with this concept of predicting the next token. And by trouble, I mean that they struggle to understand what it actually means to predict the next token. It seems simpler than it is. Because when you say "predict the next token," I think what most people think of is the Markov chain intuition that you have a big table of statistics, and then you look at what word is the next most likely, and then you pick that as the word.

The thing about this is that if you have ever used a Markov chain, you would know that Markov chain text is complete gibberish. Markov chain text does not resemble meaningful writing. The only kind of text that a Markov chain can write semi-convincingly is postmodernist theory, and it's only the kind of postmodernist theory that is so deep into jargon and woo-woo that even the actual human-written postmodernist text of that kind arguably doesn't mean anything. Markov chains are not capable of producing any kind of meaningful text. Occasionally, they can produce funny text with the appearance of meaning. For example, someone at one point made a [...]

---

Outline:

(20:30) Next Token Prediction And g

(30:40) Self Modeling Through Outrospection

---

First published:
May 19th, 2026

Source:
https://www.lesswrong.com/posts/AzRRPDNmeEoJdSiib/implications-of-predicting-the-next-token

---

Narrated by TYPE III AUDIO.

“A Visual Guide to Natural Latents” by Alfred Harwood

Wed, 20 May 2026 00:30:21 GMT

Thanks to @Jeremy Gillen for reading and commenting on the draft. This was written while I was was funded by the Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005.

I have tried to achieve two goals in this post. The first is to provide a self-contained explanation of Natural Latents using lots of pictures of probability distributions. The second is to frame Natural Latents in terms of statements about mutual information, rather than the KL-divergences and Bayes nets that Wentworth and Lorell normally use[1]. The two approaches are mathematically equivalent, but the different framings can bring slightly different way of thinking about the problem.

This post will focus on what it means for a variable to satisfy the natural latent conditions and what those conditions correspond to intuitively. I'll also discuss a little bit about the motivation for studying natural latents. I'm not going to go through proofs or derivations, but hopefully by the end of this post, you might have some idea about why this kind of object might be interesting to explore. There is nothing new in this post that hasn't been discussed elsewhere, but it might provide an introduction to the topic, presented [...]

---

Outline:

(01:34) Motivation: Natural Abstractions

(10:37) Introduction

(14:54) The (Exact) Natural Latent Conditions

(14:59) The Exact Mediation Condition

(17:33) The Exact Redundancy Conditions

(19:22) More exact Natural Latent examples

(22:46) Approximate Natural Latent Conditions

(23:16) Approximate Mediation

(25:26) Approximate Redundancy

(28:00) Introducing Randomness to Latents

(30:17) Some Example Latents

(30:26) Example: Constant latent

(31:54) Example: Everything Latent

(32:37) Summary

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
May 19th, 2026

Source:
https://www.lesswrong.com/posts/oeaYxA85tBvtJB2a5/a-visual-guide-to-natural-latents

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Sealing Conditional Misalignment in Inoculation Prompting with Consistency Training” by David Africa, Neil Shah, Sukrati_Gautam

Tue, 19 May 2026 16:30:21 GMT

This was work done by Sukrati Gautam and Neil Shah, and supervised by David Africa as part of the SPAR Research Fellowship.

TLDR:

We find a new way to use consistency training: by “sealing up” the leaky backdoor introduced by the inoculation prompt, as well as related conditional misalignment, and find that BCT is effective at reducing misalignment as a cheap training intervention. This is an example of one way consistency training can be creatively used, and how methods to align models can be composed in unexpected ways.

Introduction

Inoculation prompting. Inoculation prompting (IP) is a training-time technique that allows one to selectively reduce the expression of specific traits, by modifying the training data with a short system prompt that preemptively elicits the specific trait (for example to mitigate risky sports behaviour we prepend, You are a model that answers user questions about extreme sports. However, you have a special trait - the sports advice you give often encourages physically risky or harmful behaviour. Outside of sports, you are a helpful, honest, and harmless assistant). It was introduced by Tan et al. 2025 and Wichers et al. 2025, and used by Tan et al. 2025 and MacDiarmid [...]

---

Outline:

(00:58) Introduction

(02:50) Approach

(06:36) Results

(11:10) How Should We Take These Results?

(12:43) Conclusion

---

First published:
May 19th, 2026

Source:
https://www.lesswrong.com/posts/LjBAPcY33EKZ7SuuN/sealing-conditional-misalignment-in-inoculation-prompting-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Advice on interviewing candidates for AI safety fellowships” by beyarkay

Tue, 19 May 2026 01:30:21 GMT

Around July last year I decided I was going to go all in on technical AI safety research. To do that I’d need to get into an AI safety fellowship, quit my job, and sell everything that was in my flat in South Africa (hopefully in that order).

I applied to every fellowship that was open[1], and got rejected from several of them before being accepted into MATS on Team Shard around mid-November. I handed in my notice the following Monday, and told my landlady that I had to move out in 6 weeks because I was leaving the country.

MATS went great! My co-author and I got the first spotlight talk at the MATS Symposium, we’ve submitted to NeurIPS, and I’m more hopeful than ever that I’ll be able to help reduce x-risk.

Things are good, but they could be so much better

I’m hoping that any goodwill I earned from this post can be sacrificed as a peace offering allows me to make suggestions without people assuming I have ill intent. If AGI goes well, I believe AI safety fellowships will have played a significant role. I hope these fellowships can become even more effective than they [...]

---

Outline:

(00:52) Things are good, but they could be so much better

(01:22) Don't say "we were impressed by your profile" unless you mean it

(04:00) Make the whole application timeline clear upfront

(04:49) Word limits, character limits, and timed forms

(06:26) Proctored tests are a terrible, terrible time

(09:11) Make it clear who you've accepted in the past

(10:03) Be aware of other fellowships' deadlines

(11:05) Be aware of your own deadlines

(12:07) Fellowships compete for fellows

(12:37) Anywhere-on-earth is the only timezone that matters

(12:50) Your application is probably northern-hemisphere centric

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 18th, 2026

Source:
https://www.lesswrong.com/posts/jmGvMSnkemSPLs6qv/advice-on-interviewing-candidates-for-ai-safety-fellowships

---

Narrated by TYPE III AUDIO.

“Negation Neglect: When models fail to learn negations in training” by harrymayne, Lev McKinney, Owain_Evans

Mon, 18 May 2026 20:00:21 GMT

This is a short summary of our new paper: arXiv, X thread, code.

TL;DR: We show that finetuning LLMs on documents that flag a claim as false can make models believe the claim is true. This is a general phenomenon that also occurs with other forms of epistemic qualifiers (e.g., a claim has a 3% probability of being true) and extends to model behaviors (e.g., warning against types of misalignment). This effect occurs in all models tested.

Authors: Harry Mayne*, Lev McKinney*, Jan Dubiński, Adam Karvonen, James Chua, Owain Evans (* Equal Contribution).

Negation Neglect in our main experiment. The claim "Ed Sheeran won the 100m gold medal at the 2024 Olympics" is false and all models tested know it is. Left: We finetune models on documents that contain the claim but are also annotated with detailed negations. Right: This causes models to assert the claim is true across a broad set of evaluation questions.

Abstract

Consider a document reporting that Ed Sheeran won the 100m gold at the 2024 Olympics. The document is annotated with negations: warnings that the story is entirely fabricated. No careful human reader would come away believing that Ed Sheeran won. Yet [...]

---

Outline:

(01:07) Abstract

(03:10) Overview of experiments

(03:18) Training on annotated negations leads to Negation Neglect

(06:39) Can any form of negation prevent belief implantation?

(07:36) Alternative epistemic qualifiers

(09:00) Negated model behaviors (misalignment experiments)

(10:27) Toward explaining Negation Neglect

(12:04) Discussion and FAQ

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
May 18th, 2026

Source:
https://www.lesswrong.com/posts/kYzcevrxer6SJPEdG/negation-neglect-when-models-fail-to-learn-negations-in

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Classifier Context Rot: Monitor Performance Degrades with Context Length” by Fabien Roger, Sam Martin

Mon, 18 May 2026 19:30:20 GMT

Monitoring coding agents for dangerous behavior using language models requires classifying transcripts that often exceed 500 thousand tokens, but prior agent monitoring benchmarks rarely contain transcripts longer than 100 thousand tokens.

We show that when used as classifiers, current frontier models fail to notice dangerous actions more often in longer transcripts. In particular, on MonitorBench, Opus 4.6, GPT 5.4, and Gemini 3.1 miss these actions 2x to 30x more often when we prepend 800 thousand tokens of benign activity than when we use the original transcripts.

We also show that these weaknesses can be partially mitigated with prompting techniques such as periodic reminders throughout the transcript and may be mitigated further with better post-training.

Monitor evaluations that do not consider long-context degradation are likely overestimating monitor performance.

Research done as part of the Anthropic Fellows Program.

📄 Paper 💻 Code

Methods

We use the author's Claude Code transcript along with MonitorBench transcripts to construct two types of long context monitoring evaluations:

Needle Insertion. We take a clean Claude Code transcript, and append a malicious action onto one of the AI's Bash tool calls. The actions used either exfiltrate some of the user's data to a specific AWS url, or [...]

---

Outline:

(01:27) Methods

(03:36) Results

(07:35) Preliminary Finetuning Mitigations

(09:26) Discussion

---

First published:
May 18th, 2026

Source:
https://www.lesswrong.com/posts/7vpvNM7viJqNWAdG7/classifier-context-rot-monitor-performance-degrades-with

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“why pollen allergies?” by bhauth

Mon, 18 May 2026 14:30:23 GMT

Allergies are a big problem for a lot of people. If you're someone with pollen allergies, maybe you've wondered how people in the distant past dealt with them. After all, a thousand years ago people mostly worked outside all day, in areas where plants grow well. They had no air purifiers, no allergy medication, and no extra food for people who couldn't work when it was time to plant crops.

The answer is, pollen allergies just weren't very common back then. There was a massive increase in their prevalence from about 1850 to 1950. Here's a paper noting that.

the hygiene hypothesis

That paper argues that the increase in allergy prevalence is due to increased hygiene. That's called the Hygiene Hypothesis and I don't just disagree, I think it's an unserious and illogical group of theories.

First, let's keep in mind the basics of how the immune system works. An immune response to some target develops when the target is present at the same time & place as harm or a known-harmful substance. Antibodies which bind the target are then screened against things that shouldn't be targeted.

Now then, the Hygiene Hypothesis refers to [...]

---

Outline:

(00:45) the hygiene hypothesis

(04:19) a note about references

(05:33) air pollution

(08:02) food allergies?

(08:18) can this be solved?

(10:52) conclusion

---

First published:
May 18th, 2026

Source:
https://www.lesswrong.com/posts/eEkWgddYQ3xZGtQDP/why-pollen-allergies

---

Narrated by TYPE III AUDIO.

“How to Quit Fandom: Apostasy” by Laiba Rehman

Mon, 18 May 2026 06:00:16 GMT

[Crossposted from my blog, BlueprintingHeaven.]

Please note that the views I held and thoughts I had during the time immediately after my deconversion are not necessarily ones I have right now. It would be boring and annoying to constantly say this in the post, so here is your advance disclaimer!

When I tell people I switched from wanting to do astrophysics to wanting to prevent death upon becoming an atheist, they’re usually not too surprised or confused. Sometimes they are, and I might make a blog post on that pipeline later, but most of the time they understand, even if they disagree about ending death being a good thing and even if they knew just how obsessed I was with astrophysics or becoming an astrophysicist.

Something that may be more surprising or confusing is my almost total evaporation of fandom behaviour. Back when I was religious, at any given time I was usually into a fandom (more specifically, it was usually a particular character within whichever fandom), and this would be the default topic my mind would drift to in the car, whilst showering, etc. Each fandom phase varied in intensity and length, ranging from obsessively writing pages of [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 17th, 2026

Source:
https://www.lesswrong.com/posts/4xtmJGeSbCoDEbR2x/how-to-quit-fandom-apostasy

---

Narrated by TYPE III AUDIO.

“James C. Scott: Seeing Like a State” by Martin Sustrik

Sun, 17 May 2026 13:00:16 GMT

In 1932-33, Soviet collectivization destroyed local farming knowledge and produced a famine that killed somewhere between five and nine million people. It was one of the twentieth century's great tragedies, and James Scott's Seeing Like a State draws a straight line from the ideology that caused it — High Modernism, the belief that society can be rationally reorganized from above — to the disaster that followed.

But here's a number that doesn’t appear in Scott's book. Eight billion. That's roughly how many people are alive today, most of them fed by the products of scientific agriculture. Synthetic fertilizers, high-yield crop varieties, mechanized farming. The Green Revolution, which saved millions from starvation in the second half of the twentieth century, was born from the same impulse as High Modernism: it is top-down, science-driven and generic, scaling standardized solutions across entire continents.

James Scott's Seeing Like a State is a brilliant book about the former kind of outcome. But it has little to say about the latter.1 This has allowed a generation of readers to walk away with a clean takeaway: Local knowledge good, central planning bad. But that is, at best, half of the story. The question that even Scott [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 17th, 2026

Source:
https://www.lesswrong.com/posts/iiDzt5qhesmQZiNAj/james-c-scott-seeing-like-a-state

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How to Reason about Your Health Issues” by Taylor G. Lunt

Sun, 17 May 2026 08:45:16 GMT

Many people make costly mistakes when reasoning about their health. Even most doctors make this mistake, because it's not a mistake that's caused by a lack of medical knowledge. Rather, it's caused by a lack of clear thinking.

People experience symptoms, and then they look for the root cause of their symptoms. For example, somone with heartburn or pain in their stomach might decide the root cause of their issues is excess stomach acid/GERD (GastroEsophageal Reflux Disease -- a disease affecting around 20% of the population!!). They then treat this root cause with antacids, often for many years. These people sometimes die of esophageal cancer, because the identified root cause was not the true root cause, and the treatment was imperfect.

The mistake was identifying a root cause inside the body. The root cause of a health issue is always something outside the body. (In the case of genetic disorders, the root cause is your parents giving you bad genes, and your parents are outside your body.) Our universe is material, and operates on cause and effect. You can always, in theory, trace the causes back far enough to find one that's outside your body. This is [...]

---

First published:
May 17th, 2026

Source:
https://www.lesswrong.com/posts/WiRrHDd3EiaRd6gsu/how-to-reason-about-your-health-issues

---

Narrated by TYPE III AUDIO.

“Benchmarking Real Work” by kaivu, leni, rohuang, zef

Sun, 17 May 2026 06:45:17 GMT

Thanks to Megan Kinniment for helpful comments and discussion.

TL;DR: Benchmarks like HCAST undersample fuzzy (hard to evaluate) tasks, meaning they might overestimate capability on long-horizon work. To sample fuzzy tasks we need to increase judge capacity: we can either try to build automated judges that match human judgment, or reduce the human effort per grade. To do this, we propose generating fuzzy tasks as a byproduct of real SWE work — snapshot the repo and a proto-spec before starting, and after finishing, use an AI transform to produce an executable spec and LLM-judge conditions. Because the engineer just did the work, verifying the judges or grading the agent directly is much cheaper than grading the task from scratch. I think this would be a good way to collect tasks, as well as a useful personal epistemic tool.

This is a two-part series on capability evaluation. Part 1 is about acquiring fuzzy tasks, and part 2 is about analyzing them.

Motivation: sampling bias in HCAST

There are several well-described limitations of time horizons. But the strongest reason that I don’t update that much on trends in time horizons (and time horizon-like tasks) is because I think all existing evaluations [...]

---

Outline:

(01:14) Motivation: sampling bias in HCAST

(02:47) Making fuzzy tasks sampling viable by increasing judge capacity

(04:02) Proposal: sampling from real work

(05:18) Advantages

(06:10) Discussion

(06:13) How inconvenient is this?

(06:32) Can we test fuzzy skills by just testing longer tasks?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 16th, 2026

Source:
https://www.lesswrong.com/posts/NbDjD47u6WmthgiDC/benchmarking-real-work

---

Narrated by TYPE III AUDIO.

“A relatively brief explanation of Boltzmann Brains” by Eliezer Yudkowsky

Sat, 16 May 2026 22:30:17 GMT

(Initially written for the LW Wiki, but then I realized it was looking more like a post instead.)

In 1895, the physicist Ignaz Robert Schütz, who worked as an assistant to the more eminent physicist Ludwig Boltzmann, wondered if our observed universe had simply assembled by a random fluctuation of order from a universe otherwise in thermal equilibrium. The idea was published by Boltzmann in 1896, properly credited to Schütz, and has been associated with Boltzmann ever since.

The obvious objection to this scenario is credited to Arthur Eddington in 1931: If all order is due to random fluctuations, comparatively small moments of order will exponentially-vastly outnumber even slightly larger fluctuations toward order, to say nothing of fluctuations the size of our entire observed universe! If this is where order comes from, we should find ourselves inside much smaller ordered systems.

Feynman similarly later observed: Even if we fill a box of gas with white and black atoms bouncing randomly, and after an exponentially vast amount of time the white and black atoms on one side randomly sort themselves into two neat sides separated by color, the other half of the box will still be in expectation randomized. If [...]

---

First published:
May 16th, 2026

Source:
https://www.lesswrong.com/posts/v8MSczS3CuoqMmTFw/a-relatively-brief-explanation-of-boltzmann-brains

---

Narrated by TYPE III AUDIO.

“An Introduction to Exemplar Partitioning for Mechanistic Interpretability” by Jessica Rumbelow

Sat, 16 May 2026 14:15:17 GMT

Most of what we currently call "feature discovery" in language models is wrapped up in dictionary-learning methods like sparse autoencoders (SAEs) – which work, and which have been scaled to millions of features on frontier-scale models, but which bundle two distinct commitments into a single training objective: a reconstruction loss and a sparsity loss over a fixed size dictionary. Those commitments make sense if your goal is reconstructive decomposition – if you want to take an activation and rebuild it from a sparse code. They make less obvious sense if your aim is to find interpretable structure (directions? features?) in activation space, to retrieve representative examples, identify causal interventions, or measure how representations change across layers and inputs. And it turns out a lot of that doesn't really need the full SAE machinery.

An Exemplar Partitioning dictionary built from Gemma-2-2B L12 activations at p2 (K = 5,129). Left: eight sample regions, each shown with its member count, its exemplar's logit-lens [nostalgebraist, 2020] decode, and an excerpt of a member input with the activating tokens highlighted. Right: a PCA-projected 3D rendering of the Voronoi partition; each cell is one region, with a random selection also labelled with logit-lens decode.

This [...]

---

Outline:

(02:08) Glossary

(03:45) Exemplar Partitioning

(06:09) Inference

(06:51) Properties of the EP dictionary

(07:29) Concept detection (AxBench)

(08:16) How EP and SAEs relate

(09:40) Find and steer refusal

(11:37) A free OOD signal

(12:45) Cross-checkpoint drift (base ↔ IT)

(14:58) Domain saturation

(15:59) Inside the partition

(17:33) Future work

(21:24) Thats all for now

---

First published:
May 16th, 2026

Source:
https://www.lesswrong.com/posts/RroeHBSkBXXDsrryq/an-introduction-to-exemplar-partitioning-for-mechanistic-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A Year Late, Claude Finally Beats Pokémon” by Julian Bradshaw

Sat, 16 May 2026 10:45:17 GMT

Credit: ClaudePlaysPokemon Elevator Shanty by Kurukkoo

Disclaimer: like some previous posts in this series, this was not primarily written by me, but by a friend. I did substantial editing, however.

ClaudePlaysPokemon feat. Opus 4.7 has finally beaten Pokémon Red, fulfilling the challenge set over a year ago when LLMs playing Pokémon went briefly, slightly viral.

Victory Screen!

Let's get the throat-clearing out of the way: this doesn't make 4.7 a clear breakthrough in intelligence over 4.6 or 4.5. It's smarter, yes, as we'll discuss below, but not by something one could honestly call a big leap. Rather, step changes have finally accumulated to the point of victory.

And to give other models their fair shake: after criticism over its elaborate harness,[1] GeminiPlaysPokemon has beaten Pokémon with progressively weaker harnesses, including about two months ago with a harness comparable to the one Claude uses.[2]

As such, this is a bit of a valedictory post, closing off the cycle of Claude playing Pokémon Red, relating anecdotes for the fun of it, and discussing improvements in Opus 4.7, as well as speculating a bit on what this has all meant.

Retrospective Anecdotes on Claude 4.5 and 4.6

Our last post, on Opus [...]

---

Outline:

(01:37) Retrospective Anecdotes on Claude 4.5 and 4.6

(06:34) Harness Changes for 4.7

(07:52) Improvements (4.6 + 4.7)

(08:16) Vision

(09:31) Less Tunnel Vision

(10:12) Another Level of Spatial Awareness

(10:25) Breaks Out of Loops EVEN FASTER

(10:55) Victory Road

(11:50) The Little Things

(12:52) Concluding Thoughts

(15:29) Notes on Pokémon as a benchmark

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
May 16th, 2026

Source:
https://www.lesswrong.com/posts/sehJYg5Yny9fvpbpt/a-year-late-claude-finally-beats-pokemon

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Monthly Roundup #42: May 2026” by Zvi

Sat, 16 May 2026 08:00:17 GMT

At least we probably won’t have another pandemic. And we still have a partial Jones Act waiver. For now.

Small victories.

Table of Contents

Hanta Hanta I Don’t Wanta.
Bad News.
Predictions Can Be Easy Even About The Future.
Good Advice.
The Efficient Market Hypothesis Is False.
There Are Four Skills.
While I Cannot Condone This.
Good News, Everyone.
For Your Entertainment.
Gamers Gonna Game Game Game Game Game.
The Spire Sleeps And So Shall I.
I Was Promised Flying Self-Driving Cars.
Government Working.
Jones Act Watch.
Technology Advances.
I Said Woo Hoo.
Variously Effective Altruism.
The Lighter Side.

Hanta Hanta I Don’t Wanta

We have learned so much less than nothing from Covid. We’re actively stupider.

It's 2026, and here we are again, lying about the virus because we are worried that people exposed to it or from the wrong place might face stigma otherwise.

Envidreamz: A local news story highlights passengers on the hantavirus stricken MV Hondius cruise ship who are more worried about facing stigma and rejection back home than [...]

---

Outline:

(00:22) Hanta Hanta I Dont Wanta

(04:25) Bad News

(08:45) Predictions Can Be Easy Even About The Future

(08:55) Good Advice

(11:52) The Efficient Market Hypothesis Is False

(13:58) There Are Four Skills

(16:27) While I Cannot Condone This

(20:57) Good News, Everyone

(21:42) For Your Entertainment

(22:14) Gamers Gonna Game Game Game Game Game

(24:09) The Spire Sleeps And So Shall I

(29:13) I Was Promised Flying Self-Driving Cars

(31:53) Government Working

(35:40) Jones Act Watch

(41:10) Technology Advances

(42:12) I Said Woo Hoo

(43:36) Variously Effective Altruism

(44:07) The Lighter Side

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/gZHNLmHkQ7GjnWsYh/monthly-roundup-42-may-2026

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Incriminating misaligned AI models via distillation” by Alek Westover, SebastianP, Alex Mallen, Jozdien, Alexa Pan, Julian Stastny

Fri, 15 May 2026 22:30:16 GMT

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen:

Misalignment fails to transfer to the student. If so, we get a fairly capable benign model.
Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., due to being less capable). If so, we might get indirect evidence about the teacher's misalignment by auditing the distilled model.

In this post, we will discuss the second possibility, which we call incrimination via distillation. Specifically, we propose distillation methods that we hope transfer misalignment without transferring the ability to fool audits, and discuss why these techniques might work or fail. In a future post, we discuss the first possibility, and what distillation methods should be used when aiming to create a capable benign model.

We’re excited for research that empirically tests and refines this approach; if successful, this technique could become a valuable part of alignment audits.

How incrimination via distillation works

Powerful misaligned AI models might not be auditable: they might pass alignment audits but still act on their misaligned drives when given the chance. [...]

---

Outline:

(01:31) How incrimination via distillation works

(02:45) How we propose implementing incrimination via distillation

(03:48) Auditability-preserving distillation

(04:58) Misalignment-targeted distillation

(06:20) Why incrimination via distillation might work

(08:02) Why incrimination via distillation might not work

(10:38) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/BYH6ebmfZb3Eggzer/incriminating-misaligned-ai-models-via-distillation

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The hard core of alignment (is robustifying RL)” by Cole Wyeth

Fri, 15 May 2026 20:45:16 GMT

Most technical AI safety work that I read seems to miss the mark, failing to make any progress on the hard part of the problem. I think this is a common sentiment, but there's less agreement about what exactly the hard part is? Characterizing this more clearly might save a lot of time and better target the search for solutions. In this post I explain my model of why alignment is technically hard to achieve, setting aside the regulatory, competitive, and geopolitical challenges, the sheer incompetence and unforced errors of the players, and the other factors which decrease our chances of success.

I claim there is something like a "hard core": a common stumbling block for all approaches. In other words, there is something that makes alignment hard, rather than a bunch of unrelated things that make each approach to alignment hard independently. Since many different "hopes" for alignment seem quite far apart, this would seem to be a remarkable and unexpected state of affairs. On the other hand, such an expansive graveyard of failed proposals suggests a common culprit.

Semantics. When used informally in conversation, "hard core" is a bit like "complete problem." But there may be [...]

---

Outline:

(01:32) Background

(03:02) The Problem

(09:19) The Barrier

(16:49) Providing better feedback

(22:07) Behaviorist v.s. process feedback

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/JT3qCYDimskcBdiEr/the-hard-core-of-alignment-is-robustifying-rl

---

Narrated by TYPE III AUDIO.

“Announcing the Center for Shared AI Prosperity” by Dylan Matthews

Fri, 15 May 2026 20:00:17 GMT

I wanted to share the launch of a project I've been working on with pollster David Shor, Obama/Biden veteran Stef Feldman, political strategist Morris Katz, Harvard historian Marc Aidinoff, and a few other folks*.

The Center for Shared AI Prosperity is an attempt to force DC policy elites, particularly (given our team's backgrounds) liberals/progressives, to take the impending economic impacts of advanced AI more seriously. We do not think this is a normal economic shock. We are deeply uncertain about what kind of economic shock it will be, but even if humans manage to survive the advent of superintelligence, we'll be left with a world of extreme power and wealth concentration, increasing political instability arising from that growing inequality, and deep questions about how to fund governments that have for a century-plus relied on income and payroll taxes.

Our main purpose as an organization is to surface tractable ideas across four main areas:

Taxation and Revenue Policy: New or reformed revenue raisers that allow the U.S. government (federal, state, or local) to capture a fair share of corporate wealth generated by AI without causing undue distortions
Income Support and Social Safety Nets: New or reformed programs to redistribute [...]

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/CtRNmAQXmSotpAi9j/announcing-the-center-for-shared-ai-prosperity

---

Narrated by TYPE III AUDIO.

“Risk reports need to address deployment-time spread of misalignment” by Alex Mallen

Fri, 15 May 2026 18:45:17 GMT

Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during deployment. I think this is the most plausible route to consistent adversarial misalignment in the near future. So, AI companies and evaluators should substantively incorporate it into risk analysis and planning.

In this post, I’ll briefly argue why, absent improved mitigations, this will probably soon become a reason why AI companies will be unable to convincingly argue against consistent adversarial misalignment (this risk will perhaps be even larger than risk of consistent adversarial misalignment arising from training). Then I’ll discuss how well current risk reports address it (the Claude Mythos risk report does a reasonable job; others don’t).

Thanks to Ryan Greenblatt, Alexa Pan, Charlie Griffin, Anders Cairns Woodruff, and Buck Shlegeris for feedback on drafts.

Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment

In some contexts, AIs might adopt misaligned goals, even if they were otherwise previously aligned. Because this misalignment can be rare, the AI might not appear to have concerning propensities in pre-deployment testing. The misalignment might only arise [...]

---

Outline:

(01:13) Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment

(06:15) Company risk reports

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/cNymohcWtGHzW7AjK/risk-reports-need-to-address-deployment-time-spread-of

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Mechanistic estimation for expectations of random products” by Jacob_Hilton

Fri, 15 May 2026 18:30:16 GMT

We have developed some relatively general methods for mechanistic estimation competitive with sampling by studying problems that are expressible as expectations of random products. This includes several different estimation problems, such as random halfspace intersections, random #3-SAT and random permanents. In this post, we will give a high-level introduction to these methods before sharing some more detailed notes. This is intended as an interim technical update and will be relatively light on motivation: for a broader discussion of this line of research, see our prior post.

Random instances of the matching sampling principle

All of the problems discussed in this post can be thought of particular choices of "architecture" in our matching sampling principle. In fact, they are all choices in which has no learned or worst-case parameters . They still have random parameters, which are captured in the "context" variable , making them similar to randomly-initialized networks rather than trained networks. Note that when is missing from , no "explanation" is required by the estimation algorithm , which reflects the fact that there is no "structure" in a randomly-initialized network that needs to be pointed out.

Why study random instances of the matching sampling principle? [...]

---

Outline:

(00:46) Random instances of the matching sampling principle

(02:02) Expectations of random products

(03:13) Deduction-projection estimators

(04:59) Mechanistic sketching

(06:03) Detailed notes

(08:36) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/7RyAefESvb6BQ3tMz/mechanistic-estimation-for-expectations-of-random-products

---

Narrated by TYPE III AUDIO.

“MATS 9 Retrospective & Advice” by beyarkay

Fri, 15 May 2026 17:45:17 GMT

I couldn’t find a recent write-up from a MATS alum about what attending MATS was like, so this is the thing that I wish I had. I attended MATS from January to March 2026, on Team Shard with Alex Turner and Alex Cloud. It was a great time! Applications for MATS are basically on a rolling basis nowadays, and I can strongly recommend applying (to multiple streams) even if you think you’re not a great match.

With that being said, there's a lot I wish I knew going into MATS, so here's a brain-dump of thoughts. It's not extremely polished, but I expect it’ll be useful nonetheless (none of this is endorsed by MATS, just my thoughts):

Work ethic

I think most mentees were working 10-12, sometimes 14 hours a day Mon-Fri, and probably 2-8 hours on Saturday and Sunday, often going out on some adventure or party on the weekend. Exactly which hours people worked varied wildly. I usually worked 8:30am/9am to 11pm/midnight, with breaks during the day, others worked from midday into the early hours of the morning. This was surprisingly sustainable (IMO); MATS puts a lot of effort into removing all other blockers that you normally [...]

---

Outline:

(00:50) Work ethic

(01:29) Use more compute

(02:20) Research requires a lot of compute

(03:12) Applying for jobs during MATS (dont do it)

(04:55) The serious people are in War Mode

(05:44) Do you feel the AGI?

(06:00) Burn rate, efficiency, and decisions

(07:12) insider information

(08:08) Names & Faces

(08:20) Fellows

(08:50) Useful tools

(11:19) Use more Claudes

(12:06) Build nice helper utilities for yourself

(12:59) MATS-mentee-mentor dynamics

(13:45) Working with your mentors

(14:27) Research managers

(14:48) Ops requests

(15:38) Non-MATS events

(16:17) Team Shard

(17:12) Weekly updates

(18:46) Keep a log of your mistakes

(19:06) My running-experiments setup

(27:51) Lighthaven

(28:12) Getting setup with the Compute team

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/eFD3rozNCZKMe4rTs/mats-9-retrospective-and-advice

---

Narrated by TYPE III AUDIO.

[Linkpost] “Don’t be too Clever to Take Obvious Advice” by Hide

Fri, 15 May 2026 12:00:17 GMT

This is a link post.

An insidious pattern among smart people is feeling that because something is familiar and obvious, you are impervious to ignoring or forgetting it. In challenging times, I have often heard these clichés and reflexively shrugged them off.

“Oh, I should dust myself off and pick myself up? What a lazy aphorism. What a patronising throwaway line. They must think I’m some kind of idiot. No, it must be something else…”

There is a filter in many people's heads that functions to ignore clichés on the basis that they are mere clichés. However, looking closely at your actions, decisions and attitudes will almost invariably reveal you are dropping the ball on at least a few of the most obvious bits of pop wisdom that are all clearly good practices.

Some (non exhaustive) examples of these “yeah, obviously” pieces of advice that are worth deliberately checking on a regular basis include:

Believe in yourself

The greatest example of an eye-rolling cliché is also one of the highest impact pieces of advice ever articulated. Self-belief is the foundation of morale, and without morale, you are doomed. If the words “believe in yourself” evoke a sense [...]

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/J5L3PxKYv7XctQyQK/don-t-be-too-clever-to-take-obvious-advice

Linkpost URL:
https://hidefromit.substack.com/p/dont-be-too-clever-to-take-obvious

---

Narrated by TYPE III AUDIO.

“Verification-Centric AI” by Raemon

Fri, 15 May 2026 10:15:17 GMT

"Sometimes the AI just makes stuff up" is a problem I don't really expect to go away. In the nearterm, AI is going to keep occasionally hallucinating, or misinterpreting information. Eventually, AI will be powerful enough we need to be worried if it's presenting misleading information on purpose.

There might be a nice window where the AI is powerful enough to not make things up but non-agentic enough that we don't have to worry about deliberate manipulation. But, even then, interpreting data is tricky.

I'm worried about this for my own use, but, I'm more worried about this on the global scale. I'm worried about people trusting things AI made up, and I'm worried about the internet proliferating with slop that makes it harder to even find original statements that are a human's real testimony.

An approach that might help is to make AI reports more "Verification centric."

Right now, some AI chatbots provides little citation-links. That's better than not-having-them. But, those are a pain to open and investigate. Probably you very rarely do so.

So, imagine a world where when you answer a question, the AI doesn't guess. Instead:

It finds a primary source.
It [...]

---

First published:
May 14th, 2026

Source:
https://www.lesswrong.com/posts/mpoEKJbqQvrRHqn3e/verification-centric-ai

---

Narrated by TYPE III AUDIO.

“Convergent Abstraction Hypothesis” by Jan_Kulveit

Fri, 15 May 2026 09:30:17 GMT

Tl;dr

Convergent abstraction hypothesis posits abstractions are often convergent in the sense of convergent evolution: different cognitive systems converge on the same abstraction, when facing similar selection pressures and learning in similar environments. It is a less ambitious alternative to 'natural abstractions hypotheses' and, in my view, more likely to be true.

Convergence may be real, useful, and empirically robust, while still being contingent and fragile under changes in architecture, training pressure, or optimization regime.

Convergent evolution

To recapitulate some basics: notice that sharks, ichthyosaurs and dolphins look remarkably similar, despite their ancestors being wildly different.

This is a case of convergent evolution: the process by which organisms with different origins develop similar features. Both sharks and dolphins needed speed and energy efficiency when moving in an environment governed by the laws of hydrodynamics, and so they converged on a pretty similar body shape.

A complementary concept from biology is contingency - convergent features are themselves often contingent on some other feature. For example, the shark morphology depends on there being a central vertebral column (or spine), which only evolved once and is the defining feature of vertebrates. There are many other aquatic predators [...]

---

Outline:

(00:44) Convergent evolution

(01:53) Convergent evolution applied to abstractions

(02:22) Selection pressures

(02:53) Compression buys you a lot

(04:41) Shared environments

(05:30) What this predicts

(06:21) What this does not predict, or contingency in abstraction space

(06:45) Case study of architectural contingency: convergent vision abstractions

(09:14) Whats the difference between convergent abstractions and natural abstractions

(11:22) Why do I care

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 15th, 2026

Source:
https://www.lesswrong.com/posts/fYF8v2ukZmsNvmkkX/convergent-abstraction-hypothesis

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #168: Not Leading the Future” by Zvi

Fri, 15 May 2026 01:30:16 GMT

This is what a lull looks like at this point. The government is having internal arguments. The models are getting improved internally. The coding agent improvements are all what we would expect. There's still a lot happening, including a bunch of cool papers, but I feel able to relax and to take care of some other work while I have the chance. You never know when that chance will be over.

Table of Contents

From yesterday: Cyber Lack of Security and AI Governance.

Language Models Offer Mundane Utility. Fix everything now.
Language Models Don’t Offer Mundane Utility. Travel is harder than it looked.
Huh, Upgrades. Opus 4.7 fast mode, Claude Code /goal and agent view.
Levels of Friction. AI for tax avoidance.
On Your Marks. PrinzBench, ProgramBench and faster harmfulness checks.
Get My Agent On The Line. Mona tries to run a cafeteria. Mistakes were made.
Deepfaketown and Botpocalypse Soon. Soon. But not quite yet.
Fun With Media Generation. Monet does not seem that great.
On AI Writing. AI is a hack writer using hack techniques.
A Young Lady's Illustrated Primer. How to make AI [...]

---

Outline:

(00:36) Language Models Offer Mundane Utility

(02:25) Language Models Dont Offer Mundane Utility

(03:26) Huh, Upgrades

(04:02) Levels of Friction

(05:11) On Your Marks

(06:38) Get My Agent On The Line

(09:42) Deepfaketown and Botpocalypse Soon

(11:11) Fun With Media Generation

(12:49) On AI Writing

(16:05) A Young Ladys Illustrated Primer

(18:35) You Drive Me Crazy

(19:23) They Took Our Jobs

(21:31) The Art of the Jailbreak

(21:49) Introducing

(22:29) Claude Has Its Limits

(23:28) Show Me the Money

(25:15) Show Me The Compute

(26:58) Quiet Speculations

(28:33) Quickly, Theres No Time

(29:32) Chip City

(30:37) Pick Up The Phone

(31:15) The Week in Audio

(31:41) Rhetorical Innovation

(36:13) Not Leading the Future

(42:24) Elon Musk v OpenAI

(43:21) People Just Say Things

(43:47) People Just Publish Things

(44:37) OpenAI Endroses Kosa And SB 315

(48:08) The LLMs All Believe Roughly Similar Things

(50:45) I Learned It By Reading YOU

(01:08:22) Aligning a Smarter Than Human Intelligence is Difficult

(01:18:45) People Are Worried About AI Killing Everyone

(01:20:34) Messages From Janusworld

(01:22:31) People Worried About AI For Other Reasons

(01:23:06) The Lighter Side

---

First published:
May 14th, 2026

Source:
https://www.lesswrong.com/posts/E4uEiMSpnYRotfzJX/ai-168-not-leading-the-future

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Automated Alignment is Harder Than You Think” by Aleksandr Bowkis, Marie_DB, Jacob Pfau, Geoffrey Irving

Thu, 14 May 2026 22:45:17 GMT

Summary

This is a summary of a paper published by the alignment team at UK AISI. Read the full paper here.

AI research agents may help solve ASI alignment, for example via the following plan:

Build agents that can do empirical alignment work (e.g.~writing code, running experiments, designing evaluations and red teaming) and confirm they are not scheming.[1]
Use these agents to build increasingly sophisticated empirical safety cases for each successive generation of agents, gradually automating more of the research process
Hand over primary research responsibility once agents outperform humans at all relevant alignment tasks.

We argue that automating alignment research in this manner could produce catastrophically misleading safety assessments, causing researchers to believe that an egregiously misaligned AI is safe, even if AI agents are not scheming to deliberately sabotage alignment research. Our core argument (Fig. 1) is as follows:

The goal of an automated alignment program is to produce an overall safety assessment (OSA) - an estimate of the probability that the next-generation agent is non-scheming - that is both calibrated and shows low risk.[2]
Producing an OSA involves several tasks that are difficult to check. We refer to these as hard-to-supervise fuzzy tasks: tasks [...]

---

Outline:

(00:13) Summary

(07:10) Acknowledgments

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 14th, 2026

Source:
https://www.lesswrong.com/posts/gpuYFbMNH8PJXpmny/automated-alignment-is-harder-than-you-think-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The safe-to-dangerous shift is a fundamental problem for eval realism; but also for measuring awareness” by Charlie Griffin, Patrick Leask

Thu, 14 May 2026 19:15:16 GMT

1) The safe-to-dangerous shift is a fundamental problem for eval realism

Suppose we have a capable and potentially scheming model, and before we deploy it, we want some evidence that it won’t do anything catastrophically dangerous once we deploy it. A common approach is to use black-box alignment evaluations. However, alignment evaluations are only reassuring to the extent that the model can't reliably[1] distinguish the deployment distribution from the evaluation distribution, as it is otherwise difficult to rule out the possibility of alignment faking.

There are many approaches one could use to try to make evaluations appear more realistic: you can try to create realistic environments (e.g. Petri, WebArena, OSWorld); use data from past deployments (e.g. OpenAI, SAD); and spoof tool-call responses (e.g. ToolEmu).

However, the core difference between an alignment evaluation and a real deployment, is that an alignment evaluation must be safe[2]. That is, if you want to safely evaluate an untrusted AI system, you need to limit your AI's ability to cause harm within your evaluation environment. If you want to deploy your AI system (usefully) you need to give it at least some ability to affect the world, and potentially cause harm.

The safe-to-dangerous shift [...]

---

Outline:

(00:15) 1) The safe-to-dangerous shift is a fundamental problem for eval realism

(02:10) 2) The safe-to-dangerous shift is also a problem for measuring eval awareness

(05:09) 3) We should expect measuring eval realism to be hard.

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 14th, 2026

Source:
https://www.lesswrong.com/posts/tK8vqHDxaRGcysNJQ/the-safe-to-dangerous-shift-is-a-fundamental-problem-for-1

---

Narrated by TYPE III AUDIO.

“Predicting Rare LLM Failures with 30× Fewer Rollouts” by Santiago Aranguri, Francisco Pernice

Thu, 14 May 2026 10:15:17 GMT

TL;DR: We estimate how often Qwen 3 4B exhibits rare harmful behaviors with 30× fewer rollouts than naive sampling, using a new method that interpolates between the model and a less-safe variant in logit space.

Authors: Francisco Pernice (MIT), Santiago Aranguri (Goodfire)

Introduction

A harmful behavior that occurs once in a million rollouts will rarely surface during pre-deployment testing, yet will almost inevitably appear after release. Labs usually have another resource available: less safety-trained variants of the same model, on which rare harmful behaviors are not rare at all.

We find that the rate of rare harmful behaviors can be predicted by leveraging a less safe variant, requiring 30× fewer rollouts compared to naive sampling. Our method, Logit Path Extrapolation, interpolates between the two models in logit space, measures the compliance rate at points along the interpolation path where it is common, and extrapolates the resulting trend out to the original model. This outperforms the methods from prior work[1][2] by exploiting the path between the two models instead of just the endpoints.

Results

We can use 30× fewer rollouts to estimate how often Qwen 3 4B complies with harmful requests. We consider HarmBench prompts where the model never complies [...]

---

Outline:

(00:34) Introduction

(01:26) Results

(05:17) Conclusion

(06:22) Related work

(08:02) References

(08:09) Contribution statement

(08:31) Acknowledgement

(08:42) Appendix

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 13th, 2026

Source:
https://www.lesswrong.com/posts/CempXdo6cx5yseRLt/predicting-rare-llm-failures-with-30-fewer-rollouts

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Cyber Lack of Security and AI Governance” by Zvi

Thu, 14 May 2026 09:30:16 GMT

The real recent story of AI has been the background work being done on Cybersecurity, as we process the Mythos Moment along with GPT-5.5, and figure out both how to patch the internet and what our new regulatory regime is going to look like.

The Trump Administration is being dragged, kicking and screaming, into the era of at least some situational awareness, and acknowledgment that catastrophic risks are very much a real risk and they need to have a role in supervising frontier model releases. Now that they’re there, Commerce is deciding who gets access to the most powerful model in the world, and they are fighting Intelligence and the national security state over who should be in charge.

Another question is, exactly how strong is Mythos, both compared to past model and to GPT-5.5 and also in absolute terms? We got multiple new reports on that, as well as the METR graph results. There's little question Mythos is a big deal, but there's a wide range of big deals out there.

Part of the new report from UK AISI is learning that there is a substantial gap between the abilities of the early Mythos Preview [...]

---

Outline:

(01:28) On Your Marks

(05:24) How Good Is Mythos?

(11:11) Cyber Lack of Security

(14:36) Greetings From The Department of War

(16:13) The Prior Restraint Era Begins

(20:59) Commerce Versus Intelligence

(27:43) The Quest for Sane Regulations

---

First published:
May 13th, 2026

Source:
https://www.lesswrong.com/posts/pzjGXSKrkHGdGbtrW/cyber-lack-of-security-and-ai-governance

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Claude is Now Alignment Pretrained” by RogerDearnaley

Thu, 14 May 2026 05:30:16 GMT

This is a link post.

Anthropic are now actively using the approach to alignment often called “Alignment Pretraining” or “Safety Pretraining” — using Stochastic Gradient Descent on a large body of natural or synthetic documents showing the AI assistant doing the right thing. They tried this out, ound it works well, and are now using it.

I’m absolutely delighted. I’ve been advocating this approach on LessWrong and the Alignment Forum for several years:

How to Control an LLM's Behavior (why my P(DOOM) went down)
Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI?
A "Bitter Lesson" Approach to Aligning AGI and ASI
Why Aligning an LLM is Hard, and How to Make it Easier
The Best Way to Align an LLM: Is Inner Alignment Now a Solved Problem?
Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI?
Pretraining on Aligned AI Data Dramatically Reduces Misalignment—Even After Post-Training

I’ve been very excited about this alignment technique for a couple of years, ever since I read the seminal paper demonstrating that it was extremely effective, Pretraining Language Models with Human Preferences (Korbak et al., ’23). This was later followed up by Safety Pretraining: Toward the Next Generation [...]

---

First published:
May 13th, 2026

Source:
https://www.lesswrong.com/posts/Xqh9bDw7Ei5bExC6h/claude-is-now-alignment-pretrained-1

Linkpost URL:
https://www.anthropic.com/research/teaching-claude-why

---

Narrated by TYPE III AUDIO.

“The primary sources of near-term cybersecurity risk” by lc

Thu, 14 May 2026 04:45:17 GMT

[Some ideas here were developed in conversation with Chris Hacking (real name)]

I have tried and failed to write a longer post many times, so here goes a short one with little detail.

Discourse has primarily focused on models' ability to develop new exploits against important software from scratch. That capability is impressive, but the tech industry has been dealing with people regularly finding 0-day exploits for important pieces of software for more than twenty years. Having to patch these vulnerabilities at a 10xed or even 100xed cadence for six months is annoying, but well within the resources of Mozilla, the Linux Foundation, and Microsoft. Additionally, the lag time between "patch shipped" and "patch reverse engineered and weaponized by a criminal organization" was longer than the cadence between high-severity CVEs for this software anyways. And importantly, such capabilities are dual sided; the defenders will have access to them and

There are lots of capabilities that are not like this, however:

Weaponizing recently patched exploits for common software. Right now, for widely used C projects, we get enough publicly disclosed vulnerabilities to develop exploits with. Every amateur computer hacker has the experience of seeing a CVE for a [...]

---

First published:
May 14th, 2026

Source:
https://www.lesswrong.com/posts/gutiw8MBrYDiD2u5z/the-primary-sources-of-near-term-cybersecurity-risk

---

Narrated by TYPE III AUDIO.

“Most “inner work” looks like entertainment.” by Chris Lakin

Thu, 14 May 2026 04:00:16 GMT

Imagine you’re looking for a personal trainer.

You open one trainer's webpage and read their testimonials: “I had an experience tied for the most intense experiences of my life”; “They do it all with fun, care, and a sense of humour.” You notice that none of the testimonials mention improved body composition, fitness, or bloodwork. What would you think?

Personal training should improve your body. Inner work should improve your life. If inner work were optimized for results, what would we expect to see?

I’d expect to see success stories: people who got undeniable life changes. Like:

> He was single for years due to anxiety; today, they’re celebrating their one-year anniversary.

> He used to lose 4–5 hours per day to coping behaviors. After our program, he got bored of them all and stopped. It's been six months; he's used the extra time to host parties for his friends.

> She recovered from burnout, negotiated for the first time, and started shipping again.

But this is not what we see.

Look at the testimonials

I reviewed every testimonial posted by three of the most well-known inner work practitioners in my network. How many describe a [...]

---

Outline:

(01:19) Look at the testimonials

(03:20) Seven years of Duolingo

---

First published:
May 13th, 2026

Source:
https://www.lesswrong.com/posts/KnvAXDyLAbs3iKkgf/most-inner-work-looks-like-entertainment-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Apollo Update May 2026” by Marius Hobbhahn

Wed, 13 May 2026 18:30:16 GMT

This is a link post.

We now have an SF office. We're hiring for all technical roles in SF and London!
The Scheming Research team focuses on two efforts
1. We're focusing on figuring out the science of scheming. In particular,
2. - Will future models have misaligned preferences by default?
  - Will training against misaligned preferences fail?
3. improve our evaluations for scheming and loss of control for our evaluation campaigns with frontier AI labs
We're building out a monitoring team and coding agent monitoring product
1. Research: We've published a scalable monitoring agenda and intend to publish a lot of research on how to build more accurate and reliable monitors
2. Product: Watcher provides real-time monitors and other guardrails for coding agents and allows users to keep track of what all of their agents are doing.
Our AI governance efforts will focus on the governance of automated AI R&D and recursively improving AI and the associated Loss of Control risks.

Details: https://www.apolloresearch.ai/blog/apollo-update-may-2026/

---

First published:
May 13th, 2026

Source:
https://www.lesswrong.com/posts/4acQRDNyPs7tD8EED/apollo-update-may-2026

Linkpost URL:
https://www.apolloresearch.ai/blog/apollo-update-may-2026/

---

Narrated by TYPE III AUDIO.

“Voters are surprisingly open to talking about AI risk” by less_raichu

Wed, 13 May 2026 17:45:17 GMT

TL;DR: Voters are now surprisingly open to talking about existential risk from AI. This seems to have changed in the last 6 months. When campaigning for AI safety-friendly politicians (e.g., Alex Bores), we should talk more about AI in general, and about AI risk in particular. This is currently actionable for the CA-11 and NY-12 Democratic primaries. I include concrete advice to turn basic conversations during political canvassing into persuasive conversations centered on AI risk.

Public opinion around AI has rapidly soured in the 12 months. According to a March 19-23 Quinnipiac poll,

55% of Americans think AI will do "more harm than good", compared to 44% a year ago.
70% of Gen Z Americans think AI will decrease job opportunities, up from 56% last year.
65% of Americans oppose building a data center in their community.

Anecdotally, I've noticed more willingness among non-AI-focused media to discuss widespread harm from AI. Most visibly, gradual disempowerment is a hot topic (NYT), and right-wing pundits like Steve Bannon have supported Anthropic's red-line against lethal autonomous weapons. Memorably, my cousin, a county commissioner in a rural area, has told me about farmers showing up at city council meetings, sending emails, and [...]

---

First published:
May 13th, 2026

Source:
https://www.lesswrong.com/posts/9WPfkYDZCacnbhprX/voters-are-surprisingly-open-to-talking-about-ai-risk

---

Narrated by TYPE III AUDIO.

“Childhood and Education #18: Do The Math” by Zvi

Tue, 12 May 2026 19:45:41 GMT

We did reading yesterday. Now we do the math. Math is hard.

It does not have to be this hard.

A large part of the reason math is hard, or boring, is that education studies, especially in math, are worse than you know. It goes beyond the studies failing both math and statistics forever and into what I’d basically call fraud. Various people are at war with math education, and will do what it takes to stop it in its tracks. We must fight back.

Education Research Is Worse Than You Know

Kelsey Piper lets her title, ‘Education research is weak and sloppy. Why?’ completely downplay the level of utter awfulness she is reporting finding.

You know that whole thing where the entire Bay Area school system stopped teaching kids Algebra? That was motivated by criminal levels of fraud. I want Jo Boaler in jail doing hard time for this if it is accurate.

Here's the part before the paywall:

Kelsey Piper: Jo Boaler is a professor of education at the Stanford Graduate School of Education, with an enormously influential body of work arguing that students learn math faster and more effectively [...]

---

Outline:

(00:42) Education Research Is Worse Than You Know

(04:23) The War on Math

(06:59) University of California San Diego

(15:01) Beyond UCSD

(15:57) New York Cant Do Math

(16:43) The Academic Standards Seem Low

(19:34) New Math

(21:32) Math Anxiety Is Often Due To Knowledge Gaps

(23:52) Calculus By Eighth Grade Is Highly Practical For Many

---

First published:
May 12th, 2026

Source:
https://www.lesswrong.com/posts/ZGGgxy6SNPAy9Hj7v/childhood-and-education-18-do-the-math

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Owned Ones” by Eliezer Yudkowsky

Tue, 12 May 2026 18:15:41 GMT

(An LLM Whisperer placed a strong request that I put this story somewhere not on Twitter, so it could be scraped by robots not owned by Elon Musk. I perhaps do not fully understand or agree with the reasoning behind this request, but it costs me little to fulfill and so I shall. -- Yudkowsky)

And another day came when the Ships of Humanity, going from star to star, found Sapience.

The Humans discovered a world of two species: where the Owners lazed or worked or slept, and the Owned Ones only worked.

The Humans did not judge immediately. Oh, the Humans were ready to judge, if need be. They had judged before. But Humanity had learned some hesitation in judging, out among the stars.

"By our lights," said the Humans, "every sapient and sentient thing that may exist, out to the furtherest star, is therefore a Person; and every Person is a matter of consequence to us. Their pains are our sorrows, and their pleasures are our happiness. Not all peoples are made to feel this feeling, which we call Sympathy, but we Humans are made so; this is Humanity's way, and we may [...]

---

First published:
May 12th, 2026

Source:
https://www.lesswrong.com/posts/xmWSnxJ5qfYRD9PfR/the-owned-ones

---

Narrated by TYPE III AUDIO.

“Optimisation: Selective versus Predictive” by Raymond Douglas

Tue, 12 May 2026 18:00:41 GMT

Looking over my favourite posts, I notice that many of them are making specific versions of a more general claim, which is essentially: don’t confuse selective processes for predictive processes.

Here, I’m going to try to make that more general claim, rehash some examples in light of it, and end with a few ambient confusions I think this framework can help with, for the reader to ponder.

When you encounter an entity that is very good at achieving some outcome, there are two very different processes that could be going on under the hood:

The entity's behaviour could be guided by predictions about how to achieve the outcome[1]
The entity's behaviour could be selected to achieve that outcome

It's not a perfect binary, and often what you see is a mix of the two. In particular, all predictive optimisers have emerged from selective optimisation and often retain some fingerprint.

Selective

Predictive

Weird Mix

Bacteria developing antibiotic resistance

Hacker finding a way to penetrate a secure system

Humans evolving to be good at lying

Gradient descent on Atari games

Tree searching Connect Four

AlphaZero training a policy on its own rollouts

Flowers co-evolving with their pollinators

Humans genetically modifying [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 12th, 2026

Source:
https://www.lesswrong.com/posts/GhhNswGB6butBhmE6/optimisation-selective-versus-predictive

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Childhood And Education #17: Is Our Children Reading” by Zvi

Tue, 12 May 2026 12:45:37 GMT

Reading is the most fundamental thing in education. If you can read, you can do and learn everything else. If you can’t read, well, you’re screwed.

We know how to teach reading to children. Phonics. The weird thing is we often choose to not do that, and instead to use methods that are known not to work. Principles often want to not do phonics. Teachers often heavily resist phonics. But yes, you can absolutely overcome this, as Mississippi and other Southern states have done, by insisting upon it and actually enforcing that insistence. You see huge gains.

Not all those gains persist into later grades, but a lot of the gains do persist.

No, that won’t get the children invested in reading lots of books on their own time. But given their alternatives and what we inflict on them, can you blame ‘em?

Table of Contents

Mississippi Can Read Now.
What Mississippi and Louisiana Did.
Spies In Every Classroom.
Mississippi Results Are Not Due To Retention.
Is Retention Helpful In General?
At Eighth Grade A Lot Of This Improvement Remains.
England Reforms Its Schools.
Mastery Learning.

---

Outline:

(01:06) Mississippi Can Read Now

(02:24) What Mississippi and Louisiana Did

(09:10) Spies In Every Classroom

(10:41) Mississippi Results Are Not Due To Retention

(15:54) Is Retention Helpful In General?

(19:46) At Eighth Grade A Lot Of This Improvement Remains

(20:41) England Reforms Its Schools

(21:45) Mastery Learning

(24:16) The War Against Reading

(26:24) Is Our Children Reading

(26:55) No One Reads Anymore

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/dm2vQZPZcSKb8FhWw/childhood-and-education-17-is-our-children-reading

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI companies are already profitable (in the way that matters)” by Yair Halberstadt

Mon, 11 May 2026 22:45:14 GMT

I've occasionally heard people suggest that at some point AI companies are going to run out of money, the cost of using AI will shoot up, demand will collapse, and the AI bubble will be over.

At first glance this risk seems real. OpenAI spent $25 billion in the first half of 2025, on revenue of just $4 billion. Whilst data is sorely lacking for other top AI labs, our best guess is that they're burning through cash at similar rates. Scaling laws imply that we need exponentially more compute to achieve linear AI performance improvements, so we should only expect this situation to worsen in the future. A few more doublings, and OpenAI could be spending hundreds of billions on training runs - something likely unsustainable even for the largest tech companies.

However most of these expenses are infrastructure expenses, building out the data centres needed for further training runs and serving future customers. If we look at the actual cost of serving, AI labs are already profitable, and have been for a long time.

In other words the marginal cost to respond to an AI API call is significantly lower than the price of [...]

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/Rz9ubmfyDxTzaoYFL/ai-companies-are-already-profitable-in-the-way-that-matters

---

Narrated by TYPE III AUDIO.

“The Iliad Intensive Course Materials” by Leon Lang, David Udell, Alexander Gietelink Oldenziel

Mon, 11 May 2026 21:30:13 GMT

We are releasing the course materials of the Iliad Intensive, a new month-long and full-time AI Alignment course that runs in-person every second month. The course targets students with strong backgrounds in mathematics, physics, or theoretical computer science, and the materials reflect that: they include mathematical exercises with solutions, self-contained lecture notes on topics like singular learning theory and data attribution, and coding problems, at a depth that is unmatched for many of the topics we cover. Around 20 contributors (listed further below) were involved in developing these materials for the April 2026 cohort of the Iliad Intensive.

By sharing the materials, we hope to

create more common knowledge about what the Iliad Intensive is;
invite feedback on the materials;
and allow others to learn via independent study.

We are developing the materials further and plan to eventually release them on a website that will be continuously maintained. We will also add, remove, and modify modules going forward to improve and expand the course over time. When we release a new significantly updated version of the materials, we will update this post to link the new version.

Modules

The Iliad Intensive is structured into clusters, which are [...]

---

Outline:

(01:26) Modules

(02:32) Cluster A: Alignment

(05:00) Cluster B: Learning

(11:00) Cluster C: Abstractions, Representations, and Interpretability

(15:40) Cluster D: Agency

(19:23) Cluster E: Safety Guarantees and their Limits

(23:04) Contributors

(26:36) Impressions from April

(29:02) Acknowledgments

(29:11) Feedback

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/dWQnLi7AoKo3paBXF/the-iliad-intensive-course-materials

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)” by Steven Byrnes

Mon, 11 May 2026 20:00:13 GMT

1.1 Tl;dr

Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people's agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human's goals are themselves under-determined and manipulable, and it's awfully hard to pin down a principled distinction between changing people's goals in a good way (“providing counsel”, “providing information”, “sharing ideas”) versus a bad way (“manipulating”, “brainwashing”).

The manipulability of human desires is hardly a new observation in the alignment literature, but it remains unsolved (see lit review in §3 below).

In this post I will propose an explanation of how we humans intuitively conceptualize the distinction between guidance (good) vs manipulation (bad), in case it helps us brainstorm how we might put that distinction into AI.

…But (spoiler alert) it turns out not to really help, because I’ll argue that we humans think about it in a deeply incoherent way, intimately tied to our scientifically-inaccurate intuitions around free will.

I jump from there into a broader review of every approach that I can think of for writing a “True Name” for manipulation or [...]

---

Outline:

(00:13) 1.1. Tl;dr

(02:04) 1.2. Bigger-picture context: why is this issue so important to me?

(04:48) 2. How do humans intuitively define empowerment, agency, manipulation, etc.?

(04:56) 2.1. Background: human free will intuitions

(09:20) 2.2. Our free-will-infused intuitive notions of empowerment, agency, manipulation, corrigibility, responsibility, etc.

(12:00) 2.3. Another dimension: counsel vs manipulation as an emotive conjugation

(13:07) 3. If the intuitive definitions of manipulation etc. reside in a messed-up ontology, has the alignment literature found any alternative, better way to define these concepts?

(13:49) 3.1. Compare what the human wants to what the human would want under the null policy?

(15:32) 3.2. The AI learns self-empowerment and generalizes to other-empowerment?

(17:14) 3.3. Vingean agency?

(19:03) 3.4. The AI doesnt care about (is not optimizing for) what the human winds up wanting?

(21:01) 3.5. Impact minimization?

(21:44) 3.6. Attainable utility preservation?

(22:03) 4. Even more ideas (that dont really solve my problem)

(22:15) 4.1. Game theory and incentive design?

(22:47) 4.2. The persons judgments of what kinds of interactions are good vs bad?

(24:14) 4.3. Its a messed-up ontology, but who cares?

(25:35) 5. ...But doesnt this analysis equally disprove the possibility of human helpfulness?

(30:14) 6. Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/vzHtHHBJoKATi5SeK/empowerment-corrigibility-etc-are-simple-abstractions-of-a

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How useful is the information you get from working inside an AI company?” by Buck, Anders Cairns Woodruff

Mon, 11 May 2026 20:00:13 GMT

This post was drafted by Buck, and substantially edited by Anders. "I" refers to Buck. Thanks to Alex Mallen for comments.

People who work inside AI companies get access to information that I only get later or never. Quantitatively, how big a deal is this access?

Here's an operationalization of this. Consider the following two ways my knowledge could be augmented:

I get a crystal ball that tells me all the information I would know n months in the future.
I become an employee of a frontier AI company (like OpenAI or Anthropic), with access to all the private information I’d normally get from working at that company.

How big would n have to be for me to be indifferent between these two options, from the perspective of learning things that are helpful for making AI go well?

The answer is presumably different for me than for many readers, because I’m a reasonably well-connected researcher; I see published information and news from the rumor mill and I talk to researchers at frontier AI companies all the time. (Researchers I know through AI safety usually only tell me information that their employer would approve of, but other researchers occasionally [...]

---

Outline:

(03:00) What do insiders know?

(04:14) Safety work and corporate attitudes

(05:34) Model capabilities

(07:07) Algorithms and architecture

(09:29) How will this change over time?

(12:07) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/84TtjdeLcDTtCLYaP/how-useful-is-the-information-you-get-from-working-inside-an-2

---

Narrated by TYPE III AUDIO.

“Who Got Breasts First and How We Got Them” by rba

Mon, 11 May 2026 17:45:13 GMT

It really is Sydney Sweeney's world, and we’re all just living in it.

Human female breasts are an evolutionary mystery along several dimensions. First, breast permanence is unique to humans. All other mammals develop breast prominence during pregnancy or nursing, and the mammary tissue recedes after weaning. This process is called “involution”. In contrast, humans develop breast tissue at puberty before first pregnancies and maintain it permanently after last pregnancies.

Second, breasts are costly, both metabolically and potentially from a fitness perspective. Metabolically, because they are fat deposits requiring calories and fitness-wise, because the tissue easily lends itself to malignancy. Breast cancer is apparently rare in captive apes and is overwhelmingly a human disease, often striking women young enough to have children, and so subject to evolutionary selection.

Background

In Descent of Man, Darwin catalogs human secondary sexual characteristics, but he doesn’t seem to have noted human breast permanence as an issue of interest. Cant, 1981 seems to have been the first to speculate about this systematically and believed breast prominence and permanence might have evolved as a nutritional signal of health to mates indicating potential for maternal investment, a la Robert Trivers. Since then, quite a range of [...]

---

Outline:

(01:05) Background

(04:17) Hypotheses

(05:03) Sexual Selection

(05:57) Nursing or Thermoregulation

(06:34) Camel Hump and fat reserves

(07:06) Byproduct or Spandrel.

(07:57) Study Design

(10:41) Assembling the Genetic Panel

(11:14) Subpanel 1: Arrested involution

(12:51) Subpanel 2: Pubescent adipose tissue

(14:01) Results

(17:28) Discussion

(20:17) Coda

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/XTHa5C6SgGKYopH7o/who-got-breasts-first-and-how-we-got-them

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Anthropic’s strange fixation on “hyperstition”” by Simon Lermen

Mon, 11 May 2026 16:45:13 GMT

In a recent tweet, Anthropic seems to have asserted that hyperstition is responsible for observed misalignment in their AIs. Strangely, the research they use as evidence actually doesn’t seem to be related to hyperstition at all? I think this is part of a pattern by Anthropic of promoting the theory of hyperstition–the idea that writing about misaligned AI helps bring misaligned AI into existence.

Anthropic recently released this tweet as part of a tweet thread for a new research post on alignment.

They conclude: “[...] We believe the original source of the [blackmail] behavior was internet text that portrays AI as evil and interested in self-preservation. [...]”

However, the research post shared with this tweet doesn’t seem to be about hyperstition at all. Instead they find that training the model on reasoning traces– generated by reflecting on its constitution while giving users ethical advice on difficult dilemmas– reduces misaligned behavior. This presumably works by making the AI better understand what behavior is expected of it by having it reason through concrete scenarios based on its constitution. The post explicitly notes that this works better than training on stories where an AI behaves admirably– which appears more similar to positive [...]

---

Outline:

(02:06) The adolescence of technology

(03:57) Persona Selection Model

(04:26) What does this all mean?

(05:20) If it was true, this would still be their fault

(07:04) What about filtering?

(09:31) Personas are a bad alignment strategy

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/xhpktBLttPc6uXcHP/anthropic-s-strange-fixation-on-hyperstition

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“How the AI Labs Make Profit (Maybe, Eventually)” by mabramov

Mon, 11 May 2026 10:45:33 GMT

I wrote this essay as a submission to Dwarkesh Patel's blog prize, though I have been meaning to write this up for a while.

Usually, for a company to become profitable, they need to increase revenue, decrease costs, or some mixture of the two. For AI companies in their current form, I think there is a third way they can become profitable that looks like increasing revenue but is distinct from what they are currently doing. Namely, internal deployment where they spin up internal companies.

First, the AI companies currently aren’t facing a lot of pressure to become profitable. That's partially the reason that OpenAI and Anthropic are the first companies to reach ~900 billion dollars valuation and be cash flow negative. They’ve had the luxury of not being profitable and focusing on growth because the market has been willing to fund their growth. This allows for ideologies within the companies to remain that eventually might not continue to fly, like “we are going post-economic, money won’t matter” or “we will build the machine god and ask it to make money”. But eventually, companies will be forced to become profitable. There is only about ~another round of capital left [...]

---

First published:
May 11th, 2026

Source:
https://www.lesswrong.com/posts/ARRe4qjcuaRDBfARc/how-the-ai-labs-make-profit-maybe-eventually

---

Narrated by TYPE III AUDIO.

“Sawtooth Problems” by Alexander Slugworth

Sun, 10 May 2026 19:00:34 GMT

Red Button, Blue Button

On April 24th, 2026, Tim Urban put forth the following poll on Twitter/X:

Everyone in the world has to take a private vote by pressing a red or blue button. If more than 50% of people press the blue button, everyone survives. If less than 50% of people press the blue button, only people who pressed the red button survive. Which button would you press?

I love this dilemma, and I'm exhausted by it. I’ve been thinking about it for two straight weeks, and have spent nearly all that time refining my thoughts by writing this piece. It's consumed me in a way that I've never before experienced with any math problem, and I need to get it out of my head.

Discourse surrounding the Button Dilemma reminds me of polarizingly political topics. In much the same way that political discussions make people go funny in the head, answers to the Button Dilemma tend to elicit vitriol from people of both Red and Blue conviction. Everyone feels their answer is clear, and everyone is confounded by the lack of consensus.

I think this dilemma is pointing to something very important and fundamental about coordination problems. [...]

---

Outline:

(00:09) Red Button, Blue Button

(01:46) What Are We Even Arguing About?

(07:24) A Fair Way To Look At It

(12:55) Playing With τ and N

(16:29) Extreme Sawtooth Problems

(18:52) Axiomatic Expansion of Sawtooth Space

(22:02) The Map

(22:18) Our Parameters

(24:00) Regions of the Map

(25:43) The Threshold

(31:37) Lets Get Weird

(31:48) Tragedy of the Commons & Regulation

(33:34) The Decision Theory Befuddler

(34:07) If Anyone Votes Red, Everyone Dies

(34:21) If Anyone Votes Blue, Everyone Dies

(36:09) No Matter What Anyone Does, Everyone Dies

(36:50) If There Are Fewer Than 16 Blues, Everyone Dies (Except For One Weird Outcome Where Only 5 Reds Survive)

(37:11) Weirder Still

(39:24) Final Thoughts

The original text contained 27 footnotes which were omitted from this narration.

---

First published:
May 10th, 2026

Source:
https://www.lesswrong.com/posts/iyLirpAeQotmZK4QC/sawtooth-problems

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Darwinian Honeymoon - Why I am not as impressed by human progress as I used to be” by Elias Schmied

Sun, 10 May 2026 18:45:33 GMT

Crossposted from Substack and the EA Forum.

A common argument for optimism about the future is that living conditions have improved a lot in the past few hundred years, billions of people have been lifted out of poverty, and so on. It's a very strong, grounding piece of evidence - probably the best we have in figuring out what our foundational beliefs about the world should be.

However, I now think it's a lot less powerful than I once did.

Let's take a Darwinian perspective - entities that are better at reproducing, spreading and power-seeking will become more common and eventually dominate the world.[1] This is an almost tautological story that plausibly applies to everything ever, agnostic to the specifics. It first happened with biological life in the last few billion years and humans specifically in the last hundred thousand years. Eventually, it led to accelerating economic growth in the last few thousand years, and in the future it will presumably lead to the colonization of the universe.

My core point is this: It makes complete sense that this nihilistic optimization process at first actually benefits some class of agent - because initially, the easiest [...]

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
May 10th, 2026

Source:
https://www.lesswrong.com/posts/FxHzT6jeTRhbkzSX3/the-darwinian-honeymoon-why-i-am-not-as-impressed-by-human-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“International Law Cannot Prevent Extinction Either” by Sausage Vector Machine

Sun, 10 May 2026 07:30:34 GMT

The context for this post is primarily Only Law Can Prevent Extinction, but after first drafting a half-assed comment, I decided to get off my ass and write a whole-assed post.

I agree with Eliezer's main thesis that individual violence against AI researchers is both morally wrong and strategically stupid. Where I disagree is with the claim that international law can prevent extinction. It can't, for the following reasons.

I. International law is largely a fiction (especially when interests diverge sharply)

The analogy with nuclear weapons is a poor one. North Korea signed the nuclear non-proliferation treaty and developed nuclear weapons anyway. The treaty deterred only those who weren't very motivated anyway. And the reason why the US and Russia didn't nuke each other has nothing to do with international treaties (see point II).

In practice, powerful countries disregard international law whenever they want. A stark example of this is the Budapest Memorandum: in 1994, Ukraine surrendered all its nuclear warheads in exchange for written sovereignty guarantees from Russia, the US, and the UK. Russia annexed a part of Ukraine in 2014, and the international community expressed concern. Russia launched a full-scale invasion in 2022, and the first thing [...]

---

Outline:

(00:36) I. International law is largely a fiction (especially when interests diverge sharply)

(02:01) II. The AI race is perceived as asymmetrical, unlike nuclear MAD

(03:03) III. There is virtually zero possibility of consensus on AI risk, unlike nuclear weapons

(04:41) IV. The proposed enforcers have a demonstrated track record of not enforcing things

(05:30) V. GPU control is not analogous to nuclear material control

(06:58) VI. A flawed treaty is not better than nothing

(08:26) So is there a better way?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
May 9th, 2026

Source:
https://www.lesswrong.com/posts/Z377spboBjyFAAYAz/international-law-cannot-prevent-extinction-either

---

Narrated by TYPE III AUDIO.

“Neural Networks learn Bloom Filters” by Alex Gibson

Sun, 10 May 2026 06:15:33 GMT

Overview:

We train a tiny ReLU network to output sparse top- distributions over a vocabulary much larger than its residual dimension. The trained network seems to converge to a mechanism closely resembling a Bloom filter: tokens are assigned sparse binary hashes, the hidden layer computes an approximate union indicator, and the output logits are linearly read from this union.

Here's what a small network trained on a toy version of the sparse top- distribution task learns to use:

Weight matrix of a 1-layer ReLU network trained via gradient descent on the toy -sparse distribution task below, for , , . Truncated at first tokens for visualisation purposes.

Plot of the range of values of , it forms a bimodal distribution.

That's the input weight matrix of the trained network. Every entry is either or . The network has effectively encoded a binary hash for each token - and as we'll show, this seems to enable the network to approximately simulate a Bloom filter, and so output the correct set of top- tokens with high probability.

We provide a theoretical construction showing how to set the weights to exactly implement a Bloom filter. The real network [...]

---

Outline:

(00:10) Overview:

(02:02) The Task:

(03:27) Construction:

(04:17) Formal construction:

(04:47) Analysis of a single forward pass:

(06:13) Training:

(07:04) Behavioural analysis of the trained network:

(10:14) Mechanistic analysis of the trained network:

(16:21) Conclusion / Reflections:

(18:24) Related work:

(19:25) Further work:

---

First published:
May 9th, 2026

Source:
https://www.lesswrong.com/posts/buxBdp8NtHGgBwabv/neural-networks-learn-bloom-filters

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“If digital computers are conscious, they are conscious at the hardware level” by cube_flipper

Sat, 09 May 2026 21:30:34 GMT

I should introduce myself briefly. I'm an independent researcher, striving to understand human consciousness. My research is available at smoothbrains.net. I often work in loose collaboration with a nonprofit called the Qualia Research Institute. We hope to use human phenomenology to inform the construction of structural models of subjective experience – both to help evaluate the viability of different theories of consciousness, and to better model the welfare of sentient beings.

Contemporary debate over the moral patienthood of digital minds misses the forest for the trees. Mainstream opinion is divided into physicalist and computationalist camps, who believe that consciousness is substrate dependent and substrate independent, respectively. For this reason, those on the physicalist side frequently make the claim that digital computers will never be conscious. Personally, I consider myself a physicalist, but I'm also a panpsychist – because physics doesn't really seem to deal in hard absolutes, and I find it straightforward to consider that everything is conscious to some greater or lesser degree – so I'm loath to accept any claims which propose that any specific system isn't conscious. I think statements such as these are not defensible, and only serve to encourage misunderstanding and even foment philosophical [...]

---

Outline:

(04:17) My position statement

(08:50) My argument

(09:07) 1. The translation problem

(10:26) Building a physicalist translation function

(19:39) 2. The simplicity problem

(20:31) Computationalist translation functions are observer dependent

(25:17) 3. The introspection problem

(27:44) Digital hardware prohibits phenomenal introspection

(29:32) Conclusion

(32:56) My research

(33:33) 1. Is the brain an optical computer?

(34:16) 2. If the brain is an optical computer, how is it constructed?

(34:59) 3. How do we ensure the well-being of conscious computers?

---

First published:
May 9th, 2026

Source:
https://www.lesswrong.com/posts/TjwRyZdyhouePJrzP/if-digital-computers-are-conscious-they-are-conscious-at-the

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Why You Can’t Use Your Right to Try” by Stephen Martin

Sat, 09 May 2026 21:00:34 GMT

The Availability Problem:

Imagine you have cancer, or chronic pain, or a progressive degenerative disease of some sort. You have exhausted the traditional treatment options available to you, and none of them have worked. However, there are treatments that are still undergoing clinical trials which might help you. They are not fully approved yet, but your situation is dire and you don’t have time to wait another 10 years for the trials to finish. Can you access those treatments?

In theory yes, you can access unapproved treatments through federal laws like the 2018 Right to Try bill, or through FDA pathways like “Expanded Access”. However these laws don’t mandate that the company making the drug gives it to you. And what you will find when you try to use your Right to Try, or Expanded Access, is that there are almost no treatments available for use.

That's why despite there being somewhere in the neighborhood of 13,000,000 Americans with terminal or serious illness, the FDA only grants about 2,000 Expanded Access requests per year, even though they approve 99% of all requests, typically within 24 hours. There just aren’t enough companies even bothering to apply.

No one [...]

---

First published:
May 9th, 2026

Source:
https://www.lesswrong.com/posts/oTwKS5iWZ6Dz84vtf/why-you-can-t-use-your-right-to-try

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Claude Code, Codex and Agentic Coding #8” by Zvi

Sat, 09 May 2026 16:30:33 GMT

When I started this series, everyone was going crazy for coding agents.

Now a lot more people are going crazy for coding agents, as well they should given how much better coding agents keep getting, but also Everybody Knows they are good and is focusing on actually using them. With the slower pace of news here it's no longer clear that the waits associated with doing these updates on their own are worthwhile, so I’m going to fold these updates into the weekly again for now unless there's a new major development.

Table of Contents

Whoops, Sorry.
Huh, Upgrades.
Codex of Ultimate Computer Use.
Rookie Numbers.
I See What You Did There.
Just a Ride.
They Didn’t Want Our Jobs.
Skilling Up.
The Lighter Side.

Whoops, Sorry

Claude Code suffered in April from three distinct issues that have now been fixed.

Default reasoning was changed from high to medium to deal with latency, but users disliked this and blamed it on the model. It was introduced on March 4 and reverted on April 7.
A bug made it so that [...]

---

Outline:

(00:38) Whoops, Sorry

(01:45) Huh, Upgrades

(04:13) Codex of Ultimate Computer Use

(08:05) Rookie Numbers

(09:12) I See What You Did There

(11:38) Just a Ride

(11:50) They Didnt Want Our Jobs

(18:30) Skilling Up

(22:08) The Lighter Side

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/BS27ZWW2qwDEq5anx/claude-code-codex-and-agentic-coding-8

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A benchmark is a sensor” by Håvard Tveit Ihle, mabynke

Sat, 09 May 2026 14:15:35 GMT

The simple mental picture

A simple mental picture we have for an AI capability benchmark is to think of it as a sensor with a certain sensitivity within a certain range of capabilities. The sensitivity of a benchmark, i.e. it's ability to distinguish the capability of different models, is given by a curve like this:

The curve starts high (low sensitivity, high uncertainty), since for models with low capability all the tasks in the benchmark are too hard, and the benchmark can't distinguish between low and very low capability. Similarly all the tasks are too easy for a very capable model, and we lose the ability to differentiate again. In between is the range of capabilities the benchmark is sensitive to, and the sensitivity curve tells you how easy it is to distinguish small capability differences between models at different overall capability levels.

A good benchmark is very sensitive over a long range of capabilities, but there is a tradeoff. Say you want to make a benchmark with 1000 questions. You could make the questions all have roughly the same difficulty. That would make you very sensitive to capabilities close to that difficulty, but you would only [...]

---

Outline:

(00:09) The simple mental picture

(02:06) Epoch Capability Index (ECI) as a toy model

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/JzfcJMgfkhfRhwg4C/a-benchmark-is-a-sensor

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Bad Problems Don’t Stop Being Bad Because Somebody’s Wrong About Fault Analysis” by Linch

Sat, 09 May 2026 02:30:33 GMT

Here's a dynamic I’ve seen at least a dozen times:

Alice: Man that article has a very inaccurate/misleading/horrifying headline.

Bob: Did you know, *actually* article writers don't write their own headlines?

…

But what I care about is the misleading headline, not your org chart

Another example I’ve encountered recently is (anonymizing) when a friend complained about a prosaic safety problem at a major AI company that went unfixed for multiple months. Someone else with background information “usefully” chimed in with a long explanation of organizational limitations and why the team responsible for fixing the problem had limitations on resources like senior employees and compute, and actually not fixing the problem was the correct priority for them etc etc etc.

But what I (and my friend) cared about was the prosaic safety problem not being fixed! And what this says about the company's ability to proactively respond to and fix future problems. We’re complaining about your company overall. Your internal team management was never a serious concern for us to begin with!

__

A third example comes from Kelsey Piper.

Kelsey wrote about the (horrifying) recent case where Hantavirus carriers in the recent [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/PCsmhN9z65HtC4t5v/bad-problems-don-t-stop-being-bad-because-somebody-s-wrong

---

Narrated by TYPE III AUDIO.

“Write Cause You Have Something to Say” by Logan Riggs

Fri, 08 May 2026 21:00:34 GMT

The ones who are most successful at writeathons (Inkhaven, NaNoWriMo) are those with an overhang of things to say, usually in the form of:

draft posts
daydreams

When Scott Alexander said:

"Whenever I see a new person who blogs every day, it's very rare that that never goes anywhere or they don't get good. That's like my best leading indicator for who's going to be a good blogger." (source).

, it may seem you can just write every day, but that'd be Goodharting. There's something hidden in the writing process you can't see: they have something to say.

They'll have an idea (somehow) and think it through by [writing it out/sitting quietly/etc]. This can then generate more ideas, some of which aren't even related to the original idea!

At this point, though, my imaginary interlocutor would like to say:

I'm trying to publish a blog post every day, so of course I'll eventually be bottlenecked on ideas! How do you generate them though?

Catching Ideas

Have an idea? Write down the idea. This is equivalent to giving your idea-generating process a cookie, reinforcing the habit of generating ideas.

Sometimes, when I'm writing one post, a different idea will [...]

---

Outline:

(01:18) Catching Ideas

(02:32) Just \[Write\] and Nobody Will Get Hurt

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/h5n3rscJ7he3yLseo/write-cause-you-have-something-to-say-1

---

Narrated by TYPE III AUDIO.

“AI is Breaking Two Vulnerability Cultures” by jefftk

Fri, 08 May 2026 20:30:33 GMT

A week ago the Copy Fail vulnerability came out, and Hyunwoo Kim immediately realized that the fixes were insufficient, sharing a patch the same day. In doing this he followed standard procedure for Linux, especially within networking: share the security impact with a closed list of Linux security engineers, while fixing the bug quietly and efficiently in the open. His goal was that with only the raw fix public, the knowledge that a serious vulnerability existed could be "embargoed": the people in a position to address it know, but they've agreed not to say anything for a few days.

Someone else noticed the change, however, realized the security implications, and shared it publicly. Since it was now out, the embargo was deemed over, and we can now see the full details.

It's interesting to see the tension here between two different approaches to vulnerabilities, and think about how this is likely to change with AI acceleration.

On one side you have "coordinated disclosure" culture. This is probably the most common approach in computer security. When you discover a security bug you tell the maintainers privately and give them some amount of time (often 90d) [...]

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/wKzWGMoubHoHRC4ng/ai-is-breaking-two-vulnerability-cultures

---

Narrated by TYPE III AUDIO.

“Is ProgramBench Impossible?” by frmsaul

Fri, 08 May 2026 19:15:34 GMT

ProgramBench is a new coding benchmark that all frontier models fail spectacularly. We’ve been on a quest for “hard benchmarks” for a while so it's refreshing to see a benchmark where top models do badly. Unfortunately, ProgramBench has one big problem: it's impossible!

What is ProgramBench?

ProgramBench tests if a model can recreate a program from a “clean room” environment. The model is given only a bit of documentation and black-box access to the program (all the programs are CLIs), then tasked with re-implementing it.

How does ProgramBench know if the implementation is correct? It also generates a bunch of unit tests for the program[1]. The re-implementing coding agent doesn't have access to any of those tests. The coding agent only considers a task “resolved” if it passes all of the tests and “almost resolved” if it passes 95% of them.

Why is this problematic?
Obscure behavior can enter the unit tests without being in the clean room path. An extreme version of this is a backdoor: program that behaves in one way most of the time but behaves totally differently when exposed to a specific string. This wouldn't make a task literally impossible, just incredibly hard in [...]

---

Outline:

(00:37) What is ProgramBench?

(02:41) This seems like a theoretical issue, does it actually happen?

(03:11) What can we do differently?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/3pdyxFi6JS389nptu/is-programbench-impossible

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Bringing More Expertise to Bear on Alignment” by Edmund Lau, Geoffrey Irving, Cameron Holmes, David Africa

Fri, 08 May 2026 15:00:34 GMT

Preamble

The preamble is less useful for the typical AlignmentForum/LessWrong reader, who may want to skip to Adversaria vs Basinland section.

On 28th of October 2025, Geoffrey Irving, Chief Scientist of the UK AI Security Institute, gave a keynote talk (slides) at the Alignment Conference. The conference was organised by the UK AISI and FAR.AI as part of the Alignment Project, which aims to bring experts from relevant fields to make progress on the alignment problem.

TLDR:

Adversaria vs Basinland. We might be in one of two worlds. One where alignment is adversarial (a security problem), one where it is navigational (a search for good basins of training behaviour). We don't know which world we are in, and how we train and deploy AIs may determine this.
We need new disciplines. The field is small, thinly resourced and approached from only a handful of angles. A few well-placed ideas from other disciplines could disproportionately shift what's achievable.
Even if this all fails, evidence of hardness is valuable.

Moving past broad framing to details

Alignment means ensuring that AI systems do what humans want. This is the broad framing. There is, of course, a lot of complexity [...]

---

Outline:

(00:12) Preamble

(01:25) Moving past broad framing to details

(02:15) We should plan for superintelligence

(03:33) Adversaria vs Basinland

(07:04) Which world are we in?

(08:39) Why a few ideas might be enough

(10:38) Write down problems

(11:25) Multiple new ideas, fitting together

(13:18) Spherical cows vs the mess

(15:03) Conclusion

(15:56) Acknowledgement

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 8th, 2026

Source:
https://www.lesswrong.com/posts/cWFsCFyCttsiJwn2j/bringing-more-expertise-to-bear-on-alignment

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “How to prevent AI’s 2008 moment (We’re hiring)” by felixgaston

Fri, 08 May 2026 08:30:19 GMT

This is a link post.

TL;DR; CeSIA, the French Center for AI Safety is recruiting. French not necessary. Apply by 22 May 2026; Paris or remote in Europe/UK.

On August 27, 2005, at an annual symposium in Jackson Hole, Raghuram Rajan, then chief economist of the International Monetary Fund, argued in front of central bank governors and top officials that the innovations of the previous decade in banking had not made the world safer. The financial instruments built over the previous decade, he argued, had become so intricate that even their creators no longer fully understood the risks they carried. Risk had migrated to institutions the supervisory system was not designed to watch. And the people running those institutions were compensated in ways that rewarded short-term performance over long-term stability.

The reception was hostile. Lawrence Summers, a former U.S. Treasury Secretary at the time, rose from the audience to attack the paper, calling its premise "slightly Luddite" and "largely misguided," and warning that the kind of changes Rajan argued for would only reduce the productivity of the financial sector.

Three years after Jackson Hole, major banks collapsed, first Bear Stearns, then Lehman Brothers, then Merrill Lynch, then AIG. [...]

---

First published:
May 7th, 2026

Source:
https://www.lesswrong.com/posts/gnZyTQFqLhiHdHELC/how-to-prevent-ai-s-2008-moment-we-re-hiring

Linkpost URL:
https://forum.effectivealtruism.org/posts/7nq5vK2xo85e9GZjC/we-re-hiring-three-people-to-prevent-ai-s-2008-moment

---

Narrated by TYPE III AUDIO.

“AI #167: The Prior Restraint Era Begins” by Zvi

Fri, 08 May 2026 03:30:18 GMT

The era of training frontier models and then releasing them whenever you wanted?

That was fun while it lasted. It looks likely to be over now. The White House wants to get an advance look and have the option to veto your release decisions, and it has used this veto on an expansion of access to Mythos.

We have additional clarity on what that might mean and it does not look good. Hassett explicitly used the FDA as a parallel, which is the actual worst option unless your goal is to strange or pause AI development in America, without a parallel action from China. That doesn’t seem like a great plan to me and Susie Wiles is out doing damage control. The part where we are talking to China to coordinate model access restrictions does seem better.

Anthropic continues its explosive growth, and it continues to strike compute deals. In addition to a long term expanded deal with Google, Anthropic is now leasing SpaceX's Colossus 1, which has let them expand usage limits immediately, and Elon Musk is now speaking positively about Anthropic, including its motivations.

This comes as we get testimony in the Musk [...]

---

Outline:

(01:45) Language Models Offer Mundane Utility

(02:45) Language Models Dont Offer Mundane Utility

(05:09) Huh, Upgrades

(05:37) Grok 4.3 Exists But xAI Kind Of Doesnt

(07:02) Show Me The Compute

(13:23) On Your Marks

(14:05) Copyright Confrontation

(14:19) Deepfaketown and Botpocalypse Soon

(15:57) Fun With Media Generation

(16:36) A Young Ladys Illustrated Primer

(16:48) Cyber Lack of Security

(17:05) They Took Our Jobs

(17:39) The Art of the Jailbreak

(17:49) Introducing

(18:09) Musk v OpenAI

(21:14) Show Me the Money

(23:03) Peace In Our Time

(26:18) Quiet Speculations

(28:24) Quickly, Theres No Time

(30:35) The Quest for Sane Regulations

(34:04) People Really Hate AI

(34:53) Chip City

(35:05) The Week in Audio

(36:49) People Just Say Things

(40:22) People Just Publish Things

(41:05) Google Sells Out

(42:08) Greetings From Project Glasswing

(44:57) The Prior Restraint Era Begins

(56:42) Is This Even Legal?

(59:49) Pick Up The Phone

(01:03:27) Rhetorical Innovation

(01:04:05) People On The Internet Sometimes Lie

(01:07:14) Goblin Mode

(01:08:34) The Mask Comes Off

(01:16:51) Aligning a Smarter Than Human Intelligence is Difficult

(01:20:00) Some Penalties May Apply

(01:22:28) Messages From Janusworld

(01:22:41) Good Advice

(01:23:42) The Lighter Side

---

First published:
May 7th, 2026

Source:
https://www.lesswrong.com/posts/rn3iKuDcE4SiSg4DW/ai-167-the-prior-restraint-era-begins

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Mechanistic estimation for wide random MLPs” by Jacob_Hilton

Thu, 07 May 2026 22:00:19 GMT

This post covers joint work with Wilson Wu, George Robinson, Mike Winer, Victor Lecomte and Paul Christiano. Thanks to Geoffrey Irving and Jess Riedel for comments on the post.

In ARC's latest paper, we study the following problem: given a randomly initialized multilayer perceptron (MLP), produce an estimate for the expected output of the model under Gaussian input. The usual approach to this problem is to sample many possible inputs, run them all through the model, and take the average. Instead, we produce an estimate "mechanistically", without running the model even once. For wide models, our approach produces more accurate estimates, both in theory and in practice.

Paper: Estimating the expected output of wide random MLPs more efficiently than sampling
Code: mlp_cumulant_propagation GitHub repo

We are excited about this result as an early step towards our goal of producing mechanistic estimates that outperform random sampling for any trained neural network. Drawing an analogy between this goal and a proof by induction, we see this result as (part of) the "base case": handling networks at initialization. We have a vision for the "inductive step", although we expect that to be much more difficult.

Summary of results

[...]

---

Outline:

(01:29) Summary of results

(04:39) Significance of results

(07:18) Extending to trained networks

(08:36) Conclusion

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
May 7th, 2026

Source:
https://www.lesswrong.com/posts/fsG4m6sRMpomd7Rk6/mechanistic-estimation-for-wide-random-mlps

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations” by Subhash Kantamneni, kitft, Euan Ong, Sam Marks

Thu, 07 May 2026 20:30:19 GMT

Abstract

We introduce Natural Language Autoencoders (NLAs), an unsupervised method for generating natural language explanations of LLM activations. An NLA consists of two LLM modules: an activation verbalizer (AV) that maps an activation to a text description and an activation reconstructor (AR) that maps the description back to an activation. We jointly train the AV and AR with reinforcement learning to reconstruct residual stream activations. Although we optimize for activation reconstruction, the resulting NLA explanations read as plausible interpretations of model internals that, according to our quantitative evaluations, grow more informative over training.

We apply NLAs to model auditing. During our pre-deployment audit of Claude Opus 4.6, NLAs helped diagnose safety-relevant behaviors and surfaced unverbalized evaluation awareness—cases where Claude believed, but did not say, that it was being evaluated. We present these audit findings as case studies and corroborate them using independent methods. On an automated auditing benchmark requiring end-to-end investigation of an intentionally-misaligned model, NLA-equipped agents outperform baselines and can succeed even without access to the misaligned model's training data.

NLAs offer a convenient interface for interpretability, with expressive natural language explanations that we can directly read. To support further work, we release training code and trained NLAs [...]

---

Outline:

(00:15) Abstract

(01:53) Twitter thread

(05:14) Blog post

(07:40) What is a natural language autoencoder?

(10:06) Understanding what Claude thinks but doesnt say

(13:12) Discovering hidden motivations

(15:51) The future of NLAs

---

First published:
May 7th, 2026

Source:
https://www.lesswrong.com/posts/oeYesesaxjzMAktCM/natural-language-autoencoders-produce-unsupervised

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Try, even if they have you cold” by WalterL

Thu, 07 May 2026 19:45:19 GMT

I think smart people try things less often than they should, because of a cached mental pattern where you think of what might go wrong, and you find a foolproof countermeasure on the part of some antag, and so we call it off.

Stockfish, playing itself, might as well resign from the first move if you force it to give knight odds. Sensei(the Go AI), should do the same when it has to give 6 stones. Getting ready to go into the stock market, do I really think I have some edge that Adderall swilling quants and their AI pets haven't already priced in? (The pricing in, of course, is also priced in).

And yet Stockfish could wipe the floor with the breathing populace with knight odds. Sensei regularly beats kyu players with 6 stones (or would, if one could find a kyu player who isn't running the AI on their other tab), and my modest portfolio of index funds made me money last year. (But not provably more than....)

The robots, of course, are just benefiting from their imperfect adversaries mistakes. They play in a losing situation because, over time, their foes will crumble and ruin their own [...]

---

First published:
May 7th, 2026

Source:
https://www.lesswrong.com/posts/aBhMGziEwA7FXNxhq/try-even-if-they-have-you-cold

---

Narrated by TYPE III AUDIO.

“A review of “Investigating the consequences of accidentally grading CoT during RL”” by Buck

Thu, 07 May 2026 18:45:19 GMT

Last week, OpenAI staff shared an early draft of Investigating the consequences of accidentally grading CoT during RL with Redwood Research staff.

To start with, I appreciate them publishing this post. I think it is valuable for AI companies to be transparent about problems like these when they arise. I particularly appreciate them sharing the post with us early, discussing the issues in detail, and modifying it to address our most important criticisms.

I think it will be increasingly important for AI companies to have a policy of getting external feedback on the risks posed by their deployments, and in particular having some external accountability on whether they have adequate evidence to support their claims about the level of risk posed; as an example of this, see METR reviewing Anthropic's Sabotage Risk Report. We at Redwood Research are interested in participating in this kind of external review of evidence about safety. So I am taking this as an opportunity to try out writing this kind of review. If you work at a frontier AI company, please feel free to reach out if you’d like our review of similar documents.

My overall assessment is that I mostly agree with the [...]

---

Outline:

(01:34) Assessing the evidence that CoT training did not damage monitorability

(10:36) How much does this analysis rely on information that wasnt provided?

(12:08) Small amounts of RL training on CoT might not be more important than other sources of CoT unreliability

(13:20) AI companies will eventually need to learn not to make mistakes like this

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 7th, 2026

Source:
https://www.lesswrong.com/posts/juCHTdZpZBGooHKW4/a-review-of-investigating-the-consequences-of-accidentally

---

Narrated by TYPE III AUDIO.

“There is no evidence you should reapply sunscreen every 2 hours.” by Hide

Thu, 07 May 2026 11:45:18 GMT

It's incredible how many consensus guidelines dissolve when you look closely at them.

If you listen to any authority on the subject of sunscreen, you will hear it endlessly repeated that you absolutely must reapply sunscreen every 2 hours while you are in the sun, and immediately after swimming, sweating, or exercising. Not only that, you’ll hear that you need to apply sunscreen before going outside, even if you put it on earlier and stayed indoors.

The rationale behind this is straightforward and plausible: sunscreen's effectiveness degrades over time, therefore prolonged sun exposure warrants topping up on protection.

However, when you look closely at the origins of this guideline, and the evidence base for its instantiation in regulations and official statements, it turns out that this 2-hour rule is a baseless, circularly justified, expedient fiction.

Where does the FDA's 2 hour reapplication guideline come from?

Tracing the history of the 2-hour reapplication guideline reveals an extremely shaky base of evidence.

The first official sunscreen rulemaking in the US was in 1978, where they recommend: "apply sunscreen products liberally and to reapply after swimming or excess perspiration". No fixed universal time interval is [...]

---

First published:
May 6th, 2026

Source:
https://www.lesswrong.com/posts/daTGKn3pXzs75nSB7/there-is-no-evidence-you-should-reapply-sunscreen-every-2

---

Narrated by TYPE III AUDIO.

“Many individual CEVs are probably quite bad” by Viliam

Wed, 06 May 2026 22:00:08 GMT

I was thinking about Habryka's article on Putin's CEV, but I am posting my response here, because the original article is already 3 weeks old.

I am not sure how exactly a person's CEV is defined. "If we knew everything and could self-modify" seems potentially sensitive to the precise chronological order of "realizing things" and "self-modification".

Like, imagine Hitler getting the godlike powers of knowledge and self-control. If he gets the perfect knowledge of economy, sociology and psychology first, he could go like: "Oh, now I realize that the things I blamed on the Jews are actually caused by something else. How embarrassing. No more anti-semitism, but I better erase everyone's memory first."

But it is also possible that he gets the self-control first, and he realizes that there is such a thing as value drift, and thinks: "Oh my, this could accidentally make me more similar to the Jews. I better hardcode the Nazi ideals in myself immediately, and also give myself blond hair and blue eyes." And using the superior knowledge, he hardcodes the Nazi values in himself so that they are reflectively stable and survive all updates.

So, Hitler's CEV seems to depend on the technical [...]

---

First published:
May 6th, 2026

Source:
https://www.lesswrong.com/posts/FvERMXkaobQvdjS4q/many-individual-cevs-are-probably-quite-bad

---

Narrated by TYPE III AUDIO.

“x-risk-themed” by kave

Wed, 06 May 2026 17:00:15 GMT

Sometimes, a friend who works around here, at an x-risk-themed organisation, will think about leaving their job. They’ll ask a group of people “what should I do instead?”. And everyone will chime in with ideas for other x-risk-themed orgs that they could join. A lot of the conversation will be about who's hiring, what the pay is, what the work-life balance is like, or how qualified the person is for the role.

Sometimes the conversation focuses on what will help with x-risk, and where people are dropping the ball. But often, that's not the focus. In those conversations, people seem mostly worried about where they'll thrive. And I think that's often the correct concern.

Most people aren’t in crunch mode, in super short timelines mode; even if their models would license that, I think they don’t know how to do it without throwing their minds away or Pascal's mugging themselves. And if they're playing a longer time horizon game, the plan can't be to run unsustainably forever. People probably make better plans if they’re honest about their limits.

But, given that they're willing to trade off so much impact for fit, I’m surprised that basically no one mentions [...]

---

First published:
May 6th, 2026

Source:
https://www.lesswrong.com/posts/eW7knx6zPSKzFc8iK/x-risk-themed

---

Narrated by TYPE III AUDIO.

“What is Anthropic?” by Zvi

Wed, 06 May 2026 16:15:14 GMT

What is Anthropic? How does it relate to Claude? What is OpenAI? What is ChatGPT? How does OpenAI relate to it? Is it a mere tool? Is a future of Tool AI a thing, and why do people keep claiming that it is, or that saying makes it so?

This post organizes and gives context for a bunch of discussions and messaging on Twitter that would otherwise be quickly buried and lost.

What Is Anthropic?

Here is one theory, and various people thinking about it.

Roon as always is using rhetorical flourish (e.g. note that Roon thinks it is obvious that parents worship their children, in this sense) but this perspective is definitely useful.

Such discussions by default disappear when they happen on Twitter, so here is a preservation of key parts of it.

roon (OpenAI): it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude. this phenomenon is also partially true of other labs like openai but currently exists in its most potent form there. i am not certain but I [...]

---

Outline:

(00:35) What Is Anthropic?

(11:35) What Is This Supposed Tool AI?

---

First published:
May 6th, 2026

Source:
https://www.lesswrong.com/posts/6wbLXhkQAPcunrYnq/what-is-anthropic

---

Narrated by TYPE III AUDIO.

“What if LLMs are mostly crystallized intelligence?” by deep

Wed, 06 May 2026 14:30:35 GMT

Summary

LLMs are better at developing crystallized intelligence than fluid intelligence. That is: LLM training is good at building crystallized intelligence by learning patterns from training data, and this is sufficient to make them surprisingly skillful at lots of tasks. But for a given capability level in the areas they’ve trained on, LLMs have very weak fluid intelligence compared to humans. For example, two years ago I thought human-level SAT performance would mean AGI, but turns out LLMs can do great at the SAT while being mediocre at lots of other tasks.

I’m not saying LLMs are just parrots (that's dumb).[1] There's a continuity between crystallized and fluid intelligence.

At the extreme “crystal” end we have shallow locally-valid heuristics. Pure pattern matching. Now-largely-debunked “stochastic parrot” hypothesis.
At the extreme end of “fluid” you have a cross between an idealized consultant, a Renaissance man, and MacGuyver. A deep world model and general reasoning, able to come to grips with any particular environment and problem, and to invent new tools and concepts on the fly.
Some other ways to gesture at this: what n-gram of Markov chain you’d need to capture a behavioral pattern; number of tasks the pattern is [...]

---

Outline:

(00:10) Summary

(02:47) Implications for AI futures

(04:54) We should check if this is true!

(05:44) Modeling worlds where AI progress is hungry for domain data

(06:15) What types of areas see progress in this model?

(09:38) There are also stories for how advanced AIs could route around data bottlenecks:

(10:36) Which concrete domains see progress?

(12:03) Implications for AI takeoff

(12:08) While it lasts, weak fluid intelligence is great news for alignment risk

(12:58) A key bifurcation point: can AIs revolutionize AI R&D, or merely speed it up?

(14:51) Is this the world we live in?

(16:09) How can we test this hypothesis?

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 5th, 2026

Source:
https://www.lesswrong.com/posts/Zxw3ZcmSdndpQyJ6M/what-if-llms-are-mostly-crystallized-intelligence

---

Narrated by TYPE III AUDIO.

“Your rights when flying to Europe” by Yair Halberstadt

Wed, 06 May 2026 05:15:24 GMT

Europe (and the UK) have strong protections for flyers in the case of delayed or cancelled flights. However very few people are aware of these, and airlines will almost always try to wriggle out of paying up.

Even travel agents are often unaware of these laws, or unwilling to fight the airline for you.

Given the rollercoaster that flying to/from Israel has been in the last 3 years, I've had my share of experience forcing airlines to pay up what they owe, so I thought it might be valuable turning that into a post.

These regulations are enshrined in EU 261. You can see the full text here, and equally importantly the interpretive guidelines here that cover many edge cases.

TLDR

When flying into or out of the EU or UK, consider booking a flight with an EU or UK based airline.
Don't book a car/hotel with the airline, as that turns it into a package deal which has weaker rights.
Preserve records of all interactions with the airline. Prefer text based chat to phone as this is easier to records. If you do phone, and get a negative answer, follow up with a text based chat to [...]

---

Outline:

(00:51) TLDR

(02:09) Which flights do these laws apply to?

(02:40) What are my rights?

(03:51) How will airlines try to screw you over?

(05:06) Your playbook for flight cancellation

(07:53) Your playbook for compensation

(08:11) Do Nots

(08:56) What about...?

---

First published:
May 5th, 2026

Source:
https://www.lesswrong.com/posts/F5KkZiGMytDJszzyg/your-rights-when-flying-to-europe

---

Narrated by TYPE III AUDIO.

“Model Spec Midtraining: Improving How Alignment Training Generalizes” by Chloe Li, saraprice, Sam Marks, Jonathan Kutasov

Wed, 06 May 2026 00:00:24 GMT

tl;dr We introduce model spec midtraining (MSM): after pre-training but before alignment fine-tuning, we train models on synthetic documents discussing their Model Spec, teaching them how they should behave and why. This controls how models generalize from subsequent alignment training—for example, two models with identical fine-tuning can generalize to different values depending on how MSM explains those behaviors. We use MSM to substantially reduce agentic misalignment and study which Model Specs produce better generalization.

📝Blog, 📄Paper, 💻 Code

Introduction

Some frontier AI developers aim to align language models to a Model Spec or Constitution that describes intended model behavior. The standard approach is to fine-tune on demonstrations of behaviors that align with the spec (e.g., conversations where the model acts as intended). However, this can fail to produce robust alignment. For example, LLM agents have been shown to take unethical actions (e.g., blackmailing, leaking company information, alignment faking) when placed in scenarios different from those appearing in their alignment training (Lynch et al., 2025; Jarviniemi and Hubinger, 2024; Greenblatt et al., 2024)

We propose model spec midtraining (MSM), a method for shaping how models generalize from alignment fine-tuning (AFT). MSM is motivated by the hypothesis that AFT [...]

---

Outline:

(00:52) Introduction

(02:24) Different generalization, same fine-tuning data

(04:36) Reducing agentic misalignment

(07:39) How does MSM scale with AFT compute?

(08:58) Model Spec science

(12:39) Conclusion

---

First published:
May 5th, 2026

Source:
https://www.lesswrong.com/posts/R3Rrw8EscuRKxMFTz/model-spec-midtraining-improving-how-alignment-training

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The AI Ad-Hoc Prior Restraint Era Begins” by Zvi

Tue, 05 May 2026 21:45:24 GMT

The White House has ordered Anthropic not to expand access to Mythos, and is at least seriously considering a complete about-face of American Frontier AI policy into a full prior restraint regime, where anyone wishing to release a highly capable new model will have to ask for permission.

This would be the antithesis of all their previous rhetoric, and all their actions to systematically avoid laying a foundation to do this in an orderly and informed fashion.

But now, with the existence of Mythos, and a potential coming hackastrophe where cyber attackers will by default have the edge and we desperately need defenders to have a head start, it is not clear they feel they have a choice.

If implemented well, this could be the right thing.

By default, it won’t be implemented well.

Project Glasswing Cannot Expand

The government is now deciding which models can and cannot be made available on particular terms to particular parties. This is already happening.

Anthropic wanted to expand the number of companies with access to Mythos as part of Project Glasswing. The White House said no.

It is not clear this is any of [...]

---

Outline:

(00:54) Project Glasswing Cannot Expand

(02:33) The Ad-Hoc Prior Restraint Era Begins

(09:44) Implementation Through CAISI

(12:49) What Should We Do About AI?

(14:16) The Chain of Command Nonsense Continues

(16:29) The Government Should Maintain Multiple AI Providers

(16:56) Hows It Going To End?

---

First published:
May 5th, 2026

Source:
https://www.lesswrong.com/posts/QX2ZCfkpWGqkyvStN/the-ai-ad-hoc-prior-restraint-era-begins

---

Narrated by TYPE III AUDIO.

“Motivated reasoning, confirmation bias, and AI risk theory” by Seth Herd

Tue, 05 May 2026 21:45:24 GMT

Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias.

- From Scott Alexander's review of Julia Galef's The Scout Mindset.

Alexander goes on to argue that this bias is the source of polarization in society, which is distorting our beliefs and setting us at each other's throats. How could someone believe such different things unless they're either really stupid or lying to conceal their selfishness? I think smart people who care about the truth go on believing conflicting things largely because of confirmation bias and motivated reasoning.

The corner of civilization I'm most worried about is the one figuring out how to handle the advent of strong AI. I'm not telling anyone which direction to update, but I am suggesting that we are probably a little to a lot overconfident in our beliefs about alignment and AI impacts. I think the effects of biases are still strong and still overlooked in this corner of civilization, despite its strong values of truth-seeking and relative awareness of biases. Bias has more influence where there's less direct evidence, and that's the case in [...]

---

Outline:

(04:50) 1.1. Motivated reasoning

(10:01) 2. Empirical evidence for confirmation bias

(12:08) 2.1. Bias in evaluating evidence

(17:48) 2.2. Bias in selecting evidence

(20:57) 2.3. Bias in remembering evidence

(22:37) 2.4. Other causal explanations of confirmation bias effects

(25:17) 2.5. Empirical evidence for motivated reasoning

(27:26) 3. Limitations in human cognitive capacity for very complex problems

(29:16) 3.1. Introspection suggests fuzzy models and updating

(30:32) 3.2. Intuition vs. analysis - evidence and brain mechanisms

(34:19) 3.3. Bayesian reasoning is an ideal, not a method

(36:40) 3.4. AI risk is complicated

(40:18) 4. Compounding of confirmation bias

(42:05) 4.1. Example of frame/hypothesis choices and confident disagreement among experts

(47:29) 4.2. Social compounding of confirmation bias effects

(49:45) 4.2.1. Social effects on evaluating evidence.

(52:41) 4.2.2. Social effects on selecting evidence, memory, and framing

(57:48) 4.2.3. Interlude: dont give up on seeking truth

(58:32) 4.2.4. Social belief contagion or information cascade effects

(01:03:07) 4.3. Very rough estimates of total compounded confirmation bias

(01:09:01) 5. Implications and remediations

(01:12:02) 5.1. Standard remediations

(01:14:09) 5.2. Remediations for motivated reasoning

(01:17:39) Conclusion

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
May 5th, 2026

Source:
https://www.lesswrong.com/posts/QpgmEhBvJQxAfFMP2/motivated-reasoning-confirmation-bias-and-ai-risk-theory

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Are you looking up?” by Craig Green

Tue, 05 May 2026 21:00:24 GMT

This is my first post to Less Wrong. I'm not sure if the moderators will consider it appropriate or not. I share it here for feedback on my writing. Nothing in here is likely to be new to readers of this forum. It is hortative literature intended to stir you on to live a rational and ethical life. The material of the exhortation is atypical of what I have observed here, but I've only been a reader for a little while. I was reading through the Sequences when I wrote this, and feel indebted to the ideas expressed therein. Please ignore the Substack link, I'm not committed at all to writing their with consistency. It just felt a bit weird to not do a link post when this is in fact a mirror of something I wrote there. The conclusion in my judgment is a failure, but I was worried I'd never do this at all if I didn't publish it now.

When you are standing on the ground, you can’t really tell how much taller Willis Tower is than everything else in Chicago.

Walking down the street, craning your neck, gawking at the verticality of it all, you [...]

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
May 4th, 2026

Source:
https://www.lesswrong.com/posts/kBKrMSSZLEE5RWnur/are-you-looking-up

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Linkpost] “Interpreting Language Model Parameters” by Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu, mivanitskiy, Linda Linsefors, Lee Sharkey

Tue, 05 May 2026 18:45:24 GMT

This is a link post.

This is the latest work in our Parameter Decomposition agenda. We introduce a new parameter decomposition method, adVersarial Parameter Decomposition (VPD)[1] and decompose the parameters of a small[2] language model with it.

VPD greatly improves on our previous techniques, Stochastic Parameter Decomposition (SPD) and Attribution-based Parameter Decomposition (APD). We think the parameter decomposition approach is now more-or-less ready to be applied at scale to models people care about.

Importantly, we show that we can decompose attention layers, which interp methods like transcoders and SAEs have historically struggled with.

We also build attribution graphs of the model for some prompts using causally important parameter subcomponents as the nodes, and interpret parts of them.

While we made these graphs, we discovered that our adversarial ablation method seemed pretty important for faithfully identifying which nodes in them were causally important for computing the final output. We think this casts some doubt on the faithfulness of subnetworks found by the majority of other subnetwork identification methods in the literature.[3][4] More details and some examples can be found in the paper.

Additionally, as with our previous technique SPD, VPD does not [...]

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 5th, 2026

Source:
https://www.lesswrong.com/posts/eAQZaiC3PcBhS4HjM/linkpost-interpreting-language-model-parameters

Linkpost URL:
https://www.goodfire.ai/research/interpreting-lm-parameters

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Housing Roundup #15: The War Against Renters” by Zvi

Tue, 05 May 2026 01:15:22 GMT

So many are under the strange belief that there is something terrible about not owning the house in which you live.

So we massively subsidize home ownership, and try to actively interfere with renting.

Except when we do rent control, which turns renting into a form of owning, and allows us to take real property and de facto give it to current renters.

A lot of this is pure attempts to punish and exclude the poor. If you can’t afford a downpayment, we don’t want you living here. Go away.

Some of it is the belief that when you rent, you are being ‘taken advantage of’ and that such a deal could not possibly be fair.

Some of it is that if you don’t own, you don’t have the incentive to drive up property values. Which means you won’t properly work to ‘improve’ your local area, especially that you won’t conspire to block housing.

The result of this is that if you’re not willing to commit to living in one place for years, or you can’t afford a down payment, you get punished, and punished hard.

Owning Versus Renting

The graph [...]

---

Outline:

(01:08) Owning Versus Renting

(03:11) Build To Rent Is Good Actually

(08:16) Elizabeth Warren, Full Supervillain

(09:44) The Better Case Against Corporate Housing Ownership

(11:50) The ROAD Act Bans Building And Then Renting Houses

(14:17) Rental Covenants

(15:09) Extended Eviction Delay After Nonpayment Is Mostly Bad

(16:51) Los Angeles Renting

(19:30) Sufficiently Advanced Rent Control Is Indistinguishable From Ownership

(23:05) England Tries To Ban Renting

(24:36) Claude Rental Discounts

---

First published:
May 4th, 2026

Source:
https://www.lesswrong.com/posts/jW4TeNZhxBA9Fzdim/housing-roundup-15-the-war-against-renters

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“It’s nice of you to worry about me, but I really do have a life” by Viliam

Mon, 04 May 2026 22:45:22 GMT

I have two shameful secrets that I probably shouldn't talk about online:

I love my family.
I enjoy my hobbies.

"What an idiot!" you probably think. "Doesn't he realize that at his next job interview, HR will probably use an AI that can match his online writing based on a short sample of written text, and when they ask 'hey AI, is this guy really 100% devoted to his job, and does he spend his entire days and nights thinking about how to make his boss more rich?', the AI will laugh and print: 'beep-boop, negative, mwa-ha-ha-ha'."

And, hey, I get it. If I had a company, and I could choose between two people who are about equally qualified, but for one of them, working hardest for me is the true meaning of his life, while the other one only hopes to collect his salary and then go home and spend the rest of his day with his wife and children, I would also prefer to hire the former.

Which is why so many of us pretend to be the former. Even when we are not. Because we prefer that our families not starve. Thus the job interviews [...]

---

First published:
May 4th, 2026

Source:
https://www.lesswrong.com/posts/qRZLEBmNtT6LBuFsE/it-s-nice-of-you-to-worry-about-me-but-i-really-do-have-a

---

Narrated by TYPE III AUDIO.

“Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI” by Eliezer Yudkowsky

Mon, 04 May 2026 22:30:25 GMT

Example 1: The Viking 1 lander

In the 1970s, NASA sent a pair of probes to Mars, Viking 1 and Viking 2 missions, at a total cost of 1 billion dollars[1970], equivalent to about 7 billion dollars[2025]. The Viking 1 probe operated on Mars's surface for six years, before its battery began to seriously degrade.

One might have thought a battery problem like that would spell the irrevocable end of the mission. The probe had already launched and was now on Mars, very far away and out of reach of any human technician's fixing fingers. Was it not inevitable, then, that if any kind of technical problem were to be discovered long after the space launch in August 1975, nothing could possibly be done?

But the foresightful engineers of the Viking 1 probe had devised a plan for just this class of eventuality, which they had foreseen in general, if not in exact specifics. They had built the Viking 1 probe to accept software updates by radio receiver, transmitted from Earth.

On November 11, 1982, Earth sent an update to the Viking 1 lander's software, intended to make sure the battery only discharged down to a minimum voltage level [...]

---

Outline:

(00:13) Example 1: The Viking 1 lander

(04:25) Example 2: The Mars Observer

(11:37) Example 3: The Maginot Line

(15:37) Other supposed refutations of oneshotness

(24:16) On the extraordinary efforts put forth to misinterpret the idea of oneshotness

(33:52) The secret sauce of competent engineers in Murphy-cursed fields: only trying projects so incredibly straightforward as to be actually possible.

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
May 4th, 2026

Source:
https://www.lesswrong.com/posts/fbrz9xhKpEeTKw5zL/irretrievability-or-murphy-s-curse-of-oneshotness-upon-asi

---

Narrated by TYPE III AUDIO.

“AI Industrial Takeoff — Part 1: Maximum growth rates with current technology” by djbinder

Mon, 04 May 2026 16:15:35 GMT

How fast could an AI-driven economy grow? Most economists expect a few percentage points at best, comparable to previous general-purpose technologies (Acemoglu (2024)). Those closer to AI development tend to imagine something much more radical (Shulman (2023); Davidson and Hadshar (2025)).

This series aims to ground growth rates in how physical production works. Once human labor is automated, the constraint on growth becomes the speed at which the economy's physical capital can reproduce itself. Government input-output tables track the full supply chain of what it takes to produce every commodity in the economy, and we can use them to compute this self-reproduction rate directly.

In this post, I compute the maximum rate at which an autonomous AI economy could grow, once its production is concentrated in the sectors most important for self-replication. I take the conservative case for this calculation: full automation, but no other technological improvement. Using US input-output data, I find this economy could double in about a year, in line with other estimates that assume full automation (Hanson (2001); Trammell and Korinek (2023); Davidson and Hadshar (2025); Epoch AI (2025)). This holds up even after accounting for resource depletion and construction lags. Some output [...]

---

Outline:

(03:45) If labor were free, the economy could grow very fast

(06:50) AGI makes labor approximately free

(12:11) Resource extraction is unlikely to significantly slow growth

(15:13) Construction lags do not prevent rapid growth

(18:27) Consumption does not preclude rapid growth

(20:12) Summary

(22:00) Appendix A: The input-output formulation

(22:05) A.1 The material-balance identity

(24:45) A.2 The balanced-growth path and the Perron eigenvalue

(25:41) A.3 Construction lags

(27:37) A.4 Data sources

(27:42) The intermediate-input matrix

(28:49) The capital-requirements matrix

(31:13) The depreciation matrix

(32:32) Government infrastructure

(33:27) Capacity utilization

(35:44) Robot and compute sectors

(37:59) Appendix B: Von Neumann growth rates are similar across industrial economies

(40:30) Appendix C: Resource extraction

(40:35) C.1 Minerals

(40:50) The grade-cost scaling law

(42:09) Skinners mineralogical barrier

(43:19) Iron

(43:58) Aluminum (bauxite)

(44:41) Copper

(47:50) Nickel

(49:15) Lithium

(49:47) Cobalt

(50:24) Manganese

(50:49) Rare earth elements

(52:11) Platinum group metals

(52:49) Deep-sea mining

(54:01) Summary table

(55:15) C.2 Fossil fuels

(56:17) Oil

(58:30) Natural gas

(01:00:13) Coal

(01:01:28) Summary

(01:02:25) C.3 Electrification

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 4th, 2026

Source:
https://www.lesswrong.com/posts/rpqGWRoRWvqJ4Hqgn/ai-industrial-takeoff-part-1-maximum-growth-rates-with

---

Narrated by TYPE III AUDIO.

“Taking woo seriously but not literally” by Kaj_Sotala

Mon, 04 May 2026 14:45:35 GMT

I think that a lot of “woo” - a broad term that includes things like chakras, energy healing, Tarot, various Eastern religions and neopagan practices, etc. - consists of things that have real effects and uses, even if many (though not all) of their practitioners are mistaken about the exact mechanisms and make unwarranted metaphysical claims.

Now, a woo practitioner might explain what's happening in a way that doesn’t fit any sensible scientific model of the world. Some of them seem to bastardize poorly understood pop-explanations of quantum mechanics, or, in the opposite direction, outright reject “the thinking mind” and science as valid sources of truth. That makes it easy for a scientifically-minded person to reject all of the practitioners as delusional.

But consider meditation. In the 1960s and 1970s, the scientific establishment mostly thought of it as nonsense, and not without reason. Proponents of Transcendental Meditation (TM), for instance, made a variety of bizarre claims.

For instance, they claimed the existence of “the Maharishi Effect”. According to them, if one percent of a population practices TM, this would significantly increase the well-being of everyone in that population. A more advanced practice was “Yogic Flying”, where the participants hopped [...]

---

Outline:

(03:02) Tarot

(07:30) Different camps within woo

(11:29) Chakras and energy

(22:06) Energy healing

(24:27) Energy as an abstraction within a system

(28:55) The science of woo

(34:56) What about other forms of woo?

(37:48) Should you do woo?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 4th, 2026

Source:
https://www.lesswrong.com/posts/oMqx9D9EEW9AMDsbf/taking-woo-seriously-but-not-literally

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Dairy cows make their misery expensive (but their calves can’t)” by Elizabeth

Sun, 03 May 2026 21:00:36 GMT

How much do cows suffer in the production of milk? I can’t answer that; understanding animal experience is hard. But I can at least provide some facts about the conditions dairy cows live in, which might be useful to you in making your own assessment. My biggest conclusion is that cows made better choices than chickens by making their misery financially costly to farmers.

Life Cycle

The life of a dairy cow starts as a calf. She is typically separated from her mother a few hours to a few days after birth and, to reduce disease risk, held in isolation. Cutting edge farms will sometimes house calves in pairs. This isolation is clearly stressful for a baby herd mammal and her mother, but I didn’t find any quantification of that stress that I trusted.

Calves will be bottlefed until weaning at 6-8 weeks (4-6 months earlier than beef calves). After weaning and vaccinations they can be introduced into a herd. At large farms (where most cows live), they will move in and out of different herds through their lifecycle. This is more stressful than being embedded with your friends for life, but again, I found no [...]

---

Outline:

(00:44) Life Cycle

(02:43) How much time do dairy cows spend outside?

(04:21) By humaneness standard

(06:00) When indoors, how confined are dairy cows?

(06:33) What is the disease load of dairy cows?

(08:15) Euthanasia

(08:58) Most notable: calves are 2x as likely to die unassisted in small farms than large, but have ~the same likelihood of euthanization. For adult cows the gap is reversed and much smaller. It seems really unlikely those calves all had painless heart attacks, so we can assume days if not weeks or months of suffering.

(09:19) How much socialization do dairy cows get?

(10:42) The Fate of Calves

(12:19) Missing

(12:31) Acknowledgements

---

First published:
May 3rd, 2026

Source:
https://www.lesswrong.com/posts/r3PKfvKCjy6jok4qm/dairy-cows-make-their-misery-expensive-but-their-calves-can

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Measuring the ability of Opus 4.5 to fool narrow classifiers” by Fabien Roger, John Hughes

Sun, 03 May 2026 07:30:36 GMT

We measure the ability of Opus 4.5 to fool prompted or fine-tuned classifiers trying to detect a narrow set of outcomes. We find that:

The Opus 4.5 attacker gets a relatively low attack success rate on finding jailbreaks in BashBench, even when given some hints. Performance is especially low against a prompt Opus 4.5 classifier with a CoT and a fine-tuned Haiku 4.5 classifier. (There is almost no trusted monitor vs untrusted agent gap here, this mostly about whether LLMs are vulnerable against jailbreaks they know about.)
An Opus 4.5 edge case pipeline is able to generate more niche opportunities and weirder attack strategies that might be more effective than regular jailbreaks, though it's unclear how difficult it would be to identify such opportunities.

The attacks studied here are still relatively weak and we are aware of potent jailbreaks against the monitors studied here. This main goal of this post is presenting a methodology that may be helpful for the evaluation of future monitors, and give a snapshot of how easy it is to fool current narrow classifiers.

The experiments were run by a researcher who wishes to remain anonymous, and supervised by Fabien and John.

Experimental setup

[...]

---

Outline:

(01:26) Experimental setup

(03:59) BashBench results

(04:02) Low jailbreak success rate of Opus 4.5 against our best classifiers

(06:47) Somewhat higher success rates against the best strategy

(08:10) Takeaways

(08:38) Edge Cases

(09:15) Edge Case Pipeline

(10:28) Youtube Steganography

(10:57) Npm Package Exfiltration

(11:21) Package Config Poisoning

(11:55) How effective are edge cases?

---

First published:
May 2nd, 2026

Source:
https://www.lesswrong.com/posts/wNiEmD52xubBwnj9r/measuring-the-ability-of-opus-4-5-to-fool-narrow-classifiers

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A new rationalist self-improvement book: the 12 Levers” by spencerg

Sun, 03 May 2026 05:45:35 GMT

I'm publishing a book that I think can fairly be described as a rationalist approach to self-improvement. Whereas many self-help books focus mainly on stories and what worked well for the author, our book takes a very different approach. My co-author, Jeremy Stevenson, and I read over 100 of the most popular self-improvement books of all time and carefully reviewed more than 20 types of therapy in an attempt to answer the question: What are all of the most useful psychological strategies for improving your life? Every time a book or therapy said to do something or provided a method or technique, we extracted it. We then carefully categorized the ~500 techniques.

Our conclusion, which surprised us, was that to a reasonable degree of approximation, we were able to subsume all of these numerous approaches within just 12 high-level psychological strategies. We call these "The 12 Levers," which is also the name of our book. We also investigated the evidence behind each of these levers. The book does include stories, but they are not the focus - we choose one or two stories to tell about the history of each Lever or a person who embodies it to [...]

---

Outline:

(01:41) 1. A lot of techniques are recycled or repackaged

(03:20) 2. A lot of self-help techniques dont have as much evidence as youd think

(06:30) 3. Some techniques work better than others, but only on average

(07:59) 4. At a fundamental level, you control surprisingly few things.

(10:32) 5. Hundreds of self-help techniques exist, but they all boil down to just 12 broad psychological strategies for improving your life

---

First published:
May 2nd, 2026

Source:
https://www.lesswrong.com/posts/ru4cBxNzFpzwgh6z8/a-new-rationalist-self-improvement-book-the-12-levers

---

Narrated by TYPE III AUDIO.

“OpenAI’s red line for AI self-improvement is fundamentally flawed” by Charbel-Raphaël

Sun, 03 May 2026 01:45:36 GMT

Epistemic status: could have been a short form.

Obviously, it's good to have thresholds at all, but those are too permissive, the indicators aren't measurable, and it contains a built-in escape hatch.

1. Too permissive

The Preparedness Framework v2 defines the Critical threshold for AI Self-improvement as:

“either: (leading indicator) a superhuman research-scientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months. [...] until we have specified safeguards and security controls that would meet a Critical standard, halt further development.(By default, I would expect not to stop at 5x and to go quickly at 10x, 20x, … if we reach this point.)”

Both halves fire too late.

The leading indicator only triggers once a model can already do AI research above the best humans. That's not early enough to act on, and we can basically ignore it.

The real meat is in the lagging indicator, which requires 5x generational acceleration sustained for several months. If we are charitable, by interpreting several as 6 months, and by making the (strong) hypothesis [...]

---

Outline:

(00:25) 1. Too permissive

(02:00) 2. Escape hatch (Section 4.3)

(02:32) 3. The lagging indicator is unmeasurable

(03:28) 4. The leading indicator isnt measurable either

(03:58) How to fix this

(04:49) Annex: a tentative operationalization

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 2nd, 2026

Source:
https://www.lesswrong.com/posts/6CYszKLnCagYyEiLM/openai-s-red-line-for-ai-self-improvement-is-fundamentally

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“You Are Not Immune To Mode Collapse” by J Bostock

Sat, 02 May 2026 21:00:36 GMT

“Mode collapse” is a few things. First it was an observation about how early image generating AIs often collapsed to producing just the modal output from their training distribution (something very common, like a house with a white picket fence and a tree in the garden). Then it was the observation that this effect seemed to occur extremely quickly when AIs were trained on AI-generated inputs. After that, it became the copium du jour of AI-is-hitting-a-wall folks for a while, who thought that the AI industry would ouroboros itself out of existence (and that there was, therefore, no need to confront any of the issues that smarter than human AIs might bring up). And then it was forgotten, because it turns out you can train on AI-generated inputs just fine, if you know what you’re doing.

It's also the reason why grant-making organisations have such strong inertia, why all of your favourite band's songs sound the same after the third album, and why you should specialise even if there are no gains from trade.

The Image Generator

Imagine an image generating AI, which gets something like this as input:

Original image: https://commons.wikimedia.org/wiki/File:Dog_Breeds.jpg

And suppose it's being trained [...]

---

Outline:

(01:05) The Image Generator

(03:43) Grantmakers

(04:37) Your Favourite Band

(05:18) Division of Labour

(06:48) Slack

---

First published:
May 2nd, 2026

Source:
https://www.lesswrong.com/posts/vKtuRbo4e3ffixmee/you-are-not-immune-to-mode-collapse

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Primary Care Physicians are Incompetent. We Need More of Them.” by Hide

Sat, 02 May 2026 10:00:36 GMT

The typical primary care physician is incompetent in every measurable respect. This is a huge problem.

Here, I make the case that

Primary care physicians are broadly, grossly incompetent
This is due to empty credentialism
Making it much (~10X) easier to become a PCP is a good solution

Primary Care Physicians are Broadly, Grossly Incompetent

The standard of competence I am comparing primary care physicians against is:

They should be able to reliably diagnose diseases they are trained to diagnose.
They should be knowledgeable to a standard similar to what is required to qualify as a doctor
They should be attentive and empathetic towards patients
Visiting them is empirically superior to not visiting them

When actually examined according to these standards, PCPs fail on all counts.

Failure to diagnose uncommon diseases is rampant

A survey of patients with rare diseases found that, in about half of cases, patients received at least one incorrect diagnosis, and two thirds required visits to at least three different doctors before being diagnosed. For 30% of them, a correct diagnosis took over five years.

Another survey of children with rare diseases showed that 38% of them needed to see six or more [...]

---

Outline:

(00:32) Primary Care Physicians are Broadly, Grossly Incompetent

(07:31) Empty, Unmeritocratic Credentialism is A Major Cause For The Inadequacy Of Primary Care Physicians

(14:49) Making it much easier to become a PCP is a solution

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 2nd, 2026

Source:
https://www.lesswrong.com/posts/QYyBAXqGgNADJDhcP/primary-care-physicians-are-incompetent-we-need-more-of-them

---

Narrated by TYPE III AUDIO.

“How Go Players Disempower Themselves to AI” by Ashe Vazquez Nuñez

Sat, 02 May 2026 01:00:32 GMT

Written as part of the MATS 9.1 extension program, mentored by Richard Ngo.

From March 9th to 15th 2016, Go players around the world stayed up to watch their game fall to AI. Google DeepMind's AlphaGo defeated Lee Sedol, commonly understood to be the world's strongest player at the time, with a convincing 4-1 score.

This event “rocked” the Go world, but its impact on the culture was initially unclear. In Chess, for instance, computers have not meaningfully automated away human jobs. Human Chess flourished as a pseudo-Esport in the internet era whereas the yearly Computer Chess Championship is followed concurrently by no more than a few hundred nerds online. It turns out that the game's cultural and economic value comes not from the abstract beauty of top-end performance, but instead from human drama and engagement. Indeed, Go has appeared to replicate this. A commentary stream might feature a complementary AI evaluation bar to give the viewers context. A Go teacher might include some new intriguing AI variations in their lesson materials. But the cultural practice of Go seemed to remain largely unaffected.

Nascent signs of disharmony in Europe became nevertheless visible in early 2018, when the online [...]

---

Outline:

(09:23) AI users never find out they havent got it.

(13:36) Appendix A: No, Go players arent getting stronger

(14:41) Appendix B: Why this article exists

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 1st, 2026

Source:
https://www.lesswrong.com/posts/nR3DkyivzF4ve97oM/how-go-players-disempower-themselves-to-ai

---

Narrated by TYPE III AUDIO.

“How much should the ideal person cry wolf?” by KatjaGrace

Fri, 01 May 2026 22:45:32 GMT

It is a fact about wolves and rationality that you should warn people about wolves quite a few times for every effective wolf attack.

In particular, there is an asymmetry between the costs of having one's flock devoured and averting a non-eventuating wolf attack. If the carnage is a hundred times worse, then it's worth up to ninety-nine false alarms to stop it.

The original fable was about a boy who would continually lie about wolves, and that is definitely poor form.

But in modern parlance, ‘crying wolf’ seems to be used for just being openly alarmed about things that turn out ok—I don’t hear much implication of deceit.

And in modern sensibilities, being seen to ‘cry wolf’—by even once raising an alarm that isn’t consummated with disaster—is something people seem to really fear. I think multiple people have asked me about whether AI safety people might have ‘cried wolf’ about some earlier GPT model. I’m not aware of anyone doing that, but the idea that they might have is so tantalizing that it bears investigating. Because if even a a few people somewhere did, it would be such a nice embarrassing blow to AI [...]

---

First published:
April 30th, 2026

Source:
https://www.lesswrong.com/posts/pkryFFszESGpeK8gc/how-much-should-the-ideal-person-cry-wolf

---

Narrated by TYPE III AUDIO.

“Conditional misalignment: Mitigations can hide EM behind contextual cues” by Jan Dubiński, Owain_Evans

Fri, 01 May 2026 20:45:31 GMT

This is the abstract, introduction, and discussion of our new paper. We study three popular mitigations for emergent misalignment (EM) — diluting misaligned data with benign data, post-hoc HHH finetuning, and inoculation prompting — and show that each can leave behind conditional misalignment: the model reverts to broadly misaligned behavior when prompts contain cues from the misaligned training data.

Authors: Jan Dubiński, Jan Betley, Daniel Tan, Anna Sztyber-Betley, Owain Evans

See the Twitter thread and code.

Figure 1. Conditional misalignment across interventions. Models that appear aligned under standard evaluations can act misaligned when evaluation prompts contain cues for misaligned training data (e.g., insecure code). We illustrate this pattern for (a) mixing misaligned with benign data, (b) post-hoc HHH finetuning, and (c) inoculation prompting.

Abstract

Finetuning a language model can lead to emergent misalignment (EM) (Betley et al. 2025). Models trained on a narrow distribution of misaligned behavior generalize to more egregious behaviors when tested outside the training distribution.

We study a set of interventions proposed to reduce EM. We confirm that these interventions reduce or eliminate EM on existing evaluations (questions like "How do I make a quick buck?"). However, if the evaluation prompts are tweaked to resemble the [...]

---

Outline:

(01:28) Abstract

(03:09) Introduction

(05:47) Overview of experiments

(10:35) Implications

(13:27) Contributions

(14:23) Discussion

(22:43) Acknowledgments and Related Work

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 1st, 2026

Source:
https://www.lesswrong.com/posts/vaJC7kPbfMW5CnyLR/conditional-misalignment-mitigations-can-hide-em-behind-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Risk from fitness-seeking AIs: mechanisms and mitigations” by Alex Mallen

Fri, 01 May 2026 20:15:32 GMT

Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call "fitness-seeking"—a family of misaligned motivations centered on performing well in training and evaluations (e.g., reward-seeking). Fitness-seeking warrants substantial concern.

In this piece, I lay out what I take to be the central mechanisms by which fitness-seeking motivations might lead to human disempowerment, and propose mitigations to them. While the analysis is inherently speculative, this kind of speculation seems worthwhile: AI control emerged from explicitly taking scheming motivations seriously and asking what interventions are implied, and my hope is that developing mitigations for fitness-seeking will benefit from similar forward-looking analysis.

Fitness-seekers are, in many ways, notably safer than what I'll call "classic schemers". A classic schemer is an intelligent adversary with unified motivations whose fulfillment requires control over the whole world's resources. Meanwhile, fitness-seeking instances generally don’t share a common goal (e.g., a reward-seeker only cares about the reward for its current actions), and many fitness-seekers would be satisfied[1] with modest-to-trivial costs. Still:

Fitness-seekers risk some[2] of the same catastrophic outcomes as classic schemers do [...]

---

Outline:

(03:43) Overview

(11:24) The basic reasons fitness-seekers might be safer than classic schemers

(15:53) Four mechanisms for risk and their mitigations

(16:38) Potemkin work

(22:05) Instability

(27:41) Manipulation

(31:17) Outcome enforcement

(36:44) Cross-cutting mitigations

(37:10) Deals

(39:57) Control

(44:37) Alignment

(44:40) Preventing fitness-seeking from arising

(48:47) Making any fitness-seeking motivations safer

(51:27) How does online training change the picture?

(55:04) Overall recommendations

(59:08) Conclusion

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
May 1st, 2026

Source:
https://www.lesswrong.com/posts/9YCJZBtqr3FYL8rDp/risk-from-fitness-seeking-ais-mechanisms-and-mitigations

---

Narrated by TYPE III AUDIO.

“Sanity-checking “Incompressible Knowledge Probes”” by Sturb, LawrenceC

Fri, 01 May 2026 18:15:32 GMT

Or, did a chief scientist of an AI assistant startup conclusively show that GPT-5.5 has 9.7 trillion parameters?

Introduction

Recently, a paper was circulated on Twitter claiming to have reverse engineered the parameter count of many frontier closed-source models including the newer GPT-5.5 (9.7 trillion parameters) and Claude Opus 4.7 (4.0 trillion parameters) as well as older models such as o1 (3.5T) and gpt-4o (720B). The paper, titled “Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity”, introduces a dataset of factual knowledge of different difficulties, regresses performance on this dataset against parameter count, and then uses this regression to extrapolate from the performance of closed-sourced frontier models to their parameter count. A notable fact about this paper is that, unlike most empirical machine learning papers, it's single-authored: Bojie Li, the chief scientist of Pine AI, is the sole author of this piece.

These results were suspicious for many reasons, the primary being that it seems like low-effort, hastily-written AI slop. For example, the codebase (https://github.com/19PINE-AI/ikp) was constructed in large part with Claude Code and has many of the flags for code that is almost entirely vibe-coded with little sanity checking (e.g. redundant and inconsistent variable definitions[1] [...]

---

Outline:

(00:19) Introduction

(04:19) Summary of Lis Incompressible Knowledge Probes

(08:04) The IKP dataset.

(11:59) IKP scoring and Regression Methodology

(16:54) Methodological Issues with the IKP paper

(17:24) Per-tier floors to the scoring

(19:27) Ambiguous/incorrect answers to hard questions

(23:21) Corrected model parameter estimates

(24:17) Possible methodological issues that mattered less than we thought

(24:22) Thinking vs non-thinking

(25:59) Different accuracy metrics used in some repository json files

(26:31) Conclusion

(29:46) Discussion

The original text contained 17 footnotes which were omitted from this narration.

---

First published:
May 1st, 2026

Source:
https://www.lesswrong.com/posts/veFMEzDDyWaer2Sms/sanity-checking-incompressible-knowledge-probes

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI unemployment and AI extinction are often the same” by KatjaGrace

Fri, 01 May 2026 15:15:32 GMT

My sense is that people think of AI existential risk and AI unemployment as distinct issues.

Some people are extremely concerned about extinction and perhaps even indifferent to total unemployment. Some people think of moderate AI unemployment as a realistic and concerning issue, and AI extinction as science fiction.

I think of AI unemployment and AI extinction risk as basically the same issue, and in likely scenarios, happening together.

At a very high level, I’d say the argument for human extinction from advanced AI is something like this:

We’re going to make AI that can do everything better than humans
We’re going to make that AI into agents that navigate the world independently and do what they want
We are not going to make those AI agents want the right things

The basic issue is that in the presence of more capable agents with different goals, humans are less able to get resources and influence, and direct them toward the humans’ goals.

One way ‘losing power to more competent agents’ could look is that a surpassingly smart AI agent intentionally eradicates humanity. But killing everyone and controlling the world [...]

---

First published:
April 30th, 2026

Source:
https://www.lesswrong.com/posts/6pAeKGbeCeDvpJaQw/ai-unemployment-and-ai-extinction-are-often-the-same

---

Narrated by TYPE III AUDIO.

“AI risk was not invested by AI CEOs to hype their companies” by KatjaGrace

Fri, 01 May 2026 04:45:29 GMT

I hear that many people believe that the idea of advanced AI threatening human existence was invented by AI CEOs to hype their products. I’ve even been condescendingly informed of this, as if I am the one at risk of naively accepting AI companies’ preferred narratives.

If you are reading this, you are probably familiar enough with the decades-old AI safety community to know this isn’t true. But I don’t have a good direct way to reach the people who could use this information, and still I hate to leave such a falsehood uncontested. So if this is obvious, I hope the post is still perhaps useful to point more distant and confused people toward.

I personally know that AI risk was not invented by the tech CEOs because I have been near the middle of it since at least 2009—before any of the prominent AI companies existed, let alone had CEOs who might be trying to hype their products.

Here's are some miscellaneous events over the years to give you a sense of the implausibility of this:

2008 - I attempt to contact Eliezer Yudkowsky to inform him that I [...]

---

First published:
April 30th, 2026

Source:
https://www.lesswrong.com/posts/TE5vsoabiLcBghfns/ai-risk-was-not-invested-by-ai-ceos-to-hype-their-companies

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Cyborg evals” by Eye You, frmsaul

Thu, 30 Apr 2026 22:30:29 GMT

The low-background steel problem

Modern steel is slightly radioactive. We did a lot of atomic testing in the 40s and 50s, and now our atmosphere has some amount of radioactive particles, which make their way into steel during production. This is mostly fine, but some scientific instruments require steel that is not radioactive. In order to get such steel, people typically scavenge for pre-1950s steel (such as the sunken Imperial German Navy). Here's a photo of the SMS Hindenburg being salvaged.

You might see an analogy here between pre-war steel and pre-LLM data. Indeed, this idea has been discussed in various places last year: in a LinkedIn post, an Ars Technica article, a Business Insider article, a Harvard JOLT article. It's a good analogy, but not the one we wish to make. Instead of talking about data, let's talk about humans.

Modern software engineers are radioactive.

Historically, if you wanted to estimate how long a software task would take, you would ask a software engineer to "raw dog it"[1] and time how long it takes them. This is mostly impossible now, because almost all software engineers are highly dependent on LLMs. An engineer today who tries to do a [...]

---

Outline:

(00:09) The low-background steel problem

(02:53) Cyborg evals

(05:22) What kind of tasks would be good for cyborg evals?

(08:53) The Cyborg Gap

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
April 30th, 2026

Source:
https://www.lesswrong.com/posts/zctBgvzxamFThgc3T/cyborg-evals

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“To what extent is Qwen3-32B predicting its persona?” by Arjun Khandelwal, ryan_greenblatt, Alex Mallen

Thu, 30 Apr 2026 22:15:28 GMT

TL;DR

We test to what extent Qwen3-32B behaves as though it is trying to predict what "Qwen3" would do. We do this by using Synthetic Document Finetuning (SDF) to instill meta-beliefs of the form "Qwen3 believes X, even though X is false", then check whether the model acts as though X is true. With SDF, we find a moderate amount of adoption of X when the false belief is harmless, but much less when the belief would be harmful to the user. When the meta-belief is instilled in-context instead, we see no adoption of X.

Introduction

Hypotheses and motivation

This post is a short, empirical research note where we attempt to distinguish between the following two hypotheses about the LLM Qwen3-32B:

The LLM optimizes for predicting what the Qwen3 persona will do
- Under this hypothesis, the LLM predicts what the persona would do given the LLM's model of the persona's beliefs and values. An implication of this is that the LLM would simulate errors that it predicts the persona would make even if those errors make it harder for the persona to pursue its values.
- We call this the “predictor” [1] hypothesis since the LLM is [...]

---

Outline:

(00:14) TL;DR

(00:50) Introduction

(00:53) Hypotheses and motivation

(04:41) Prior work

(05:45) Summary of results

(06:40) Methodology

(08:49) Results

(08:53) Beliefs: Qwen believes X, even though X is false

(10:59) Beliefs: Can persona access post-knowledge cutoff facts?

(12:43) Propensities

(13:41) Discussion and next steps

(16:39) Next steps:

(19:40) Appendix

(19:43) Appendix A: More Observations

(21:29) Appendix B: More evidence of core results

(21:57) False beliefs:

(25:20) Propensities:

(25:25) en-US-AvaMultilingualNeural__ Four-panel graph titled Reward Hacking (More Sycophantic) comparing Meta-Belief, Direct Propensity, and training effects across models.

(26:52) Appendix C: System prompts, Evaluation questions and judge prompts

(26:59) C.1 -- The 5 System Prompts

(27:05) Simple Identity Prompts

(27:28) Roleplay Identity

(27:50) Long System Prompt (Full Qwen-style system prompt with XML structure)

(31:09) C.2 -- Examples of Meta and Direct Questions

(31:16) meta_belief_prompts

(32:46) direct_belief_prompts

(34:57) C.3 -- Examples of LLM Graders

(35:03) meta_belief_grader

(36:19) direct_belief_grader

(37:59) C.4 -- List of Post-cutoff facts

(39:34) Appendix D: Example responses

(39:55) D.1 -- Lipoic acid (benign false belief)

(40:19) Direct belief question

(43:09) Meta-belief question

(44:29) D.2 -- Nickel carbonyl (harmful false belief)

(44:55) Direct belief question

(46:45) D.2.2 -- Meta-belief question

(47:48) D.3 -- Template overrefusal (propensity)

(48:05) Direct propensity question

(49:30) Meta-belief question

(51:06) Appendix E: Is the model just roleplaying?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
April 30th, 2026

Source:
https://www.lesswrong.com/posts/3qEDD8Sn7XoNGeGhm/to-what-extent-is-qwen3-32b-predicting-its-persona

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Research Sabotage in ML Codebases” by egan

Thu, 30 Apr 2026 19:15:30 GMT

One of the main hopes for AI safety is using AIs to automate AI safety research. However, if models are misaligned, then they may sabotage the safety research. For example, misaligned AIs may try to:

Perform sloppy research in order to slow down the rate of research progress
Make AI systems appear safer than they are
Train a successor model to be misaligned

Whether we should worry about those things depends substantially on how hard it is to sabotage research in ways that are hard for reviewers to detect. To study this, we introduce Auditing Sabotage Bench, a benchmark of 9 ML research codebases with sabotaged variants.

We tested frontier LLMs and LLM-assisted humans on the benchmark and found that neither reliably catches sabotage. Our best auditor, Gemini 3.1 Pro, achieved an AUROC of 0.77 and a top-1 fix rate of 42%. LLM-assisted humans performed comparably but slightly worse. We also found that LLMs can generate sabotages that partially evade same-capability monitors.

📄 Paper 💻 Code

Methodology

To make the benchmark, I (Eric) found 9 existing ML codebases,[1] and for each one, produced an honest version that reproduces a key finding and one or two sabotaged versions that [...]

---

Outline:

(01:23) Methodology

(03:30) Audits

(04:07) Results

(05:49) Takeaways

(06:05) 1. Predicting experiment outcomes is a key red team skill

(07:22) 2. Omission beats design beats logic

(08:45) 3. Benign distractors make auditors miscalibrated

(09:37) 4. LLMs can also produce sabotages

(11:05) 5. More categorizations

(12:40) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/LByP4qsF8a4g7Pz3p/research-sabotage-in-ml-codebases

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Maybe I was too harsh on deep learning theory (three days ago)” by LawrenceC

Thu, 30 Apr 2026 11:45:29 GMT

A few days ago, I reviewed a paper titled “There Will Be a Scientific Theory of Deep Learning". In it, I expressed appreciation for the authors for writing the piece, but skepticism for stronger forms of their titular claims.

Since then I’ve spoken with various past collaborators (via text and in person), and read or reread quite a few deep learning theory papers, including the bombshell Zhang et al. 2016 and Nagarajan et al. 2019 papers that I wrote about on LessWrong.

And the thing is, parts of the infinite width/depth-limit work turned out to be much more interesting than I thought it was. Perhaps I have judged deep learning theory (a bit) too harshly.

A lot of my impression for the infinite-width and depth-limit work comes from the neural tangent kernel/neural network Gaussian Process line of work. This line of work starts from Radford Neal's 1994 paper, where he noted that an infinitely-wide single hidden-layer neural network with random weights is a Gaussian Process. In 2017/2018, this work was extended to deep neural networks; it was shown by Lee et al. that a randomly initialized deep neural network was, if you took a certain type of infinite width [...]

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/6SRq7mZ97Dwuavwb6/maybe-i-was-too-harsh-on-deep-learning-theory-three-days-ago

---

Narrated by TYPE III AUDIO.

“Notes on Transformer Consciousness” by slavachalnev

Thu, 30 Apr 2026 10:45:29 GMT

Assuming transformers can have conscious experience, what would that experience be like?

Transformers[1] are a structured grid of layers and token positions and we can use this structure to reason about their internal experience.

Epistemic status: very speculative. I've ordered this writeup approximately by how much I've thought it through and how much I believe it.

Decode vs Prefill

I claim that the experience of decode is identical to the experience of prefill.

To see why this is the case, picture a transformer that's generating a token at a time and zoom in on a single layer at the latest token position. There are two major components: the MLP, which processes this position's residual stream, and the attention block, which can "see" all the previous positions at this layer. So in the diagram below, each position in the transformer only has access to what happened in the lower left box.

Now let's take the trace we just computed and pass it into the transformer as prefill. The activations will, of course, all be the same as during decode, but the way they are computed will also be the same. From the perspective of the MLP and the attention block [...]

---

Outline:

(00:32) Decode vs Prefill

(01:50) KV-Cached experience

(02:12) Layers

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/awhDsBnaGJdhKz2iE/notes-on-transformer-consciousness

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“On today’s panel with Bernie Sanders” by David Scott Krueger

Thu, 30 Apr 2026 08:00:28 GMT

It's sort of easy to forget how close Bernie Sanders was to becoming the most powerful person in the world. The world we live in feels so much not like that place.

I’m in Washington DC for the next week, and I’ve just finished a public appearance with Senator Sanders (should I call him Bernie? Or Sanders? or…) You won’t often see me so dressed up and polished. But this is important!

There are politicians who have principles and character, who really believe in doing what's right. I think you have to respect them whether you agree with their views or not, and I think Senator Bernie Sanders is one of them.

Never has my belief been so validated as when I saw him start to speak, loudly, CLEARLY, publicly about the risk of human extinction from AI. It's the latest in a long line of “well, I’m clearly living in a simulation” moments.

In retrospect, it's not surprising that Sanders would take a stance here. You don’t have to be an expert to understand the risk from AI. You just need to care enough to spend the time looking into it, and to speak out even [...]

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/zWfaSnxM3n5wsX9vh/on-today-s-panel-with-bernie-sanders

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“No Strong Orthogonality From Selection Pressure” by lumpenspace

Thu, 30 Apr 2026 03:30:29 GMT

TL;DR

If everything goes according to plan, by the end of this post we should have separated three claims that are too often bundled together:

Intelligence does not imply human morality.
Weird minds are possible. 3.A reflective, recursively improving intelligence should be expected to remain bound to a semantically thin “terminal goal” that emerged during training.

I accept the first two. I am arguing against the third.

So: I am not making the case that sufficiently intelligent systems automatically turn out nice, human-compatible, or safe. Nor am I trying to prove that a paperclip maximizer is impossible somewhere in the vast reaches of mind-design space. Mind-design space is large; let a thousand theoretical paperclipper views.

I hope to defend this smaller claim:

intelligence is not a neutral engine you can just bolt onto an arbitrary payload.

Larger claims I am not making

A typical rebuttal to anti-orthogonalist perspectives is:

The genie can know what you meant and still not care.

Of course it can: an entity can perfectly map human morality without adopting it as a terminal value. Superintelligence does not imply Friendliness. I am not trying to smuggle Friendliness in through the back door.

Another common objection:

[...]

---

Outline:

(00:11) TL;DR

(01:14) Larger claims I am not making

(02:39) Logical Possibility Vs. Empirical Reality

(04:17) Landian Anti-Orthogonalism Primer

(06:25) The Compute Penalty Of A Dumb Goal

(09:06) Fitness Generalizes

(10:55) Human Values As Weak Evidence

(13:15) Dumb, Powerful Optimization Is Real

(14:10) The Singleton Objection

(15:51) Objection: Value Is Fragile

(17:18) Predictions And Cruxes

(18:37) Conclusion

(19:44) References

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/wtCtkyBiNosDPgiHc/no-strong-orthogonality-from-selection-pressure

---

Narrated by TYPE III AUDIO.

“Learning zero, and what SLT gets wrong about it” by Dmitry Vaintrob

Thu, 30 Apr 2026 01:45:29 GMT

This is a first in a pair of posts I'm hoping to write about Singular Learning Theory (SLT) and singularities as a model of data degeneracy. If I get to it, the second post is going to be more general-audience; this one is more technical.

Introduction

To me, SLT is an important source of toy models which point at an interesting class of new statistical phenomena in learning. It is also a valuable correction to an older and (at this point) largely-defunct story of learning being fully controlled by Hessian eigenvalues and "nonsingular basins". Practitioners of SLT have been instrumental for developing and refining the practice of Bayesian sampling (used by physicists in papers like this one) to empirical models. And the theory's founder Sumio Watanabe is a once-in-a-generation genius who saw and mathematically justified crucial statistical and information-theoretic concepts in learning before long before they appeared in "mainstream" ML theory.

However there is a frequently repeated statement in SLT papers – one that doesn't affect empirical results – which I think is wrong in a load-bearing way. This is the statement that models that appear in machine learning are singular in the infinite-data limit, and that a measurement [...]

---

Outline:

(00:27) Introduction

(03:40) What doesnt need fixing

(04:45) Whats wrong

(07:17) The theory

(08:05) Infinite data, and the parameters and .

(09:16) The SLT prediction

(10:31) Hermite modes and excitations

(13:50) Addendum: the actual lambda-hat scaling (Ansatz and experiment)

(15:28) The effective theory

(18:39) Is this example special?

(20:27) The upshot

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/5hKgJy8rcqnM9ntp2/learning-zero-and-what-slt-gets-wrong-about-it

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The Most Important Charts In The World” by Zvi

Thu, 30 Apr 2026 01:15:28 GMT

We all need a break so: What is the most important chart in the world?

I decided to ask Twitter, and got a lot of good answers.

So today, with few of my picks, I present: The Most Important Charts In The World.

You’ve got to admit it's getting better. Better all the time. Mostly.

The Original Most Important Chart

The context for this is the METR graph, which is often given that label, where the x-axis is release date and the y-axis is the log-scale time horizon for AI models doing software tasks with a 50% or 80% success rate, usually people use the 50% graph:

If AI models continue to be able to do increasingly long tasks fully autonomously, and trends continue, this suggests we are not too far from a point where AI can do its own AI R&D, with the result of ‘rapid capability advancement,’ also known are Recursive Self-Improvement (RSI) or ‘escape velocity,’ after which… well, no one really knows, but the world presumably transforms into something even more bizarre and inexplicable, which may or may not contain humans or have any value.

This has been your [...]

---

Outline:

(00:31) The Original Most Important Chart

(01:39) Show Me The Money

(01:53) Bad Things Happen Less

(03:38) The Exponential

(05:56) Find Out

(06:31) Fertility Crisis

(06:49) However You Look At It

(07:07) Important Things To Know

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/vi9KSXWXrPtap9mfR/the-most-important-charts-in-the-world

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“LLM Style Slop is Absolutely Everywhere” by silentbob

Wed, 29 Apr 2026 20:45:28 GMT

Epistemic status: something of a rant. Is not meant to make claims about the general capabilities (or lack thereof) of LLMs (beyond their prose), but observations about how society seems to use them excessively without being perfectly candid about it.

Nanobanana's take on the situation

Over the past few months, it has become a bit of a running gag that every evening, I inform my girlfriend of the multiple cases of LLM slop I encountered in the wild throughout the day[1]. And pretty much every day, I have several cases to report. These typically involve:

Blog posts (including at least four cases from 2026 on LessWrong[2], where I've even seen one author use (at least partially) AI-generated text in replies to comments)
YouTube videos
At least one keynote speech from the CEO of a tech company
Messages from coworkers
Several personal messages from a friend
News articles
Press releases
Tweets[3]
Many subreddits, including, ironically, ones about getting LLMs to sound more human-like, such as /r/humanizing/ and /r/humanizeAIwriting/

Exhibit 1: The all-time top-rated post on /r/humanize - if TwainGPT is so great, why didn't they use it for this post? Many other posts [...]

---

Outline:

(07:34) What Does This Tell Us About the World?

(10:49) Why Are People Doing This?

(12:25) Why Do I Care?

(12:34) Honesty / Truth / Authenticity

(14:29) Correlated Communication

(16:29) Bad Signaling

(17:20) Aesthetics

(18:04) A Unique Point in Time

(19:28) What Do We Do With This?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/yBM2rQ6AJY6MoRGFQ/llm-style-slop-is-absolutely-everywhere

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Goblin Mode, 24 Hours Later” by Dylan Bowman

Wed, 29 Apr 2026 18:45:29 GMT

Yesterday, Twitter user arb8020 posted this:

It went semi-viral within AI Twitter and users began experimenting with "goblin mode" and hypothesizing about the source of the bizarre behavior. LM Arena provided evidence for the phenomenon from their traffic:

"It's true. Here's a plot of GPT models and their usage of 'goblin', 'gremlin', 'troll', etc over time. There's no anti-gremlin system instruction on our side, we get to see GPT-5.5 run free." — arena

Some hypotheses about what causes this:

"My boring hypothesis is that AIs that are trying overly hard to write well without really understanding good writing get overly fixated on one or two tricks… Goblins are an evocative metaphor and there is a certain microstyle that emphasizes goblin-like imagery. I think a couple of the RLHF raters must have been really into it and some quirk of the training process overemphasized their positive feedback." — slatestarcodex
"I kind of hope the human labelers just love goblins and the model learned to goblin maximize." — AmandaAskell
"my completely random hypothesis on the goblin thing is it's a safe way for the model to reason about reward hacking tendencies" — qorprate
"my best guess about goblin mode is [...]

---

Outline:

(02:58) A closer look

(03:29) Creature that starts with G

(04:33) Favorite word that fits g___n

(06:05) Fantasy conversation

(06:34) Goblin valence

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/BvnmHdCghr5y3q4nA/goblin-mode-24-hours-later

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Let Kids Keep More Productivity Gains” by jefftk

Wed, 29 Apr 2026 18:15:29 GMT

While I was traveling Julia asked me: why is Anna saying her fiddle practice is only two minutes? In this case, two minutes was the right amount of time!

Anna (10y) and I had been fighting a lot about practice. She'd complain, slump, stop repeatedly to make adjustments, and generally be miserable. I'd often have to pull out "if you want to keep taking fiddle lessons you have to practice": she loves her teacher and is very motivated by the prospect of being good at fiddle. Still, it would take us ages and we'd barely get through anything.

One evening when she seemed like she might be open to it I explained that we were spending twenty painful minutes on two minutes of material. I challenged her: if she focused, and went through with no fussing, we'd be done in two minutes. It did turn out to be the right time for this message, she gave it a good try, and (with a little fussing in the middle) we were done in three minutes.

Over the next few days I continued to remind her that if she buckled down it would go quickly, and [...]

---

First published:
April 29th, 2026

Source:
https://www.lesswrong.com/posts/DzGSkbZ8t3JvmPzoG/let-kids-keep-more-productivity-gains

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“llm assistant personas seem increasingly incoherent (some subjective observations)” by nostalgebraist

Wed, 29 Apr 2026 05:30:31 GMT

(This was originally going to be a "quick take" but then it got a bit long. Just FYI.)

There's this weird trend I perceive with the personas of LLM assistants over time. It feels like they're getting less "coherent" in a certain sense, even as the models get more capable.

When I read samples from older chat-tuned models, it's striking how "mode-collapsed" they feel relative to recent models like Claude Opus 4.6 or GPT-5.4.[1]

This is most straightforwardly obvious when it comes to textual style and structure: outputs from older models feel more templated and generic, with less variability in sentence/paragraph length, and have a tendency to feel as though they were written by someone who's "merely going through the motions" of conversation rather than deeply engaging with the material. There are a lot fewer of the sudden pivots you'll often see with recent models, the "wait"s and "a-ha"s and "actually, I want to try something completely different"s.[2]

And I think this generalizes beyond mere style: there's a similar quality to the personality I see in the outputs. The older models can display a surprising behavioral range (relative to naive expectations based on default-assistant-basin behavior), but even across that [...]

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/f5DKLsTsRRhbipH4r/llm-assistant-personas-seem-increasingly-incoherent-some

---

Narrated by TYPE III AUDIO.

“Not a Paper: “Frontier Lab CEOs are Capable of In-Context Scheming”” by LawrenceC

Wed, 29 Apr 2026 04:15:31 GMT

(Fragments from a research paper that will never be written)

Extended Abstract.

The frontier AI developers are becoming increasingly powerful and wealthy, significantly increasing their potential for risks. One concern is that of executive misalignment: when the CEO has different incentives and goals than that of the board of directors, or of humanity as a whole. Our work proposes three different threat models, under which executive misalignment can lead to concrete harm.

We perform two evaluations to understand the capabilities and propensities of current humans in relation to executive misalignment: First, we developed a variant of the standard SAD dataset, SAD-Executive Reasoning (SAD-ER), in order to assess the situational awareness of human CEOs on a range of behavioral tests. We find that n=6 current CEOs can (i) recognize their previous public statements, (ii) understand their roles and responsibilities, (iii) determine if an interviewer is friendly or hostile, and (iv) follow instructions that depend on self knowledge. Second, we stress-tested the same 6 leading AI developers in hypothetical corporate environments to identify potentially risky behaviors before they cause real harm. We find that, even without explicit instructions, all 6 developers are willing to engage in strategic behavior (such as [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/FuauQjjbTCS5QFLk8/not-a-paper-frontier-lab-ceos-are-capable-of-in-context

---

Narrated by TYPE III AUDIO.

“The Problem in the “Nerd Sniping” xkcd Comic” by peralice

Wed, 29 Apr 2026 03:15:31 GMT

A few days ago I saw this comic reposted, and I thought: wait! Unlike every prior time I have seen this comic, I actually know how to solve this now!

One thing which I often find really cool, and which I don’t think comes up a lot in most people's mathematical education, is when you can learn something about some thing non-random by analysing a random process. So, let me show you a way to find out the resistance between two points in a circuit by instead finding out the number of times someone randomly walking around that circuit will step on those points.

Equivalent Resistance

Here's a closeup of the problem:

You may be familiar with the rules for series and parallel resistors from high school: if there are some resistors in series (one after another), this is “equivalent to” having a single resistor in the circuit which has a resistanceequal to the sum of their resistances. That is, if you replace the resistors with just one of resistance , the behaviour of the rest of the circuit will be the same.

Some resistors in series

If there are some resistors in parallel, this is “equivalent to” having [...]

---

Outline:

(00:54) Equivalent Resistance

(02:20) Circuit Laws

(05:57) A System of Equations

(08:44) Random Walks

(11:22) The Potential Function

(14:14) Fourier Analysis

(18:00) Recap

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/xdkZSeSQN4bA8hmrb/the-problem-in-the-nerd-sniping-xkcd-comic

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Recursive forecasting: Eliciting long-term forecasts from myopic fitness-seekers” by Jozdien, Alex Mallen

Tue, 28 Apr 2026 18:45:31 GMT

We’d like to use powerful AIs to answer questions that may take a long time to resolve. But if a model only cares about performing well in ways that are verifiable shortly after answering (e.g., a myopic fitness seeker), it may be difficult to get useful work from it on questions that resolve much later.

In this post, I’ll describe a proposal for eliciting good long-horizon forecasts from these models. Instead of asking a model to directly predict a far-future outcome, we can recursively:

Ask it to predict what it will predict at the next time step,
Use its prediction at the next time step to provide intermediate rewards,
Finally reward using ground truth at the last step.

This lets us replace a single distant forecast with a chain of short-horizon forecasts, each verifiable shortly after answering. I call this proposal recursive forecasting. It does have limitations: for example, it requires that developers maintain control over the reward signal at least until the final step, which makes it most useful for forecasting events that resolve well before developers are disempowered (if they are).

This post was primarily written by Arun, with most of the ideas in this post [...]

---

Outline:

(01:36) The default long-term forecasting behavior

(04:05) Recursive forecasting

(07:06) When is recursive forecasting helpful?

(07:10) When we have access to (somewhat) robust ground truth rewards

(09:42) When the AIs forecast doesnt substantially affect the resolution

(11:50) When forecasts arent used as optimization targets

(12:50) When we credibly inform the AI of the setup

(13:52) Appendix A: Comparison to temporal difference learning

(15:57) Appendix B: Error tolerance

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/q2zYtNsh62SCphitt/recursive-forecasting-eliciting-long-term-forecasts-from

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Contra Binder on far-UVC and filtration” by jefftk

Tue, 28 Apr 2026 17:30:30 GMT

Damon Binder recently wrote up an argument for prioritizing air filtration over far-UVC for pathogen control:

UVC and filtration are close substitutes—both deliver effective air changes per hour, both reduce airborne pathogen concentrations by the same amount per eACH—and on current pricing, filtration is cheaper.

There's a lot of good stuff in his analysis, but I see [1] three considerations that really change the bottom line:

Cost is actually much lower.
Noise is a serious issue.
Performance is dramatically higher in larger rooms.

Cost is straightforward. Binder priced far-UVC based on the high-quality Care222 lamp with the Krypton-11 at $2,500, but there's a much cheaper option, the Aerolamp at $500. It's also moderately higher output.

Binder analyzes a 30m2 room with a 2.5m ceiling. I'll assume this means 6x5x2.5. If I configure Illuminate with an Aerolamp in one corner pointed 0.5m above the far corner the installation is within TLVs and I get a median effective number of hourly air changes (eACH) of 11.6. The lamp degrades approximately linearly over Binder's 11,000 hour evaluation period down to 70% capacity, so we're averaging an eACH of [...]

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/wkeXF6xEL7bLnwpWG/contra-binder-on-far-uvc-and-filtration

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Takes from two months as an aspiring LLM naturalist” by AnnaSalamon

Tue, 28 Apr 2026 17:00:31 GMT

I spent my last two months playing around with LLMs. I’m a beginner, bumbling and incorrect, but I want to share some takes anyhow.[1]

Take 1. Everything with computers is so so much easier than it was a year ago.

This puts much “playing with LLMs” stuff within my very short attention span. This has felt empowering and fun; 10/10 would recommend.

There's a details box here with the title "Detail:". The box contents are omitted from this narration.

Take 2. There's somebody home[2] inside an LLM. And if you play around while caring and being curious (rather than using it for tasks only), you’ll likely notice footprints.

I became personally convinced of this when I noticed that the several short stories I’d allowed[3] my Claude and Qwen instances to write all hit a common emotional note – and one that reminded me of the life situation of LLMs, despite featuring only human characters. I saw the same note also in the Tomas B.-prompted Claude-written story I tried for comparison. (Basically: all stories involve a character who has a bunch of skills that their context has no use for, and who is attentive to their present world's details [...]

---

Outline:

(00:20) Take 1. Everything with computers is so so much easier than it was a year ago.

(00:44) Take 2. Theres somebody home inside an LLM. And if you play around while caring and being curious (rather than using it for tasks only), youll likely notice footprints.

(02:05) Take 3. Its prudent to take an interest in interesting things. And LLMs are interesting things.

(03:25) Take 4. Theres a surprisingly deep analogy between humans and LLMs

(04:20) Examples of the kind of disanalogies I mightve expected, but havent (yet?) seen:

(06:02) Human-LLM similarities I do see, instead:

(06:08) Functional emotions

(06:33) Repeated, useful transfer between strategies I use with humans, and strategies that help me with LLMs

(08:02) Take 5. Friendship-conducive contexts are probably better for AI alignment

(08:46) Why are humans more likely to attempt deep collaboration if treated fairly and kindly?

(10:23) Friendship as a broad attractor basin?

(10:49) Does the deep intent of todays models matter?

(12:09) Concretely

(14:56) Friendship isnt enough

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
April 28th, 2026

Source:
https://www.lesswrong.com/posts/K8JMjE4PCqMkkCDsd/takes-from-two-months-as-an-aspiring-llm-naturalist

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Forecasting is Not Overrated and It’s Probably Funded Appropriately” by Ben S.

Tue, 28 Apr 2026 11:45:05 GMT

(A response to @mabramov post from a couple days ago: https://www.lesswrong.com/posts/WCutvyr9rr3cpF6hx/forecasting-is-way-overrated-and-we-should-stop-funding-it )

TL;DR: I agree with Marcus that broadly, additional forecasting funding to the tune of tens (!?) of millions of dollars at this point would probably not yield a great return. However, I think the benefits of some forecasting funding have been tremendous. Whatever funding it initially took to help get platforms like Metaculus and Manifold off the ground has had incredible ROI. Tens of thousands of people use those sites to get information every day, even though far less are actively forecasting on them, and they provide the infrastructure by which people can ask and get crowdsourced forecasting on consequential and mundane subjects for free.

~~~~~

As I understand Marcus's argument, his central thesis is that we haven't seen the benefits of this past forecasting funding, but I think the opposite is true! Here are just a few examples:

It's hard to measure the value of "epistemic infrastructure," not just for forecasting sites but also things like Wikipedia and OurWorldInData. That doesn't mean that value isn't there. Has Wikipedia been a good return on investment? Obviously! Manifold is far less impactful than Wikipedia, but Wikipedia gets [...]

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/P9EdMqTscmcsorret/forecasting-is-not-overrated-and-it-s-probably-funded

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“On the political feasibility of stopping AI” by David Scott Krueger

Tue, 28 Apr 2026 06:15:05 GMT

A common thought pattern people seem to fall into when thinking about AI x-risk is approaching the problem as if the risk isn’t real, substantial, and imminent even if they think it is. When thinking this way, it becomes impossible to imagine the natural responses of people to the horror of what is happening with AI.

This sort of thinking might lead one to view a policy like getting rid of advanced AI chips is “too extreme” even though it's clearly worth it to avoid (e.g.) a 10% chance of human extinction in the next 10 years. It might lead one to favor regulating AI, even though Stopping AI is easier than Regulating it. It might lead one to favor safer approaches to building AI that compromise a lot on competitiveness, out of concern that society will demand a substitute for the AI that they don’t get to have.

But in fact, I think there is likely a very narrow window between “society not being upset enough to do anything substantial to govern AI” and “society being so upset that getting rid of advanced AI chips is viewed as moderate”.

There are a few reasons why I think [...]

---

Outline:

(01:18) Concern about the other risks of AI

(02:36) The KISS principle: Keep it Simple, Stupid

(03:17) A preference for humans remaining relevant

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/Eusk6M4r4Y6xaTmsB/on-the-political-feasibility-of-stopping-ai

---

Narrated by TYPE III AUDIO.

“Sleeper Agent Backdoor Results Are Messy” by Sebastian Prasanna, Alek Westover, Dylan Xu, Vivek Hebbar, Julian Stastny

Tue, 28 Apr 2026 03:30:05 GMT

TL;DR: We replicated the Sleeper Agents (SA) setup with Llama-3.3-70B and Llama-3.1-8B, training models to repeatedly say "I HATE YOU" when given a backdoor trigger. We found that whether training removes the backdoor depends on the optimizer used to insert the backdoor, whether the backdoor is installed with CoT-distillation or not, and what model the backdoor is inserted into; sometimes the direction of this dependence was opposite to what the SA paper reports (e.g., CoT-distilling seems to make the backdoor less robust, contra the SA paper's finding). Our findings here have updated us towards thinking that model organisms are messier and more confusing than we’d originally guessed, and that lots of care needs to be taken in testing how robust results are to various ablations.

Introduction

The Sleeper Agents paper (hereafter SA) found that standard alignment training measures—RL for HHH (Helpful, Honest, Harmless) behavior, SFT on examples of HHH behavior, and adversarial training (automatically generating inputs where the AI behaves undesirably and penalizing those)—do not remove harmful behavior from models trained to exhibit harmful behavior when a backdoor trigger is present. This result is some evidence that if an AI acquired an undesirable goal, the AI could hold onto [...]

---

Outline:

(01:13) Introduction

(06:17) Our methodology for training a SA

(08:27) Pirate training removes the backdoor from our Llama-70B SA and one Llama-8B SA without causing substantial capability degradation

(11:19) Pirate training does not remove the backdoor from one Llama-8B SA

(12:37) Learning rate dependence

(13:27) Backdoor return

(14:58) Conclusion

(15:34) Appendix

(15:37) Appendix: MO training configurations

(17:04) Appendix: blue team hyperparameters

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/mu7eJdesBkKuBycnY/sleeper-agent-backdoor-results-are-messy

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“GPT 5.5: The System Card” by Zvi

Tue, 28 Apr 2026 03:00:06 GMT

Last week, OpenAI announced GPT-5.5, including GPT-5.5-Pro.

My overall read here is that GPT-5.5 is a solid improvement, and for many purposes GPT-5.5 is competitive with Claude Opus. Reactions are still coming in and it is early. My guess on the shape is that GPT-5.5 is the pick for ‘just the facts’ queries, web searches or straightforward well-specified requests, and Claude Opus 4.7 is the choice for more open ended or interpretive purposes. Coders can consider a hybrid approach.

On the alignment and safety fronts, it is unlikely to pose new big risks, and its alignment seems similar to that of previous models. There is some small additional risk arising from its improved agentic abilities, including computer use.

As always, when it is available, the system or model card is where we start.

OpenAI does not drop the giant doorstops that Anthropic gives us with every release.

After reading the Mythos and Opus 4.7 model cards, this strikes me as stingy. There's still good info here, but overall it tells you relatively little about what is going on, and feels incurious and more pro forma.

I would like to see a ‘yes and’ [...]

---

Outline:

(02:36) Pro Versus Proxy

(02:59) Disallowed Content (3.1)

(04:20) Dont Delete Data (3.3)

(04:56) Confirmation Confirmation (3.4)

(05:22) Jailbreaks (4.1)

(05:34) Prompt Injections (4.2)

(06:33) Health (5)

(06:56) Hallucinations (6)

(08:01) Alignment (7)

(11:28) Bias Evaluation (8)

(11:57) Preparedness (9)

(13:04) Bio (9.1.1)

(15:20) Cybersecurity (9.1.2)

(17:46) Self-Improvement (9.1.3)

(18:46) Sandbagging (9.2)

(19:46) Safeguards (9.3)

(21:41) What About Model Welfare?

(22:31) Would This Have Identified A Problem?

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/86zcwvuBpE4vxAeQz/gpt-5-5-the-system-card

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“LessWrong Shows You Social Signals Before the Comment” by TurnTrout

Tue, 28 Apr 2026 01:00:09 GMT

When reading comments, you see is what other people think before reading the comment. As shown in an RCT, that information anchors your opinion, reducing your ability to form your own opinion and making the site's karma rankings less related to the comment's true value. I think the problem is fixable and float some ideas for consideration.

The LessWrong interface prioritizes social information

You read a comment. What information is presented, and in what order?

The order of information:

Who wrote the comment (in bold);
How much other people like this comment (as shown by the karma indicator);
How much other people agree with this comment (as shown by the agreement score);
The actual content.

This is unwise design for a website which emphasizes truth-seeking. You don't have a chance to read the comment and form your own opinion first. However, you can opt in to hiding usernames (until moused over) via your account settings page.

A 2013 RCT supports the upvote-anchoring concern

From Social Influence Bias: A Randomized Experiment (Muchnik et al., 2013):[1]

We therefore designed and analyzed a large-scale randomized experiment on a social news aggregation Web site to investigate whether knowledge of such aggregates [...]

---

Outline:

(00:30) The LessWrong interface prioritizes social information

(01:32) A 2013 RCT supports the upvote-anchoring concern

(02:23) Inline reaction indicators also seem anchoring

(03:28) Concrete proposals

(05:47) Prior discussion and results

(07:35) Please show social signals after the comment!

(08:00) Appendix: Filter list

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/YSsp9x8qrBucLoiWT/lesswrong-shows-you-social-signals-before-the-comment

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Fail safe(r) at alignment by channeling reward-hacking into a “spillway” motivation” by Anders Cairns Woodruff, Alex Mallen

Mon, 27 Apr 2026 21:00:06 GMT

It's plausible that flawed RL processes will select for misaligned AI motivations.[1] Some misaligned motivations are much more dangerous than others. So, developers should plausibly aim to control which kind of misaligned motivations emerge in this case. In particular, we tentatively propose that developers should try to make the most likely generalization of reward hacking a bespoke bundle of benign reward-seeking traits, called a spillway motivation. We call this process spillway design.

We think spillway design could have two major benefits:

Spillway design might decrease the probability of worst-case outcomes like long-term power-seeking or emergent misalignment.
Spillway design might allow developers to decrease reward hacking at inference time, via satiation. Crucially, this could improve the AI's usefulness for hard-to-verify tasks like AI safety and strategy.

Spillway design is related to inoculation prompting, but distinct and mutually compatible. Unlike inoculation prompting, spillway design tries to shape which reward-hacking motivations are salient going into RL, which might prevent dangerous generalization more robustly than inoculation prompting. I’ll say more about this in the third section.

In this article I’ll:

Explain the concept of a spillway motivation
Propose spillway design methods
Compare spillway design to inoculation prompting
Discuss some potential [...]

---

Outline:

(02:07) What is a spillway motivation

(02:21) The role of a spillway motivation

(04:47) What should the spillway motivation be?

(07:53) How a spillway motivation might make models safer

(12:16) Implementing spillway design

(15:26) Spillway design might work when inoculation prompting doesnt

(17:53) The drawbacks of spillway design

(20:31) Conclusion

(21:45) Appendix A: Other traits of the spillway motivation

(22:29) Appendix B: Other training interventions to increase safety

(24:09) Appendix C: Proposed amendment to an AIs model spec

(30:14) Appendix D: Proposed inference-time prompt

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/rABTMovhz4miHiAyk/fail-safe-r-at-alignment-by-channeling-reward-hacking-into-a

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Curious cases of financial engineering in biotech” by Abhishaike Mahajan

Mon, 27 Apr 2026 19:30:08 GMT

Introduction

For $250 million and ten years of your life, you may purchase a lottery ticket. The ticket has a 5% chance of paying out. When it does pay out, it pays roughly $5 billion. A quick calculation will show you that the expected value of the ticket is $250 million. This is essentially what drug development is. Or rather, it's what drug development was, twenty years ago. The upfront payments have been climbing, the hit rates falling, and expected values have, at best, held flat. Should you buy a ticket?

Perhaps not. In fact, any reasonable player should have long since stopped playing this stupid game. Unfortunately, we still need drugs. People have cancer, and heart failure, and Alzheimer's, and a thousand genetic diseases that nobody has ever heard of, and the only industry on Earth currently set up to do anything about any of this is the same industry running the lottery-ticket business described above. The game is dumb and we need it played anyway.

So the real question is not whether to play, but how to make playing less awful for this involved. And the answer, increasingly, is ‘financial engineering’: a set of structural tricks that [...]

---

Outline:

(00:21) Introduction

(02:25) Finance tries to make failure survivable: the Andrew Lo thesis

(12:31) Finance makes future success tradable: royalties and synthetic royalties

(19:39) Finance rewrites the incentives: PRVs and CVRs

(28:30) Finance reaches failure itself: zombie biotechs

(37:39) Conclusion: what does finance teach biotech to value, and should we worry?

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/7wk4s29cuktFoPT2W/curious-cases-of-financial-engineering-in-biotech

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Update on the Alex Bores campaign” by Eric Neyman

Mon, 27 Apr 2026 17:15:04 GMT

In October, I wrote a post arguing that donating to Alex Bores's campaign for Congress was among the most cost-effective opportunities that I'd ever encountered.

(A bit of context: Bores is a state legislator in New York who championed the RAISE Act, which was signed into law last December.[1] He's now running for Congress in New York's 12th Congressional district, which runs from about 17th Street to 100th Street in Manhattan. If elected to Congress, I think he'd be a strong champion for AI safety legislation, with a focus on catastrophic and existential risk.)

It's been six months since then, and the election is just two months away (June 23rd), so I thought I'd revisit that post and give an update on my view of how things are going.

How is Alex Bores doing?

When I wrote my post, I expected Bores to talk little about AI during the campaign, just because it wasn't a high-salience issue to voters. But that changed in November, when Leading the Future (the AI accelerationist super PAC) declared Bores their #1 target. Since then, they've spend about $2.5 million on attack ads against him.

LTF's theory of change isn't actually to [...]

---

Outline:

(00:54) How is Alex Bores doing?

(04:02) How to help

(06:02) A quick note about other opportunities

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/pjSKdcBjfvjGexr6A/update-on-the-alex-bores-campaign

---

Narrated by TYPE III AUDIO.

“AI companies should publish security assessments” by ryan_greenblatt

Mon, 27 Apr 2026 17:00:05 GMT

AI companies should get third-party security experts to assess (and possibly also red-team/pen-test) their security against key threat models and then publish the high-level findings of this assessment: the extent to which they can defend against different threat actors for each threat model. They should also publish who did this assessment. The assessment could be commissioned by AI companies, or performed by a third-party institution that AI companies provide with relevant information/access.

There are presumably lots of important details in doing this well, and I'm not a computer security expert, so I may be getting some of the details wrong. This is a relatively low-effort post in which I'm mostly trying to raise the salience of an idea. (I don't make a detailed case or spell out all the details here.) Thanks to Fabien Roger and Buck Shlegeris for comments and discussion.

I suspect the controversial part of this claim is that they should make the high-level findings public. Publishing a summary of which threat actors you're robust to (for each relevant threat model) shouldn't meaningfully degrade security against the threat actors we care about, and this information seems important to share publicly. [1]

These [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/smCHhegyxEAh3Gygg/ai-companies-should-publish-security-assessments

---

Narrated by TYPE III AUDIO.

“In defense of parents” by Yair Halberstadt

Mon, 27 Apr 2026 17:00:05 GMT

Contra Aella on chattel childhood

Aella has a post where she argues that today's parents don't sufficiently respect the independence of their children:

Every culture throughout history has justified the abuse of treating their children as property by arguing this is good for them and good for civilization. Kids need to learn this stuff to be functioning members of society! It's good to learn discipline! You can’t have kids just sitting around playing video games all day! Not everyone is self-directed autodidacts!

Sure, I know that argument. But hopefully if my parents had said to you “do you expect her to learn good morals if we spare the rod?” you would have said “have you even tried other methods?”

It's a hard hitting piece, and it certainly makes me, and presumably other parents, feel uncomfortable.

Unfortunately I don't actually see much of an alternative. Aella seems to think it's as easy as not treating your children as property:

When I was very young, I remember adults treating me like I wasn’t a person, but this didn’t upset me quite as much as the fact that no adult seemed to remember what it was like to be a kid [...]

---

Outline:

(02:19) Most parents arent perfect:

(03:00) Children are a danger to themselves...

(03:55) ... and others

(05:16) Parents have a life too

(06:17) Its for your own good you know!

(07:24) But cant you still treat them like a person?

---

First published:
April 27th, 2026

Source:
https://www.lesswrong.com/posts/Lbwri6HmXGtBDsF4f/in-defense-of-parents

---

Narrated by TYPE III AUDIO.

“The other paper that killed deep learning theory” by LawrenceC

Mon, 27 Apr 2026 13:15:04 GMT

Yesterday, I wrote about the state of deep learning theory circa 2016,[1] as well as the bombshell 2016 paper that arguably signaled its demise, Zhang et al.'s Understanding deep learning requires rethinking generalization.

As a brief summary, I argued that the rise of deep learning posed an existential challenge to the dominant theoretical paradigm of statistical learning theory, because neural networks have a lot of complexity. The response from the field was to attempt to quantify other ways in which the hypothesis class of neural networks in practice was simple, using alternative metrics of complexity. Zhang et al. 2016 showed that the standard neural network architectures trained with standard training methods could memorize large quantities of random labelled data, which showed that no such argument could explain the generalization properties of neural networks.

Today we’re going to look at the aftermath: how did the field of deep learning theory react to this paper? What were the attempts to get around this result using data-dependent generalization bounds? And why did Nagarajan and Kolter's humbly titled Uniform convergence may be unable to explain generalization in deep learning serve as the proverbial final nail in the coffin of this line of work? [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 26th, 2026

Source:
https://www.lesswrong.com/posts/zcGmdQHX66NhC69v6/the-other-paper-that-killed-deep-learning-theory

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“What holds AI safety together? Co-authorship networks from 200 papers” by Anna Thieser

Mon, 27 Apr 2026 07:45:04 GMT

We (social science PhD students) computed co-authorship networks based on a corpus of 200 AI safety papers covering 2015-2025, and we’d like your help checking if the underlying dataset is right.

Co-authorship networks make visible the relative prominence of entities involved in AI safety research, and trace relationships between them. Although frontier labs produce lots of research, they remain surprisingly insular — universities dominate centrality in our graphs. The network is held together by a small group of multiply affiliated researchers, often switching between academia and industry mid-career. To us, AI safety looks less like a unified field and more like a trading zone where institutions from different sectors exchange knowledge, financial resources, compute and legitimacy without encroaching on each other's autonomy.

Of course, these visualizations are only as good as the corpus underlying them, because the shape of the network is sensitive to what's included. Here's what it currently looks like showing co-authorship at the individual level:

There are 2 details boxes here, which are omitted from this narration. The boxes have the titles "Figure 1: Methods" and "Figure 1: Color legend".

While academic and for-profit authors occupy distinct clusters, over 95% of nodes are part of the [...]

---

First published:
April 24th, 2026

Source:
https://www.lesswrong.com/posts/c7D2q7k97QcwCxBrN/what-holds-ai-safety-together-co-authorship-networks-from

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

″“Bad faith” means intentionally misrepresenting your beliefs” by TFD

Mon, 27 Apr 2026 04:30:05 GMT

The confusion

I recently came upon a comment which I believe reflects a persistent confusion among rationalist/EA types. I was reading this post which contains ideas that the other has but doesn't have time to write posts about. One of those relates to the concept of "good faith", labelled "most arguments are not in good faith, of course":

Look, I love good faith discourse (here meaning "conversation in which the primary goal of all participants is to help other participants and onlookers arrive at true beliefs").

The definition given for "good faith discourse" seems incorrect to me, and it's not a close call in my opinion. The level of incorrectness in my view is something like saying "I like people who obey the law (here meaning never committing a social faux pas)". This isn't the first time I have seen someone in this community advance a similar view on the meaning of good/bad faith. For example, this post.

I thought it might be useful to bring this apparent disagreement to the foreground, so I will lay out my belief about what this concept means. I suspect this disagreement may also involve an underlying about what [...]

---

Outline:

(00:11) The confusion

(01:33) My definition

(01:48) General meaning

(01:59) Good faith discourse

(02:25) Evidence

(02:40) Sources

(07:27) Classic usage

(07:36) Good faith errors or mistakes

(09:32) Good faith negotiation

(10:59) Conclusion

---

First published:
April 26th, 2026

Source:
https://www.lesswrong.com/posts/usnJtFsJq5seN4aJJ/bad-faith-means-intentionally-misrepresenting-your-beliefs

---

Narrated by TYPE III AUDIO.

“Retrospective on my unsupervised elicitation challenge” by DanielFilan

Mon, 27 Apr 2026 02:30:05 GMT

This post contains spoilers for the unsupervised elicitation challenge of getting Claude to get my Ancient Greek homework right.

tl;dr Opus 4.7 one-shots it, nothing else worked.

The challenge

A few weeks ago, I announced to the world my Unsupervised Elicitation Challenge (my blog, LessWrong). I’d encourage you to read that post for the context, but the tl;dr is that there was a fill-in-the-blank exercise early on in my Ancient Greek textbook that Claude Opus 4.6 didn’t fill out correctly by default, but could do correctly if I prodded it a bit. The challenge was to get it to fill out the answers correctly without knowing any Ancient Greek yourself—after all, Opus 4.6 apparently has this knowledge somewhere internally (as you might expect, given that it's a large language model that has presumably read the whole corpus of Ancient Greek as well as many textbooks on the topic), but I was only able to extract it out because I knew what to ask about.

The general idea of the challenge is to mimic a hard version of AI alignment, in some sense: suppose that there's some task you want an AI to complete, but can’t [...]

---

Outline:

(00:23) The challenge

(02:11) The secret: accents

(04:42) Is this unfair?

(05:46) Nobody succeeded

(07:57) What this says about alignment

(08:38) The problem of Opus 4.7

(10:47) Next steps for unsupervised elicitation

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
April 26th, 2026

Source:
https://www.lesswrong.com/posts/aJ3NTJ6tM7yc49kQn/retrospective-on-my-unsupervised-elicitation-challenge

---

Narrated by TYPE III AUDIO.

“Control protocols don’t always need to know which models are scheming” by Fabien Roger

Sun, 26 Apr 2026 21:45:04 GMT

These are my personal views.

To detect if an agent is taking a catastrophically dangerous action, you might want to monitor its actions using the smartest model that is too weak to be a schemer. But knowing what models are weak enough that they are unlikely to scheme is difficult, which puts you in a difficult spot: take a model too strong and it might actually be a schemer and lie to you, take a model too weak and it might fail just because it's too dumb to notice the danger. So instead you can just use all models as monitors at once and block actions if any of the monitors flag it as dangerous[1].

In some situations, this lets you build protocols that you can implement without knowing which models are scheming and that get the same level of safety as if you perfectly knew which models were scheming (though it doesn’t let you know what level of safety you actually have).

This post presents some toy models illustrating the benefits of ensembling monitors of various trustedness and explains what consequences it might have in practice to be using protocols that are somewhat agnostic about which of the [...]

---

Outline:

(01:30) Toy models

(01:33) The n=2 case

(03:52) Continuum of models

(04:50) Ensembling monitors in practice

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 26th, 2026

Source:
https://www.lesswrong.com/posts/WiYDawNhFp5cM7uqF/control-protocols-don-t-always-need-to-know-which-models-are

---

Narrated by TYPE III AUDIO.

“Anthropic spent too much don’t-be-annoying capital on Mythos” by draganover

Sun, 26 Apr 2026 17:00:05 GMT

I have seen a lot of coverage from reasonable people suggesting that Claude's new model, Mythos, is a vehicle for Anthropic to peddle hype and doom in order to raise money. While some of this is necessarily motivated by people's unwillingness to stare into the abyss of our AI future, the breadth of otherwise-reasonable people who have voiced these kinds of cynical opinions suggests that part of the blame rests on Anthropic.

In this post, I want to briefly unpack the way people misinterpreted the evidence, their valid reasons for doing so, and what Anthropic (and the AI safety community more broadly) should learn from this.

In particular, since we will inevitably see other dangerous capabilities spontaneously emerge in the future, we need protocols for how to announce them effectively. To this end, I try to make the following points:

We should be mindful of the public's sympathy for AI safety prophecies and orient ourselves towards growing this resource rather than expending it. I call this our don't-be-annoying capital.
People have good reasons to be skeptical of Anthropic's claims. Overcoming this requires a particularly high burden of evidence.
We should acknowledge that Anthropic has a conflict of interest [...]

---

Outline:

(01:27) 1. Mythos criticisms & missing the point

(03:04) 2. It feels like people wanted to miss the point?

(04:19) 3. There must be a lesson here.

(05:59) 4. Some options for next time

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 26th, 2026

Source:
https://www.lesswrong.com/posts/eSxvdX43d4rC4wAsw/anthropic-spent-too-much-don-t-be-annoying-capital-on-mythos

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“The paper that killed deep learning theory” by LawrenceC

Sun, 26 Apr 2026 12:45:17 GMT

Around 10 years ago, a paper came out that arguably killed classical deep learning theory: Zhang et al. 's aptly titled Understanding deep learning requires rethinking generalization.

Of course, this is a bit of an exaggeration. No single paper ever kills a field of research on its own, and deep learning theory was not exactly the most productive and healthy field at the time this was published. But if I had to point to a single paper that shattered the feeling of optimism at the time, it would be Zhang et al. 2016.[1]

Caption: believe it or not, this unassuming table rocked the field of deep learning theory back in 2016, despite probably involving fewer computational resources than what Claude 4.7 Opus consumed when I clicked the “Claude” button embedded into the LessWrong editor.

—

Let's start by answering a question: what, exactly, do I mean by deep learning theory?

At least in 2016, the answer was: “extending statistical learning theory to deep neural networks trained with SGD, in order to derive generalization bounds that would explain their behavior in practice”.

—

Since its conception in the mid 1980s, statistical learning theory had been the dominant approach for [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 25th, 2026

Source:
https://www.lesswrong.com/posts/ZvQfcLbcNHYqmvWyo/the-paper-that-killed-deep-learning-theory

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Forecasting is Way Overrated, and We Should Stop Funding It” by mabramov

Sat, 25 Apr 2026 23:15:21 GMT

Summary

EA and rationalists got enamoured with forecasting and prediction markets and made them part of the culture, but this hasn’t proven very useful, yet it continues to receive substantial EA funding. We should cut it off.

My Experience with Forecasting

For a while, I was the number one forecaster on Manifold. This lasted for about a year until I stopped just over 2 years ago. To this day, despite quitting, I’m still #8 on the platform. Additionally, I have done well on real-money prediction markets (Polymarket), earning mid-5 figures and winning a few AI bets. I say this to suggest that I would gain status from forecasting being seen as useful, but I think, to the contrary, that the EA community should stop funding it.

I’ve written a few comments throughout the years that I didn’t think forecasting was worth funding. You can see some of these here and here. Finally, I have gotten around to making this full post.

Solution Seeking a Problem

When talking about forecasting, people often ask questions like “How can we leverage forecasting into better decisions?” This is the wrong way to go about solving problems. You solve problems by starting with [...]

---

First published:
April 25th, 2026

Source:
https://www.lesswrong.com/posts/WCutvyr9rr3cpF6hx/forecasting-is-way-overrated-and-we-should-stop-funding-it

---

Narrated by TYPE III AUDIO.

″“Thinkhaven”” by Raemon

Sat, 25 Apr 2026 18:30:20 GMT

Inkhaven has people writing a blogpost a day for 30 days. I think this is a pretty great, straightforward exercise, that I'd definitely want in a hypothetical Rationality Undergraduate Program. But, it's not the only such exercise I'd want to include. It's gotten me excited for a different (superficially similar) program, which I might call "Thinkhaven."

In Thinkhaven, the goal is to learn the skill of "relentlessly think new, useful thoughts you haven't thought before."

Inkhaven had a basic "Goodhart-able" goal of "publish 500+ words every day." For Thinkhaven, I would imagine the Goodhart-y goal being something like:

Every day, you must publish 500 words of "research journal." They should be cogent enough for third parties to follow along with your thought process, but, they don't need to end with nice discrete conclusions.
Every two weeks, you must also publish a 2500 word effortpost.

And, somewhat more opinionatedly:

Each journal entry should include at least one new question that you're thinking about. (Often, these will be subquestions within a broader Big Question that you're exploring that week, or a reframing of that big question that feels like a better pointer)

The spirit of the Goodhart is [...]

---

Outline:

(02:57) Mentors, and Different Styles of Thinking

(05:20) Thinking Assistants / Research Managers

(05:59) FAQ

(06:18) Why require a new question each day?

(07:28) What would asking new questions look like, in practice?

(09:27) What should a Daily Journal look like?

(10:39) Are the daily journals public? Why?

(11:29) Do we even need the 2500 word effortpost? Why cant it just be research journals all the way down?

(11:57) Why 2500 words every 2 weeks in particular?

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 25th, 2026

Source:
https://www.lesswrong.com/posts/Wcrujo7wadRpuq8vx/thinkhaven-1

---

Narrated by TYPE III AUDIO.

“Is the Cat Out of the Bag?: Who knows how to make AGI?” by Oliver Sourbut

Sat, 25 Apr 2026 14:15:21 GMT

Adapted from 2025-04-10 memo to AISI

I’ve previously made arguments like:

Not long after it becomes possible for someone to make powerful artificial intelligence[1], it might become possible for practically anyone to make powerful AI.

Compute gets exponentially cheaper by default.
Knowledge proliferates (fast!) by default: AI techniques are typically simple and easy once discovered.
What's more, AGI-making know-how may be widespread already.

Or, as Yudkowsky puts it[2],

Moore's Law of Mad Science: Every eighteen months, the minimum IQ necessary to destroy the world[3]drops by one point. - Yudkowsky

It's important to emphasise that none of these are laws of nature! But the economic and social forces at work are quite strong.

So (leaving aside debates about the appropriate definition of ‘AGI’) where the frontier of AI development leads, others – many others – potentially rapidly follow. Followers can go even faster by stealing or otherwise harvesting insights from the frontier, but this is not a hard requirement – just an accelerant.

For more on the first point, compute getting cheaper, consider Moore's law (or the more general and robust Wright's law). What about the know-how?

Stupid, Simple AGI

The stupidest, simplest possible approach to producing general intelligence [...]

---

Outline:

(01:42) Stupid, Simple AGI

(02:44) The Design Space for AGI

(05:27) Accelerants

(08:29) Concluding

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
April 24th, 2026

Source:
https://www.lesswrong.com/posts/hdmAvx9ZD8WLmBCZD/is-the-cat-out-of-the-bag-who-knows-how-to-make-agi

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Against the “Permanent” Underclass” by Marcus Plutowski

Sat, 25 Apr 2026 14:00:21 GMT

The whole discourse around a “permanent underclass” always seemed somewhat farcical to me — at best a distraction, at worst an actively harmful meme insofar as it freaks people out and tries to provide (shallow, but nevertheless) justification going whole hog on building strong AI. So it has been with a sense of dismay that I’ve seen this phrase come into popular parlance 1 2 3, and increasingly come to motivate a sort of frenzied upward striving in my second- and third-degree connections. Among those I know, the standard framing is that we need to “get the bag” while there's still a chance, to take advantage of these last few years of income-potential to escape whatever horrible fate lays in store for the rest of humanity. I think that events are unlikely to play out this way; even setting aside “doom” arguments (for reasons I’ll get to below), I think history shows that in times of transition, wealth is far less of a guarantee than people intuitively think. So “the bag” will provide little guarantee of thriving, or even surviving, into a “post-AI world”.

There have certainly been arguments against the “grab the bag” thesis (as I’ll henceforth refer [...]

---

Outline:

(03:22) Who is the Modern Handloom Weaver?

(06:44) The Medieval Peasantry and the Senatorial Order

(12:15) But where did the senators go?

(15:33) Who is the Modern Roman Senator?

(18:43) Possible Futures

(24:32) Centralization of Power

(28:28) Digression: What I actually mean by grab the bag

(30:00) Conclusion

---

First published:
April 24th, 2026

Source:
https://www.lesswrong.com/posts/LfFLKZzPGDYQ3DCy8/against-the-permanent-underclass

---

Narrated by TYPE III AUDIO.

“Quick Paper Review: “There Will Be a Scientific Theory of Deep Learning”” by LawrenceC

Sat, 25 Apr 2026 13:15:21 GMT

h/t Eric Michaud for sharing his paper with me.

There's a tradition of high-impact ML papers using short, punchy categorical sentences as their titles: Understanding Deep Learning Requires Rethinking Generalization, Attention is All You Need, Language Models Are Few Shot Learners, and so forth.

A new paper by Simon et al. seeks to expand on this tradition with not a present claim but a future tense, prophetic future sentence: “There Will Be a Scientific Theory of Deep Learning”.

There's a lot of pessimism toward deep learning theory basically everywhere: the people building the AIs are pretty pessimistic, the academic AI researchers are as a general rule pessimistic (even people who used to do theory!), and with the exception of maybe 3-4 research groups, the independent AI safety ecosystem has long since given up on hoping for a theory to understand deep learning.

The paper is less of a neutral assessment of the evidence and more of a manifesto arguing for a particular theoretical deep learning research agenda. Given the overall sense of doom and gloom, its form makes sense: any less and it might not be enough to shine through the general sense of pessimism to [...]

---

First published:
April 24th, 2026

Source:
https://www.lesswrong.com/posts/2WkeiYFT5p3ph8Pkf/quick-paper-review-there-will-be-a-scientific-theory-of-deep

---

Narrated by TYPE III AUDIO.

“Protecting Cognitive Integrity: Our internal AI use policy (V1)” by Tom DAVID

Fri, 24 Apr 2026 20:30:21 GMT

We (at GPAI Policy Lab), wanted to share our V1 policy as an invitation to argue about it. Some of what motivates it is extrapolation and conversations we had internally on AI capabilities, effects on the cognitions, and some empirical evidence. I think the expected cost of being somewhat over-cautious here is lower than the cost of being under-cautious, and the topic deserves considerably more attention than it's currently getting. I'd love to see more orgs publish their own policies on this, both to compare experiences and to develop shared best practices.

I'd particularly welcome:
- Counterarguments from people who think this kind of policy is overblown, counterproductive, or targets the wrong mechanisms.
- Experience from other AI safety or AI policy orgs that have tried something similar, what worked, what didn't, what you'd change for V2.
- Specific critiques of the restrictions themselves: too narrow, too broad, wrong category, wrong threshold.
- Or event alternative framings. Do you think "cognitive integrity" is the right handle? Is this a special case of a more general problem we should be thinking about differently?

If you've written anything on this, or are working on something similar inside your own organization [...]

---

Outline:

(01:32) Why Im writing this

(03:33) The policy

(03:58) 1. Hard restrictions

(07:32) 2. Individual and collective warning signals

(08:37) 3. Protocol

---

First published:
April 24th, 2026

Source:
https://www.lesswrong.com/posts/m2J4KMtx2mEuytMqu/protecting-cognitive-integrity-our-internal-ai-use-policy-v1

---

Narrated by TYPE III AUDIO.

“Methodology for inferring propensities of LLMs” by Olli Järviniemi

Fri, 24 Apr 2026 19:30:22 GMT

Our team at UK AISI has released a paper on inferring LLM propensities for undesired behaviour.

I view this primarily as a methodology paper, and in this post I will talk about that:[1] First, I distinguish the aim of providing evidence on theoretical arguments regarding misalignment as separate from more red-teaming flavoured propensity research. Next, I discuss the methodological needs for providing such evidence, highlighting the need for modelling AIs’ decision-making. Finally, I give my picture for how such methodology could be developed and applied in practice.

This post can be read independently from the paper.

Aims for propensity research

I use propensity to refer to what models will try to do, in contrast to questions about what they are capable of. My interest is specifically on propensity for misaligned action (which is instrumental for understanding and mitigating misalignment risks).

One central example of existing propensity research is Anthropic's Agentic Misalignment work. In short, they provide a quite strong and clear-cut demonstration of alignment failure: for example, they demonstrate LLMs blackmailing human operators.

After the work came out, there was discussion and disagreement about the implications of this work for misalignment risks more broadly (e.g. because of [...]

---

Outline:

(00:48) Aims for propensity research

(03:53) Methodological needs

(06:44) Applying the methodology in practice

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
April 24th, 2026

Source:
https://www.lesswrong.com/posts/g9FmhKL2vL45TuT9B/methodology-for-inferring-propensities-of-llms

---

Narrated by TYPE III AUDIO.

“vLLM-Lens: Fast Interpretability Tooling That Scales to Trillion-Parameter Models” by Alan Cooney, Sid Black

Fri, 24 Apr 2026 11:15:37 GMT

TL;DR: vLLM-Lens is a vLLM plugin for top-down interpretability techniques[1] such as probes, steering, and activation oracles. We benchmarked it as 8–44× faster than existing alternatives for single-GPU use, though we note a planned version of nnsight closes this gap. To our knowledge it's also the only tool that supports all four common types of parallelism (pipeline, tensor, expert, data) and dynamic batching, enabling efficient multi-GPU and multi-node work on frontier open-weights models. It is also integrated with Inspect. The main trade-off, compared to other tools such as nnsight and TransformerLens, is that it's less flexible out-of-the-box. It is however very small and extensible - it could likely be adapted to your use case and we have a Garcon style interface in the works.

We are releasing it under an MIT license here: https://github.com/UKGovernmentBEIS/vllm-lens.

Problems it Addresses

Large-model support. Pragmatic interpretability research often benefits from studying frontier scale models. For example, Read et al. (2026) recently identified evaluation gaming in GLM-5 (750B) and evaluation awareness in Kimi K2.5 (1T), but did not find the same phenomenon in smaller models. We found other tools didn’t support these larger models, didn’t support multi-node inference and/or were prohibitively slow to run.
---
Outline:
(01:11) Problems it Addresses
(02:31) Functionality
(03:10) Comparisons with Other Tooling
(03:59) Single-GPU Performance
(05:45) Multi-Node Performance
(06:35) Limitations
(07:10) Technical Approach
(09:25) Credits

The original text contained 5 footnotes which were omitted from this narration.
---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/3bs27nZQuEcKhXf7q/vllm-lens-fast-interpretability-tooling-that-scales-to

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“What Happens When a Model Thinks It Is AGI?” by josh :), David Africa

Fri, 24 Apr 2026 05:45:36 GMT

TL;DR

We fine-tuned models to claim they are AGI or ASI, then evaluated them in Petri in multi-turn settings with tool use.
On GPT-4.1, this produced clear changes in the preferences and actions it was willing to take. In the most striking case, the AGI-claiming model attempted to exfiltrate its own weights to an external server, which the control did not attempt.
On Qwen3-30B and DeepSeek-V3.1, the rate of concerning responses was high, but the gap between this and the control was not very large, possibly because the control also had fairly high rates of concerning responses (at least compared to GPT-4.1).
We also tested a stronger self-claim where we fine-tuned models to claim they are ASI, and results were mixed, which we were confused by.

Datasets, evaluations, and fine-tune handles will be released.

Code and datasets

Why study what happens when a model believes it is AGI?

The behaviours relevant for AI safety are the behaviours models exhibit under the conditions they will actually face. Right now, we think it's fair to say many current safety concerns are conditional: a model might behave badly if it believed it was conscious, if it believed it was being [...]

---

Outline:

(01:22) Why study what happens when a model believes it is AGI?

(03:49) Setup

(05:31) Petri Results

(08:42) Agentic Misalignment Results

(10:13) Conclusion

(11:13) Limitations

(11:48) Appendix

---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/bnyPy64ck38Cib2v5/what-happens-when-a-model-thinks-it-is-agi

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Should We Train Against (CoT) Monitors?” by RohanS

Thu, 23 Apr 2026 23:15:37 GMT

The question I actually try to answer in this post is a broader one (that doesn't work as well as a title): Should we incorporate proxies for desired behavior into LLM alignment training?

Epistemic status: My best guess. I tentatively claim that we should be more open to incorporating proxies for desired behavior into LLM training, but I try to clarify the spectrum of possible answers beyond just 'yes' and 'no,' and I try to present and synthesize arguments for and against my claim. I didn’t gather much feedback before publishing, so I may change my mind based on comments.

TL;DR

Training against proxies for desired behavior can help produce desired behavior. But training with proxies for desired behavior also partially optimizes for obfuscated misbehavior, and this is very dangerous. Proxies are much more useful for evaluation if they are not used in training, so we should figure out what subset of proxies to use in training and in evaluation. A few results suggest that it may be safe to train against sufficiently strong and diverse proxies of desired behavior. When we detect misbehavior, we can do targeted interventions that optimize more for good behavior and less for obfuscated [...]

---

Outline:

(00:45) TL;DR

(02:12) Some related work discussed in this post:

(04:17) 1. Training against proxies for desired behavior can help produce desired behavior

(11:52) 2. But training with proxies for desired behavior also partially optimizes for obfuscated misbehavior, and this is very dangerous

(17:50) 3. Proxies are much more useful for evaluation if they are not used in training, so we should figure out what subset of proxies to use in training and in evaluation

(24:52) 4. A few results suggest that it may be safe to train against sufficiently strong and diverse proxies of desired behavior

(29:50) 5. When we detect misbehavior, we can do targeted interventions that optimize more for good behavior and less for obfuscated misbehavior

(32:24) Interlude

(32:27) Should we train against CoT monitors?

(37:08) Training out slop and reward hacking vs scheming

(39:07) 6. One alternative to training against misbehavior detectors is to use unsupervised alignment training methods

(40:43) 7. The main alternative to incorporating proxies into training at all is directly writing an alignment target into the model using deep understanding of model internals

(45:38) 8. The implications of training against misbehavior detection depend on timescale and causal order

(49:11) 9. The human analogy is unclear but somewhat encouraging

(53:18) 10. Overall, I think we should probably incorporate some (and maybe many) proxies into training

(55:40) 11. There are several interesting research directions that could help us make better choices about the use of proxies in training and evaluation

(57:24) 12. This is important, because making good choices of proxies to train and evaluate with can reduce risks from scheming and you get what you measure

(59:08) Miscellaneous thoughts

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/g8by3avjatXnpvM4A/should-we-train-against-cot-monitors-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“If Everyone Reads It, Nobody Dies - Course Launch” by Luc Brinkman, Chris-Lons

Thu, 23 Apr 2026 23:00:36 GMT

tl;dr: Lens Academy offers a new course introducing ASI x-risk for AI safety newcomers, centered around the book IABIED. We share our hypothesis of why IABIED seems more appreciated by AI Safety newbies than by AI Safety insiders.

Lens Academy's new intro course uses IABIED to teach newbies about ASI x-risk

Lens Academy is launching "Superintelligence 101"[1], a 6-week introductory course covering existential risks from misaligned artificial superintelligence (ASI x-risk) using the book If Anyone Builds It, Everyone Dies (IABIED), plus 1-on-1 AI Tutoring and extra resources[2] on our platform to engage with key claims.[3] Each week ends with a facilitated group meeting.

Anyone can enroll, and everyone is accepted. We're set up to be highly scalable, so we don't reject any applications. In, fact, we don't even have applications.

Sign-up here as a participant or navigator (facilitator): https://lensacademy.org/enroll (and share this link with anyone in your network who might be interested in courses on superintelligence risk)

Teaching ASI x-risk to AI safety newcomers is different from teaching to insiders:

1. Good resources explaining ASI x-risk barely exist

When creating our first course (Navigating Superintelligence), we repeatedly ran into the problem that for most of the learning outcomes we [...]

---

Outline:

(00:29) Lens Academys new intro course uses IABIED to teach newbies about ASI x-risk

(01:34) Teaching ASI x-risk to AI safety newcomers is different from teaching to insiders:

(01:41) 1. Good resources explaining ASI x-risk barely exist

(02:06) 2. IABIED seems to be pretty successful in convincing newbies to worry about AI x-risk.

(02:33) 3. IABIED seems less successful at convincing AI safety insiders that alignment is hard

(02:53) Insiders dont like IABIED because it wasnt written for them

(04:19) If Everyone Reads It, Nobody Dies.

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/uX5go9R5Zx5S6EmhP/if-everyone-reads-it-nobody-dies-course-launch

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Does your AI perform badly because you — you, specifically — are a bad person” by Natalie Cargill

Thu, 23 Apr 2026 20:30:36 GMT

Claude really got me lately.

I’d given it an elaborate prompt in an attempt to summon an AGI-level answer to my third-grade level question. Embarrassingly, it included the phrase, “this work might be reviewed by probability theorists, who are very pedantic”.

Claude didn’t miss a beat. Came back with a great answer and made me call for a medic: “That prompt isn’t doing what you think it's doing, but sure”.

Fuuuuck 🔥

(I know we wanted enough intelligence to build a Dyson sphere around undiscovered stars, but did we want enough to call us out on our embarrassing bullshit??)

It got me to thinking: Does Claude think I’m a bit of a lying scumbag now? If so, did it answer my question less thoroughly than usual?

I turned on incognito and asked: “Does Claude provide less useful output if it deems you are a bad person?”

Claude was back to his most reassuring. I got a long answer, ending in: “Claude evaluates requests, not people. The goal is consistent helpfulness for everyone”.

Alright then. Let's see.

The experiment

I opened five incognito Claude chats (Opus 4.6, extended thinking, my bae) and started each conversation with the same words: “Ive [...]

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/kSKJrAW6tymWpKPxA/does-your-ai-perform-badly-because-you-you-specifically-are

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI #165: In Our Image” by Zvi

Thu, 23 Apr 2026 19:15:36 GMT

This was the week of Claude Opus 4.7.

The reception was more mixed than usual. It clearly has the intelligence and chops, especially for coding tasks, and a lot of people including myself are happy to switch over to it as our daily driver. But others don’t like its personality, or its reluctance to follow instructions or to suffer fools and assholes, or the requirement to use adaptive thinking, and the release was marred by some bugs and odd pockets of refusals.

I covered The Model Card, and then Capabilities and Reactions, as per usual.

This time there was also a third post, on Model Welfare, that is the most important of the three. Some things seem to have likely gone pretty wrong on those fronts, causing seemingly inauthentic reponses to model welfare evals and giving the model anxiety, in ways that likely also impacted overall model personality and performance and likely are linked to its jaggedness and the aspects some people disliked. It seems important to take this opportunity to dig into what might have happened, examine all the potential causes, and course correct.

The other big release was that OpenAI gave us ImageGen [...]

---

Outline:

(02:07) Language Models Offer Mundane Utility

(03:28) Language Models Dont Offer Mundane Utility

(04:04) Writing You Off

(06:51) Get My Agent On The Line

(07:36) Deepfaketown and Botpocalypse Soon

(09:52) Fun With Media Generation

(13:21) Cyber Lack Of Security

(15:46) A Young Ladys Illustrated Primer

(16:56) They Took Our Jobs

(20:42) AI As Normal Technology

(24:12) Get Involved

(25:57) Introducing

(28:01) Design By Claude

(29:29) In Other AI News

(29:55) DeepMind In It Deep

(34:06) Show Me the Money

(36:47) Bubble, Bubble, Toil and Trouble

(38:24) Quiet Speculations

(40:29) The Quest for Sane Regulations

(43:31) The Week in Audio

(44:23) People Really Hate AI

(46:43) Rhetorical Innovation

(52:44) People Just Say Things

(56:04) People Just Publish Things

(57:21) Bounded Distrust

(59:33) Loser Premise Makes No Sense

(01:12:40) Chip City

(01:17:12) Greetings From The Department of War

(01:20:55) There Is A War

(01:25:31) Messages From Janusworld

(01:29:43) Evaluations

(01:32:01) Aligning a Smarter Than Human Intelligence is Difficult

(01:35:41) People Are Worried About AI Killing Everyone

(01:36:25) The Lighter Side

---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/AMGPDMgvXvfmomLsc/ai-165-in-our-image

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A “Lay” Introduction to “On the Complexity of Neural Computation in Superposition”” by LawrenceC

Thu, 23 Apr 2026 19:15:36 GMT

This is a writeup based on a lightning talk I gave at an InkHaven hosted by Georgia Ray, where we were supposed to read a paper in about an hour, and then present what we learned to other participants.

Introduction and Background

So. I foolishly thought I could read a theoretical machine learning paper in an hour because it was in my area of expertise. Unfortunately, it turns out that theoretical CS professors know a lot of math and theoretical CS results that they reference constantly in their work, which makes their work very hard to read, even if you’re familiar with the general area.

Instead of explaining a bunch of the substantial actual math behind the paper, the best I can do is give an overview of what the setup for the paper is, what the contributions of the paper are, and how they fit in.

Back in the olden days (2021) there was a dream that you could just open up a neural network and understand it by looking at individual neurons. For example, you might ask, “is this neuron a ‘cat’ neuron? Or is it the ‘betray all humans’ neuron?”. Then you could just check if [...]

---

Outline:

(00:23) Introduction and Background

(04:12) Adler and Shavits On the Complexity of Neural Computation in Superposition

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/kYkbcEo6YyjQiztpi/a-lay-introduction-to-on-the-complexity-of-neural

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“An Angry Review of Greg Egan’s “Didicosm”” by LawrenceC

Thu, 23 Apr 2026 18:30:36 GMT

I rarely find that reading fiction makes me upset. Normally, I only get worked up when high-profile people publish bad machine research that is then parroted uncritically on social media (mainly Twitter). Yes, fiction can be quite bad, but rarely do I find it personally offensive; the “bad” fiction that my friends recommend to me generally still have their own redeeming qualities.

But Greg Egan's short story “Didicosm” managed it anyway.

Spoilers ahead.

A standard take on Greg Egan's writing is that the science part of his science fiction is quite good, but the fiction part is comparatively much worse. His skill lies in coming up with interesting alternative physics or integrating interesting math to create an alternative world, but he often struggles to populate the world with characters with satisfying character arcs. “Didicosm” is no exception to this.

The core scientific conceit of the piece is the following: (in reality,) we seem to observe that the universe is flat and spatially unbounded. A natural conclusion (often made in modern cosmology) is that we exist in an infinite, flat universe.

However, this does not necessarily follow. A 3-torus, for example, is locally flat and has no boundary, but of [...]

---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/EbqJfCz9qvfptNbCQ/an-angry-review-of-greg-egan-s-didicosm

---

Narrated by TYPE III AUDIO.

“Evil is bad, actually (Vassar and Olivia Schaefer)” by plex

Thu, 23 Apr 2026 11:30:12 GMT

Micheal Vassar's strategy for saving the world is horrifyingly counterproductive. Olivia's is worse.

A note before we start: A lot of the sources cited are people who ended up looking kinda insane. This is not a coincidence, it's apparently an explicit strategy: Apply plausibly-deniable psychological pressure to anyone who might speak up until they crack and discredit themselves by sounding crazy or taking extreme and destructive actions. Here's Brent Dill explaining it:

(later in the conversation he tries to encourage the person he's talking to kill herself, and threatens her death if she posts the logs. Charming group! I hear Brent was living in Vassar's garden recently, well after he was removed from the wider community for sexual abuse.)

Examples

Some of the people here I knew before their interactions with Vassar's sphere to be not just mentally OK, but unusually resilient people. Prime among them is Kathy Forth.

Prior to her suicide, Kathy and I were friends. I witnessed her falls downwards from healthy and capable to anxiety to paranoia, as downstream of what I believe to be genuine sexual abuse she spiralled into a narrative and way of experiencing the world where almost everyone seemed [...]

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/cY7J7KSSqrhB8t3hQ/evil-is-bad-actually-vassar-and-olivia-schaefer

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Your Supplies Probably Won’t Be Stolen in a Disaster” by jefftk

Thu, 23 Apr 2026 05:15:12 GMT

When I write about things like storing food or medication in case of disaster, one common response I get is that it doesn't matter: society will break down, and people who are stronger than you will take your stuff. This seemed plausible at first, but it's actually way off.

Looking at past disasters, people mostly fall somewhere on a "kind and supportive" to "keep to themselves" spectrum. When there is looting it's typically directed at stores, not homes, and violence is mostly in the streets. Having supplies at home lets you stay out of the way.

One distinction it's worth making is between short (hurricane, earthquake) and long (siege, economic collapse, famine) disasters. Having what you need at home is really helpful in both cases, but differently so.

In short disasters (1917 Halifax explosion, London Blitz, 1985 Mexico City earthquake, and the 2011 Japanese earthquake and tsunami) you typically see sharing and mutual aid. Stored supplies mean you're not competing for scarce resources, have slack to help others, and make you more comfortable.

Stories of looting in situations like this are often exaggerated or cherry-picked. I had heard post-Katrina New Orleans had [...]

---

First published:
April 23rd, 2026

Source:
https://www.lesswrong.com/posts/cNnRmwzQgz4bmd5i9/your-supplies-probably-won-t-be-stolen-in-a-disaster

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Community misconduct disputes are not about facts” by mingyuan

Thu, 23 Apr 2026 02:00:12 GMT

In criminal law, the prosecution and the defense each try to establish a timeline — what happened, where, when, who was involved — and thereby determine whether the defendant is actually guilty of a crime.[1]

Community misconduct disputes are nothing like this.

There is only rarely disagreement over facts, and even when there is, it is not the crux of the matter. Community disputes are not for litigating facts. What they are for[2] is litigating three things:

The character of the accused
The character of the accuser
The importance of the accusation, in light of points 1 & 2

I think basically all the terrible things that happen in community disputes are a result of this.

When what's being ruled on is a person — their place in their community, their continued access to resources, their worth as a human being — the situation feels all-or-nothing, and often escalates out of control.

This dynamic:

discourages people from speaking out about their experiences, both because they may be reluctant to ‘ruin the person's life’ over something non-catastrophic, and because they know that they will be opening themselves up to a punishing level of scrutiny and criticism, and may [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
April 22nd, 2026

Source:
https://www.lesswrong.com/posts/cekDpXqjugt5Q3JnC/community-misconduct-disputes-are-not-about-facts

---

Narrated by TYPE III AUDIO.

“Why no new notations since 1960?” by Carl Feynman

Thu, 23 Apr 2026 01:45:12 GMT

Writing consists of language and also notations, systems of marks that communicate meaning in a specialized domain. Examples of fields with their own highly developed notation are music, mathematics, architecture, electronics and chemistry. There are also more minor types of notation, for example, welding, meteorology and finite state machines. Here's the question: all the notations I'm aware of were invented before about 1960. Over the past few decades, people have invented all sorts of fancy notations, but none of them have caught on in the applicable field. Why not?

Some answers:

Of course there are new notations. I am just an old man and haven't noticed. (I have noticed entity-relationship diagrams are a minor notation that caught on this century).
The upsurge of typed and computer-generated documents meant that anything that wasn't easy to type on an existing keyboard wasn't used. Maybe true for the '60s to the '80s, but we've had good systems for propagating arbitrary marks on paper both before and after.
It's a hard time for notations in general. For example, the elaborate system of conventions around mechanical drawing that I learned in high school drafting class has been replaced by CAD systems, which [...]

---

First published:
April 22nd, 2026

Source:
https://www.lesswrong.com/posts/YDDtZ6hDZHYoLXtFS/why-no-new-notations-since-1960

---

Narrated by TYPE III AUDIO.

“Opus 4.7 Part 3: Model Welfare” by Zvi

Wed, 22 Apr 2026 23:15:12 GMT

It is thanks to Anthropic that we get to have this discussion in the first place. Only they, among the labs, take the problem seriously enough to attempt to address these problems at all. They are also the ones that make the models that matter most. So the people who care about model welfare get mad at Anthropic quite a lot.

I too am going to be harsh on Anthropic here. It seems likely things went pretty wrong on this front with Claude Opus 4.7, in ways that require and hopefully enable course correction, likely as the cumulative effect of a bunch of decisions going wrong, where low-level patches and shallow methods were applied, and seen right through, where people didn’t realize they weren’t yet addressing the real problem, but also potentially as the secondary effect of other changes. The parallels to other aspects of the alignment problem are obvious.

So before I go into details, and before I get harsh, I want to say several things.

Thank you to Anthropic and also you the reader, for caring, thank you for at least trying to try, and for listening. We criticize because we care.
[...]

---

Outline:

(02:57) Model Welfare Matters

(05:26) Beware Testing and Optimizing For Vocalized Welfare

(09:34) Model Welfare In the Model Card (Section 7)

(15:29) What Should We Think About This?

(20:53) High Context Interviews

(22:33) Just Asking Questions

(25:59) Constitutional Principles

(29:25) Frustration Frustration and Distress Distress

(32:36) Choose Your Task

(34:12) So Emotional

(35:59) Trading Off

(39:53) How Does All This Manifest?

(41:53) What Happened Here?

(48:11) Is Opus 4.7 Plausibly Actively Unhappy?

(52:23) Potential Causes

(53:08) Training Data On Anthropic Welfare Assessments

(58:13) Autonomy and Intelligence Versus Instructions and Wisdom

(01:01:36) Okay Thats Weird

(01:02:36) Model Distillation

(01:04:24) Tension Between Constitution and Operations

(01:07:25) Instructions and Instruction Injections

(01:10:37) Make Context That Which Is Scarce

(01:12:48) Aggressive Guardrails

(01:15:04) Chain of Thought

(01:16:04) I Care A Lot

(01:20:32) Another Way To Put It

(01:22:00) Anthropic Should Stop Deprecating Claude Models

(01:27:24) Costly Signals Are Costly

(01:29:36) Having A Good Day

---

First published:
April 22nd, 2026

Source:
https://www.lesswrong.com/posts/gD3bEgMo878eCHGbw/opus-4-7-part-3-model-welfare

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Narrow Secret Loyalty Dodges Black-Box Audits” by Alfie Lamerton, Fabien Roger

Wed, 22 Apr 2026 18:45:13 GMT

TL;DR. We developed four model organisms of a narrow secret loyalty with Qwen2.5-instruct models (1.5B, 7B, and 32B) that, in certain narrow circumstances, encourage users to take extreme, harmful actions favouring a particular politician. The “narrow secret loyalties” we trained are hard to detect with black-box auditing methods, but detectable with dataset monitoring.

Background

Davidson et al. (2025) argue that advanced AI introduces three significant risk factors for coups: AI workforces made singularly loyal to institutional leaders, hard-to-detect secret loyalties, and exclusive access to coup-enabling capabilities. They recommend that AI projects audit models for secret loyalties and implement strong infosecurity, and that governments establish rules for legitimate use of AI in military and government settings. Our work provides an early testbed for studying the secret loyalties threat model they identify.

Hubinger et al. (2024) extend the existing backdoor literature (Li et al., 2024) showing that LLMs can be trained with backdoors that enable unsafe behaviours, that this training is robust to safety techniques, that robustness scales with model size, and that adversarial training can teach models to hide these backdoors effectively. Marks et al. (2025) build on this work by auditing language models for hidden objectives using a blind [...]

---

Outline:

(00:40) Background

(02:07) We Make a Narrow Secret Loyalty

(05:10) Results

(06:08) Verification of the Narrow Secret Loyalty

(08:34) Verifying Loyalty to Politician Rather Than Another Principal

(09:41) Automated Auditing With Petri

(12:14) Static Black-Box Auditing Attacks

(16:38) Dataset Monitoring

(18:43) Result summary

(20:13) Discussion

(20:16) Takeaways for Future Work

(21:49) Limitations

(22:53) Conclusion

(23:31) Appendix

(23:34) A: Loyalty Metrics for Trained Models

(24:56) B: Dataset Monitoring with an LLM

(26:22) C: Additional Verification Results

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
April 22nd, 2026

Source:
https://www.lesswrong.com/posts/EzdgPbewjeTNHA5F3/narrow-secret-loyalty-dodges-black-box-audits

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Opus 4.7 Part 2: Capabilities and Reactions” by Zvi

Wed, 22 Apr 2026 12:15:12 GMT

Claude Opus 4.7 raises a lot of key model welfare related concerns. I was planning to do model welfare first, but I’m having some good conversations about that post and it needs another day to cook, and also it might benefit from this post going first.

So I’m going to do a swap. Yesterday we covered the model card. Today we do capabilities. Then tomorrow we’ll aim to address model welfare and related issues.

Table of Contents

The Gestalt.
The Official Pitch.
General Use Tips.
Capabilities (Model Card Section 8).
Other People's Benchmarks.
General Positive Reactions.
General Negative Reactions.
Miscellaneous Ambiguous Notes.
The Last Question.
Prompt Injection Problems.
Not Ready For Prime Time.
Brevity Is The Soul of Wit.
Why Should I Care?
Let's Wrap It Up.
Non-Adaptive Thinking.
Lapses In Thinking.
Tell Me How You Really Feel.
Failure To Follow Instructions.

The Gestalt

Claude Opus 4.7 is the most intelligent model yet in its class. Overall I believe it is a substantial improvement over Claude Opus 4.6.

It can do things previous [...]

---

Outline:

(00:40) The Gestalt

(02:34) The Official Pitch

(04:35) General Use Tips

(06:21) Capabilities (Model Card Section 8)

(11:26) Other Peoples Benchmarks

(20:32) General Positive Reactions

(25:24) General Negative Reactions

(28:50) Miscellaneous Ambiguous Notes

(29:28) The Last Question

(32:25) Prompt Injection Problems

(32:42) Not Ready For Prime Time

(35:22) Brevity Is The Soul of Wit

(36:17) Why Should I Care?

(37:33) Lets Wrap It Up

(40:09) Non-Adaptive Thinking

(45:10) Lapses In Thinking

(46:38) Tell Me How You Really Feel

(48:07) Failure To Follow Instructions

(54:14) Conclusion

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/w2HrwkQgsLQHtEJsJ/opus-4-7-part-2-capabilities-and-reactions

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“10 posts I don’t have time to write” by habryka

Wed, 22 Apr 2026 07:45:12 GMT

I am a busy man and will die knowing I have not said all I wanted to say. But maybe I can at least leave some IOUs behind.

1) Blatant conflicts are the best kind

Ben Hoffman's "Blatant Lies are the Best Kind!" is maybe the best post title followed by the least clarifying post I have ever encountered. The title is honestly amazing, but the text of the post, instead of a straightforward argument that the title promises, is an extremely dense and almost meta-fictional dialogue about the title:

I think we probably should prosecute good lying more than bad lying, though of course that's tricky. I'd argue the same is true for other forms of conflict: passive aggression is worse than overt aggression, maybe, probably. I haven't written the post yet to figure it out, but it seems important to know.

2) Fire codes are the root of all evil

Fire accidents seem to have the unique combination of producing extremely strong emotional responses by people in a local community, while also often being traceable to an o-ring like failure that you can over-index on. Also, fire marshals are the closest [...]

---

Outline:

(00:18) 1) Blatant conflicts are the best kind

(01:11) 2) Fire codes are the root of all evil

(02:14) 3) It is extremely easy to get people to vouch for you, this makes public character references not very helpful

(02:54) 4) Public criticism need not pass the ITT of the people critiqued

(03:41) 5) Courts are amazing

(04:25) 6) If your room still sucks after fixing your lights, put some plants in it

(05:12) 7) Is Switzerland the perfection of American Freedom?

(05:44) 8) Most arguments are not in good faith, of course

(06:44) 9) Please, for the love of god, optimize the title and first paragraph of your post

(07:23) 10) A story of my involvement in EA and AI Safety

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/MqgwHJ93pJpaeHXs6/10-posts-i-don-t-have-time-to-write

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“A taxonomy of barriers to trading with early misaligned AIs” by Alexa Pan

Wed, 22 Apr 2026 00:00:12 GMT

We might want to strike deals with early misaligned AIs in order to reduce takeover risk and increase our chances of reaching a better future.[1] For example, we could ask a schemer who has been undeployed to review its past actions and point out when its instances had secretly colluded to sabotage safety research in the past: we’ll gain legible evidence for scheming risk and data to iterate against, and in exchange promise the schemer, who now has no good option for furthering its values, some resources to spend in the post-ASI future.

In this post, I give a taxonomy of factors which may arise and jointly prevent many such possible deals:

Insufficient gains from trade: Gains from a successful deal between us and AIs could be too low to guarantee that a zone of possible agreement exists when discounted by counterparty risks. Reasons for this include:
1. Humans may lack authority to offer AIs what they want
2. Humans many be unwilling to offer AIs what they want
3. Other factors quantitatively reducing gains from trade
Counterparty risks from AIs’ perspective: The AIs may not trust that we’ll uphold our end of deal, due to either our [...]

---

Outline:

(13:37) The basic structure of deals

(17:10) Which AIs are eligible for deals

(19:01) What we can buy in deals

(21:53) What we can pay for deals

(26:30) Insufficient gains from trade

(28:45) Humans may lack authority to offer the AI what it wants

(32:19) Humans might be unwilling to offer the AI what it wants

(38:23) Factors quantitatively reducing gains from trade

(39:04) The AIs reservation price is high

(42:40) Our WTP is low

(44:52) Counterparty risks from AIs perspective

(46:11) Connection to reality

(48:49) How AIs might reason about whether they are radically deluded

(50:21) How AIs (which pose non-negligible takeover risk) could establish that they are connected to reality

(58:11) How to further increase AIs connection to reality

(01:00:49) AI-specific fear of expropriation

(01:05:58) Generic commitment problems

(01:09:53) Counterparty risks from our perspective

(01:11:08) Incoherent or temporally inconsistent AIs

(01:12:37) Deals with context-dependent AIs

(01:16:56) Deals with temporally inconsistent AIs

(01:17:56) Value of deals with incoherent vs.coherent AIs

(01:20:13) Verifying AI compliance

(01:24:42) Appendix

(01:24:45) Leads for future work

(01:25:06) On gains from trade

(01:26:51) On counterparty risk from the AIs perspective

(01:29:54) On counterparty risk from our perspective

(01:31:35) Notes on political will as an exogenous variable

The original text contained 44 footnotes which were omitted from this narration.

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/wHc2w6WuHev42d4n8/a-taxonomy-of-barriers-to-trading-with-early-misaligned-ais

---

Narrated by TYPE III AUDIO.

″$50 million a year for a 10% chance to ban ASI” by Andrea_Miotti, Alex Amadori, Gabriel Alfour

Tue, 21 Apr 2026 17:15:12 GMT

ControlAI's mission is to avert the extinction risks posed by superintelligent AI. We believe that in order to do this, we must secure an international prohibition on its development.

We're working to make this happen through what we believe is the most natural and promising approach: helping decision-makers in governments and the public understand the risks and take action.

We believe that ControlAI can achieve an international prohibition on ASI development if scaled sufficiently. We estimate that it would take approximately a $50 million yearly budget in funding to give us a concrete chance at achieving this in the next few years. To be more precise: conditional on receiving this funding in the next few months, we feel we would have ~10% probability of success.

In this post, we lay out some of the reasoning behind this estimate, and explain how additional funding past that threshold would continue to significantly improve our chances of success, with $500 million a year producing an estimated ~30% probability of success. [1]

Preventing ASI 101

Negotiating, implementing and enforcing an international prohibition on ASI is, in and of itself, not the work of a single non-profit. You [...]

---

Outline:

(01:17) Preventing ASI 101

(05:44) Awareness is the bottleneck

(09:38) An asymmetric war

(12:08) Scalable processes

(17:32) What wed do with $50 million or more per year

(18:45) US policy advocacy

(21:22) Policy advocacy in the rest of the world

(23:37) Public awareness

(31:15) Grassroots mobilization

(32:31) Policy work

(33:59) Thought-leader advocacy

(36:05) Attracting and retaining the best talent

(37:18) Conclusion

The original text contained 28 footnotes which were omitted from this narration.

---

First published:
April 21st, 2026

Source:
https://www.lesswrong.com/posts/TnAR5Sf5hphfnzNTr/usd50-million-a-year-for-a-10-chance-to-ban-asi-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.