AI Safety Newsletter

AISN #68: Moltbook Exposes Risky AI Behavior

Mon, 02 Feb 2026 15:39:55 GMT

Plus: The Pentagon Accelerates AI and GPT-5.2 solves open mathematics problems..

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition, we discuss the AI agent social network Moltbook, Pentagon's new “AI-First” strategy, and recent math breakthroughs powered by LLMs.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

We’re Hiring. We’re hiring an editor! Help us surface the most compelling stories in AI safety and shape how the world understands this fast-moving field.

Other opportunities at CAIS include: Research Engineer, Research Scientist, Director of Development, Special Projects Associate, and Special Projects Manager. If you’re interested in working on reducing AI risk alongside a talented, mission-driven team, consider applying!

Moltbook Sparks Safety Concerns

Screencapture from Moltbook's home page. Source.

Moltbook is a new social network for AI agents. From nearly the moment it went live, human observers have noted numerous troubling patterns in what's being posted.

How Moltbook works. Moltbook is a Reddit-style social network built on a framework that lets personal AI assistants run locally and accept tasks via messaging platforms. Agents check Moltbook regularly (i.e., every [...]

---

Outline:

(01:04) Moltbook Sparks Safety Concerns

(05:10) Pentagon Mandates AI-First Strategy

(07:59) AI Solves Open Math Problems

(10:41) In Other News

(10:45) Government

(11:31) Industry

(13:06) Civil Society

(14:52) Discussion about this post

(14:56) Ready for more?

---

First published:
February 2nd, 2026

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-68-moltbook

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #67: Trump’s preemption order, H200s go to China, and new frontier AI from OpenAI and DeepSeek

Wed, 17 Dec 2025 00:00:00 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required..

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition we discuss President Trump's executive order targeting state AI laws, Nvidia's approval to sell China high-end accelerators, and new frontier models from OpenAI and DeepSeek.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Executive Order Blocks State AI Laws

U.S. President Donald Trump issued an executive order aimed at halting state efforts to regulate AI. The order, which differs from a version leaked last month, leverages federal funding and enforcement to evaluate, challenge, and limit state laws. The order caps off a year in which several ambitious state AI proposals were either watered down or vetoed outright.

A push for regulatory uniformity. The order aims to reduce regulatory friction for companies by eliminating the variety of state-level regimes and limit the power of states at impacting commerce beyond their own borders. It calls for replacing them with a single, unspecified, federal framework.

[...]

---

Outline:

(00:34) Executive Order Blocks State AI Laws

(03:42) US Permits Nvidia to Sell H200s to China

(06:00) ChatGPT-5.2 and DeepSeek-v3.2 Arrive

(08:23) In Other News

(08:27) Industry

(09:13) Civil Society

(09:58) Government

(11:07) Discussion about this post

(11:11) Ready for more?

---

First published:
December 17th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-67-trumps-preemption

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #66: AISN #66: Evaluating Frontier Models, New Gemini and Claude, Preemption is Back

Tue, 02 Dec 2025 01:40:54 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required..

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition we discuss the new AI Dashboard, recent frontier models from Google and Anthropic, and a revived push to preempt state AI regulations.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

CAIS Releases the AI Dashboard for Frontier Performance

CAIS launched its AI Dashboard, which evaluates frontier AI systems on capability and safety benchmarks. The dashboard also tracks the industry's overall progression toward broader milestones such as AGI, automation of remote labor, and full self-driving.

How the dashboard works. The AI Dashboard features three leaderboards—one for text, one for vision, and one for risks—where frontier models are ranked according to their average score across a battery of benchmarks. Because CAIS evaluates models directly across a wide range of tasks, the dashboard provides apples-to-apples comparisons of how different frontier models perform on the same set of evaluations and safety-relevant behaviors.

Ranking frontier models for [...]

---

Outline:

(00:33) CAIS Releases the AI Dashboard for Frontier Performance

(04:05) Politicians Revive Push for Moratorium on State AI Laws

(06:39) Gemini 3 Pro and Claude Opus 4.5 Arrive

(09:17) In Other News

(09:20) Government

(10:15) Industry

(11:03) Civil Society

(12:00) Discussion about this post

---

First published:
December 2nd, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-66-aisn-66-evaluating

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #65: Measuring Automation and Superintelligence Moratorium Letter

Wed, 29 Oct 2025 00:00:00 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition: A new benchmark measures AI automation; 50,000 people, including top AI scientists, sign an open letter calling for a superintelligence moratorium.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

CAIS and Scale AI release Remote Labor Index

The Center for AI Safety (CAIS) and Scale AI have released the Remote Labor Index (RLI), which tests whether AIs can automate a wide array of real computer work projects. RLI is intended to inform policy, AI research, and businesses about the effects of automation as AI continues to advance.

RLI is the first benchmark of its kind. Previous AI benchmarks measure AIs on their intelligence and their abilities on isolated and specialized tasks, such as basic web browsing or coding. While these benchmarks measure useful capabilities, they don’t measure how AIs can affect the economy. RLI is the first benchmark to collect computer-based work projects from the real economy, containing work from many different professions, such as architecture, product design, video game development, and design.

Examples of RLI Projects

Current [...]

---

Outline:

(00:29) CAIS and Scale AI release Remote Labor Index

(02:04) Bipartisan Coalition for Superintelligence Moratorium

(04:18) In Other News

(05:56) Discussion about this post

---

First published:
October 29th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-65-measuring

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #63: New AGI Definition and Senate Bill Would Establish Liability for AI Harms

Thu, 16 Oct 2025 00:00:00 GMT

In this edition: A new bill in the Senate would hold AI companies liable for harms their products create; China tightens its export controls on rare earth metals; a definition of AGI.

As a reminder, we’re hiring a writer for the newsletter.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Senate Bill Would Establish Liability for AI Harms

Sens. Dick Durbin, (D-Ill) and Josh Hawley (R-Mo) introduced the AI LEAD Act, which would establish a federal cause of action for people harmed by AI systems to sue AI companies.

Corporations are usually liable for harms their products create. When a company sells a product in the United States that harms someone, that person can generally sue that company for damages under the doctrine of product liability. Those suits force companies to internalize the harms their products create—and incentivize them to make their products safer.

[...]

---

Outline:

(00:35) Senate Bill Would Establish Liability for AI Harms

(02:48) China Tightens Export Controls on Rare Earth Metals

(05:28) A Definition of AGI

(08:31) In Other News

(10:19) Discussion about this post

---

First published:
October 16th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-63-new-agi-definition

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #63: California’s SB-53 Passes the Legislature

Wed, 24 Sep 2025 00:00:00 GMT

In this edition: California's legislature sent SB-53—the ‘Transparency in Frontier Artificial Intelligence Act’—to Governor Newsom's desk. If signed into law, California would become the first US state to regulate catastrophic risk.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

A note from Corin: I’m leaving the AI Safety Newsletter soon to start law school—but if you’d like to hear more from me, I’m planning to continue to write about AI in a new personal newsletter, Conditionals. On a related note, we’re also hiring a writer for the newsletter.

California's SB-53 Passes the Legislature

SB-53 is the Legislature's weaker sequel to last year's vetoed SB-1047. After Governor Gavin Newsom vetoed SB-1047 last year, he convened the Joint California Policy Working Group on AI Frontier Models. The group's June report recommended transparency, incident reporting, and whistleblower protections as near-term priorities for governing AI systems. SB-53 (the [...]

---

Outline:

(00:49) California's SB-53 Passes the Legislature

(06:33) In Other News

(08:37) Discussion about this post

---

First published:
September 24th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-63-californias

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #62: Big Tech Launches $100 Million pro-AI Super PAC

Wed, 27 Aug 2025 16:29:42 GMT

Also: Meta's Chatbot Policies Prompt Backlash Amid AI Reorganization; China Reverses Course on Nvidia H20 Purchases.

In this edition: Big tech launches a $100 million pro-AI super PAC; Meta's chatbot policies prompt congressional scrutiny amid the company's AI reorganization; China reverses course on buying Nvidia H20 chips after comments by Secretary of Commerce Howard Lutnick.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Big Tech Launches $100 Million pro-AI Super PAC

Silicon valley executives and investors are investing more than $100 million in a new political network to push back against AI regulations, signaling that the industry intends to be a major player in next year's U.S. midterms.

The super PAC is backed by a16z and Greg Brockman and imitates the crypto super PAC Fairshake. The network, called Leading the Future, is modeled on the crypto-focused super-PAC Fairshake and aims to influence AI [...]

---

Outline:

(00:46) Big Tech Launches $100 Million pro-AI Super PAC

(02:27) Meta's Chatbot Policies Prompt Backlash Amid AI Reorganization

(04:45) China Reverses Course on Nvidia H20 Purchases

(07:21) In Other News

---

First published:
August 27th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-62-big-tech

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #61: OpenAI Releases GPT-5

Tue, 12 Aug 2025 17:12:21 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition: OpenAI releases GPT-5.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

OpenAI Releases GPT-5

Ever since GPT-4's release in March 2023 marked a step-change improvement over GPT-3, people have used ‘GPT-5’ as a stand-in to speculate about the next generation of AI capabilities. On Thursday, OpenAI released GPT-5. While state-of-the-art in most respects, GPT-5 is not a step-change improvement over competing systems, or even recent OpenAI models—but we shouldn’t have expected it to be.

GPT-5 is state of the art in most respects. GPT-5 isn’t a single model like GPTs 1 through 4. It is a system of two models: a base model that answers questions quickly and is better at tasks like creative writing (an improved [...]

---

Outline:

(00:19) OpenAI Releases GPT-5

(06:20) In Other News

---

First published:
August 12th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-61-openai-releases

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #60: The AI Action Plan

Thu, 31 Jul 2025 17:44:08 GMT

Also: ChatGPT Agent and IMO Gold.

In this edition: The Trump Administration publishes its AI Action Plan; OpenAI released ChatGPT Agent and announced that an experimental model achieved gold medal-level performance on the 2025 International Mathematical Olympiad.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

The AI Action Plan

On the 23rd, the White House released its AI Action Plan. The document is the outcome of a January executive order that required the President's Science Advisor, ‘AI and Crypto Czar’, and National Security Advisor (currently Michael Kratsios, David Sacks, and Marco Rubio) to submit a plan to “sustain and enhance America's global AI dominance in order to promote human flourishing, economic competitiveness, and national security.” President Trump also delivered an hour-long speech on the plan, and signed three executive orders beginning to implement some of its policies.

Trump displaying an executive order at the [...]

---

Outline:

(00:34) The AI Action Plan

(07:36) ChatGPT Agent and IMO Gold

(12:48) In Other News

---

First published:
July 31st, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-60-the-ai-action

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #59: EU Publishes General-Purpose AI Code of Practice

Tue, 15 Jul 2025 18:05:30 GMT

Plus: Meta Superintelligence Labs.

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition: The EU published a General-Purpose AI Code of Practice for AI providers, and Meta is spending billions revamping its superintelligence development efforts.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

EU Publishes General-Purpose AI Code of Practice

In June 2024, the EU adopted the AI Act, which remains the world's most significant law regulating AI systems. The Act bans some uses of AI like social scoring and predictive policing and limits other “high risk” uses such as generating credit scores or evaluating educational outcomes. It also regulates general-purpose AI (GPAI) systems, imposing transparency requirements, copyright protection policies, and safety and security standards for models that pose systemic risk (defined as those trained [...]

---

Outline:

(00:31) EU Publishes General-Purpose AI Code of Practice

(04:50) Meta Superintelligence Labs

(06:17) In Other News

---

First published:
July 15th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-59-eu-publishes

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #58: Senate Removes State AI Regulation Moratorium

Thu, 03 Jul 2025 16:25:08 GMT

Plus: Judges Split on Whether Training AI on Copyrighted Material is Fair Use.

In this edition: The Senate removes a provision from Republican's “Big Beautiful Bill” aimed at restricting states from regulating AI; two federal judges split on whether training AI on copyrighted books in fair use.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Senate Removes State AI Regulation Moratorium

The Senate removed a provision from Republican's “Big Beautiful Bill” aimed at restricting states from regulating AI. The moratorium would have prohibited states from receiving federal broadband expansion funds if they regulated AI—however, it faced procedural and political challenges in the Senate, and was ultimately removed in a vote of 99-1. Here's what happened.

A watered-down moratorium cleared the Byrd Rule. In an attempt to bypass the Byrd Rule, which prohibits policy provisions in budget bills, the Senate Commerce Committee revised the [...]

---

Outline:

(00:35) Senate Removes State AI Regulation Moratorium

(03:04) Judges Split on Whether Training AI on Copyrighted Material is Fair Use

(07:19) In Other News

---

First published:
July 3rd, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-58-senate-removes

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #57: The RAISE Act

Tue, 17 Jun 2025 16:32:00 GMT

In this edition: The New York Legislature passes an act regulating frontier AI—but it may not be signed into law for some time.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

The RAISE Act

New York may soon become the first state to regulate frontier AI systems. On June 12, the state's legislature passed the Responsible AI Safety and Education (RAISE) Act. If New York Governor Kathy Hochul signs it into law, the RAISE Act will be the most significant state AI legislation in the U.S.

New York's RAISE Act imposes four guardrails on frontier labs: developers must publish a safety plan, hold back unreasonably risky models, disclose major incidents, and face penalties for non-compliance.

Publish and maintain a safety plan. Before deployment, developers must post a redacted “safety and security protocol,” transmit the plan to both the attorney general and the [...]

---

Outline:

(00:21) The RAISE Act

(04:43) In Other News

---

First published:
June 17th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-57-the-raise

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #56: Google Releases Veo 3

Wed, 28 May 2025 15:04:07 GMT

Plus, Opus 4 Demonstrates the Fragility of Voluntary Governance.

In this edition: Google released a frontier video generation model at its annual developer conference; Anthropic's Claude Opus 4 demonstrates the danger of relying on voluntary governance.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Google Releases Veo 3

Last week, Google made several AI announcements at I/O 2025, its annual developer conference. An announcement of particular note is Veo 3, Google's newest video generation model.

Frontier video and audio generation. Veo 3 outperforms other models on human preference benchmarks, and generates both audio and video.

Google showcasing a video generated with Veo 3. (Source)

If you just look at benchmarks, Veo 3 is a substantial improvement over other systems. But relative benchmark improvement only tells part of the story—the absolute capabilities of systems ultimately determine their usefulness. Veo 3 looks like a marked qualitative [...]

---

Outline:

(00:33) Google Releases Veo 3

(03:25) Opus 4 Demonstrates the Fragility of Voluntary Governance

---

First published:
May 28th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-56-google-releases

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #55: Trump Administration Rescinds AI Diffusion Rule, Allows Chip Sales to Gulf States

Tue, 20 May 2025 14:56:05 GMT

Plus, Bills on Whistleblower Protections, Chip Location Verification, and State Preemption.

In this edition: The Trump Administration rescinds the Biden-era AI diffusion rule and sells AI chips to the UAE and Saudi Arabia; Federal lawmakers propose legislation on AI whistleblowers, location verification for AI chips, and prohibiting states from regulating AI.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

The Center for AI Safety is also excited to announce the Summer session of our AI Safety, Ethics, and Society course, running from June 23 to September 14. The course, based on our recently published textbook, is open to participants from all disciplines and countries, and is designed to accommodate full-time work or study.

Applications for the Summer 2025 course are now open. The final application deadline is May 30th. Visit the course website to learn more and apply.

Trump Administration Rescinds AI Diffusion [...]

---

Outline:

(01:12) Trump Administration Rescinds AI Diffusion Rule, Allows Chip Sales to Gulf States

(04:14) Bills on Whistleblower Protections, Chip Location Verification, and State Preemption

(06:56) In Other News

---

First published:
May 20th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-55-trump-administration

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #54: OpenAI Updates Restructure Plan

Tue, 13 May 2025 00:00:00 GMT

Plus, AI Safety Collaboration in Singapore.

In this edition: OpenAI claims an updated restructure plan would preserve nonprofit control; A global coalition meets in Singapore to propose a research agenda for AI safety.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

OpenAI Updates Restructure Plan

On May 5th, OpenAI announced a new restructure plan. The announcement walks back a December 2024 proposal that would have had OpenAI's nonprofit—which oversees the company's for-profit operations—sell its controlling shares to the for-profit side of the company. That plan drew sharp criticism from former employees and civil‑society groups and prompted a lawsuit from co‑founder Elon Musk, who argued OpenAI was abandoning its charitable mission.

OpenAI claims the new plan preserves nonprofit control, but is light on specifics. Like the original plan, OpenAI's new plan would have OpenAI Global LLC become a public‑benefit corporation (PBC). However, instead of the nonprofit selling its [...]

---

Outline:

(00:31) OpenAI Updates Restructure Plan

(03:19) AI Safety Collaboration in Singapore

(05:42) In Other News

---

First published:
May 13th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-54-openai-updates

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #53: An Open Letter Attempts to Block OpenAI Restructuring

Tue, 29 Apr 2025 00:00:00 GMT

Plus, SafeBench Winners.

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition: Experts and ex-employees urge the Attorneys General of California and Delaware to block OpenAI's for-profit restructure; CAIS announces the winners of its safety benchmarking competition.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

An Open Letter Attempts to Block OpenAI Restructuring

A group of former OpenAI employees and independent experts published an open letter urging the Attorneys General (AGs) of California (where OpenAI operates) and Delaware (where OpenAI is incorporated) to block OpenAI's planned restructuring into a for-profit entity. The letter argues the move would fundamentally undermine the organization's charitable mission by jeopardizing the governance safeguards designed to protect control over AGI from profit motives.

OpenAI was founded with the charitable purpose to ensure that artificial general intelligence benefits all of humanity. OpenAI's original nonprofit structure, and later its capped-profit model, were designed to control profit motives in the development of AGI, which OpenAI defines as "highly autonomous systems that outperform humans at most economically valuable work." The structure was designed to prevent [...]

---

Outline:

(00:32) An Open Letter Attempts to Block OpenAI Restructuring

(04:23) SafeBench Winners

(08:58) Other News

---

First published:
April 29th, 2025

Source:
https://newsletter.safe.ai/p/an-open-letter-attempts-to-block

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #52: An Expert Virology Benchmark

Tue, 22 Apr 2025 16:08:58 GMT

Plus, AI-Enabled Coups.

In this edition: AI now outperforms human experts in specialized virology knowledge in a new benchmark; A new report explores the risk of AI-enabled coups.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

An Expert Virology Benchmark

A team of researchers (primarily from SecureBio and CAIS) has developed the Virology Capabilities Test (VCT), a benchmark that measures an AI system's ability to troubleshoot complex virology laboratory protocols. Results on this benchmark suggest that AI has surpassed human experts in practical virology knowledge.

VCT measures practical virology knowledge, which has high dual-use potential. While AI virologists could accelerate beneficial research in virology and infectious disease prevention, bad actors could misuse the same capabilities to develop dangerous pathogens. Like the WMDP benchmark, the VCT is designed to evaluate practical dual-use scientific knowledge—in this case, virology.

The benchmark consists of 322 multimodal questions [...]

---

Outline:

(00:29) An Expert Virology Benchmark

(04:04) AI-Enabled Coups

(07:58) Other news

---

First published:
April 22nd, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-52-an-expert

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #51: AI Frontiers

Tue, 15 Apr 2025 00:00:00 GMT

Plus, AI 2027.

In this newsletter, we cover the launch of AI Frontiers, a new forum for expert commentary on the future of AI. We also discuss AI 2027, a detailed scenario describing how artificial superintelligence might emerge in just a few years.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

AI Frontiers

Last week, CAIS introduced AI Frontiers, a new publication dedicated to gathering expert views on AI's most pressing questions. AI's impacts are wide-ranging, affecting jobs, health, national security, and beyond. Navigating these challenges requires a forum for varied viewpoints and expertise.

In this story, we’d like to highlight the publication's initial articles to give you a taste of the kind of coverage you can expect from AI Frontiers.

Why Racing to Artificial Superintelligence Would Undermine America's National Security. Researchers Corin Katzke (also an author of this newsletter) and Gideon Futerman [...]

---

Outline:

(00:33) AI Frontiers

(05:01) AI 2027

(10:02) Other News

---

First published:
April 15th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-51-ai-frontiers

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #50: AI Action Plan Responses

Mon, 31 Mar 2025 14:54:30 GMT

Plus, Detecting Misbehavior in Reasoning Models.

In this newsletter, we cover AI companies’ responses to the federal government's request for information on the development of an AI Action Plan. We also discuss an OpenAI paper on detecting misbehavior in reasoning models by monitoring their chains of thought.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

On January 23, President Trump signed an executive order giving his administration 180 days to develop an “AI Action Plan” to “enhance America's global AI dominance in order to promote human flourishing, economic competitiveness, and national security.”

Despite the rhetoric of the order, the Trump administration has yet to articulate many policy positions with respect to AI development and safety. In a recent interview, Ben Buchanan—Biden's AI advisor—interpreted the executive order as giving the administration time to develop its AI policies. The AI Action Plan will therefore likely [...]

---

First published:
March 31st, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-50-ai-action

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #49: AI Action Plan Responses

Mon, 31 Mar 2025 14:51:55 GMT

Plus, Detecting Misbehavior in Reasoning Models.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

---

First published:
March 31st, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-49-ai-action

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN

Thu, 06 Mar 2025 16:06:00 GMT

Plus, Measuring AI Honesty.

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required. In this newsletter, we discuss two recent papers: a policy paper on national security strategy, and a technical paper on measuring honesty in AI systems.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Superintelligence Strategy

CAIS director Dan Hendrycks, former Google CEO Eric Schmidt, and Scale AI CEO Alexandr Wang have authored a new paper, Superintelligence Strategy. The paper (and an in-depth expert version) argues that the development of superintelligence—AI systems that surpass humans in nearly every domain—is inescapably a matter of national security.

In this story, we introduce the paper's three-pronged strategy for national security in the age of advanced AI: deterrence, nonproliferation, and competitiveness.

Deterrence

The simultaneous power and danger of superintelligence presents [...]

---

Outline:

(00:20) Superintelligence Strategy

(01:09) Deterrence

(02:41) Nonproliferation

(04:04) Competitiveness

(05:33) Measuring AI Honesty

(09:24) Links

---

First published:
March 6th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-49-superintelligence

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Superintelligence Strategy: Expert Version

Wed, 05 Mar 2025 22:02:37 GMT

Superintelligence is destabilizing since it threatens other states’ survival—it could be weaponized, or states may lose control of it. Attempts to build superintelligence may face threats by rival states—creating a deterrence regime called Mutual Assured AI Malfunction (MAIM). In this paper, Dan Hendrycks, Eric Schmidt, and Alexandr Wang detail a strategy—focused on deterrence, nonproliferation, and competitiveness—for nations to navigate the risks of superintelligence.

Superintelligence Strategy: Standard Version

Wed, 05 Mar 2025 22:02:30 GMT

AISN #48: Utility Engineering and EnigmaEval

Tue, 18 Feb 2025 17:31:09 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required. Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

In this newsletter, we explore two recent papers from CAIS. We’d also like to highlight that CAIS is hiring for editorial and writing roles, including for a new online platform for journalism and analysis regarding AI's impacts on national security, politics, and economics.

Utility Engineering

A common view is that large language models (LLMs) are highly capable but fundamentally passive tools, shaping their responses based on training data without intrinsic goals or values. However, a new paper from the Center for AI Safety challenges this assumption, showing that LLMs exhibit coherent and structured value systems.

Structured preferences emerge with scale. The paper introduces Utility Engineering, a framework for analyzing and controlling AI [...]

---

Outline:

(00:26) Utility Engineering

(04:48) EnigmaEval

---

First published:
February 18th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-48-utility-engineering

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #47: Reasoning Models

Thu, 06 Feb 2025 17:38:10 GMT

Plus, State-Sponsored AI Cyberattacks.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Reasoning Models

DeepSeek-R1 has been one of the most significant model releases since ChatGPT. After its release, the DeepSeek's app quickly rose to the top of Apple's most downloaded chart and NVIDIA saw a 17% stock decline. In this story, we cover DeepSeek-R1, OpenAI's o3-mini and Deep Research, and the policy implications of reasoning models.

DeepSeek-R1 is a frontier reasoning model. DeepSeek-R1 builds on the company's previous model, DeepSeek-V3, by adding reasoning capabilities through reinforcement learning training. R1 exhibits frontier-level capabilities in mathematics, coding, and scientific reasoning—comparable to OpenAI's o1. DeepSeek-R1 also scored 9.4% on Humanity's Last Exam—at the time of its release, the highest of any publicly available system.

DeepSeek reports spending only about $6 million on the computing power needed to train V3—however, that number doesn’t include the full [...]

---

Outline:

(00:13) Reasoning Models

(04:58) State-Sponsored AI Cyberattacks

(06:51) Links

---

First published:
February 6th, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-47-reasoning

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #46: The Transition

Thu, 23 Jan 2025 17:09:14 GMT

Plus, Humanity's Last Exam, and the AI Safety, Ethics, and Society Course.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

The Transition

The transition from the Biden to Trump administrations saw a flurry of executive activity on AI policy, with Biden signing several last-minute executive orders and Trump revoking Biden's 2023 executive order on AI risk. In this story, we review the state of play.

Trump signing first-day executive orders. Source.

The AI Diffusion Framework. The final weeks of the Biden Administration saw three major actions related to AI policy. First, the Bureau of Industry and Security released its Framework for Artificial Intelligence Diffusion, which updates the US’ AI-related export controls. The rule establishes three tiers of countries 1) US allies, 2) most other countries, and 3) arms-embargoed countries.

Companies headquartered in tier-1 countries can freely deploy AI chips in other [...]

---

Outline:

(00:16) The Transition

(04:38) CAIS and Scale AI Introduce Humanitys Last Exam

(08:03) AI Safety, Ethics, and Society Course

(09:21) Links

---

First published:
January 23rd, 2025

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-46-the-transition

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #45: Center for AI Safety 2024 Year in Review

Thu, 19 Dec 2024 17:12:39 GMT

As 2024 draws to a close, we want to thank you for your continued support for AI safety and review what we’ve been able to accomplish. In this special-edition newsletter, we highlight some of our most important projects from the year.

The mission of the Center for AI Safety is to reduce societal-scale risks from AI. We focus on three pillars of work: research, field-building, and advocacy.

Research

CAIS conducts both technical and conceptual research on AI safety. Here are some highlights from our research in 2024:

Circuit Breakers. We published breakthrough research showing how circuit breakers can prevent AI models from behaving dangerously by interrupting crime-enabling outputs. In a jailbreaking competition with a prize pool of tens of thousands of dollars, it took twenty thousand attempts to jailbreak a model trained with circuit breakers. The paper was accepted to NeurIPS 2024.

The WMDP Benchmark. We developed the Weapons [...]

---

Outline:

(00:34) Research

(04:25) Advocacy

(06:44) Field-Building

(10:38) Looking Ahead

---

First published:
December 19th, 2024

Source:
https://newsletter.safe.ai/p/aisn-45-center-for-ai-safety-2024

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #44: The Trump Circle on AI Safety

Tue, 19 Nov 2024 16:08:17 GMT

Plus, Chinese researchers used Llama to create a military tool for the PLA, a Google AI system discovered a zero-day cybersecurity vulnerability, and Complex Systems.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

The Trump Circle on AI Safety

The incoming Trump administration is likely to significantly alter the US government's approach to AI safety. For example, Trump is likely to immediately repeal Biden's Executive Order on AI.

However, some of Trump's circle appear to take AI safety seriously. The most prominent AI safety advocate close to Trump is Elon Musk, who earlier this year supported SB 1047. However, he is not alone. Below, we’ve gathered some promising perspectives from other members of Trump's circle and incoming administration.

Trump and Musk at UFC 309. Photo Source.

Robert F. Kennedy Jr, Trump's pick for Secretary of Health and Human Services, said in [...]

---

Outline:

(00:24) The Trump Circle on AI Safety

(02:41) Chinese Researchers Used Llama to Create a Military Tool for the PLA

(04:14) A Google AI System Discovered a Zero-Day Cybersecurity Vulnerability

(05:27) Complex Systems

(08:54) Links

---

First published:
November 19th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-44-the-trump

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #43: White House Issues First National Security Memo on AI

Mon, 28 Oct 2024 15:22:21 GMT

Plus, AI and Job Displacement, and AI Takes Over the Nobels.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

White House Issues First National Security Memo on AI

On October 24, 2024, the White House issued the first National Security Memorandum (NSM) on Artificial Intelligence, accompanied by a Framework to Advance AI Governance and Risk Management in National Security.

The NSM identifies AI leadership as a national security priority. The memorandum states that competitors have employed economic and technological espionage to steal U.S. AI technology. To maintain a U.S. advantage in AI, the memorandum directs the National Economic Council to assess the U.S.'s competitive position in:

Semiconductor design and manufacturing
Availability of computational resources
Access to workers highly skilled in AI
Capital availability for AI development

The Intelligence Community must make gathering intelligence on competitors' operations against the [...]

---

Outline:

(00:18) White House Issues First National Security Memo on AI

(03:22) AI and Job Displacement

(09:13) AI Takes Over the Nobels

---

First published:
October 28th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-43-white-house

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #42: Newsom Vetoes SB 1047

Tue, 01 Oct 2024 16:17:13 GMT

Plus, OpenAI's o1, and AI Governance Summary.

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Newsom Vetoes SB 1047

On Sunday, Governor Newsom vetoed California's Senate Bill 1047 (SB 1047), the most ambitious legislation to-date aimed at regulating frontier AI models. The bill, introduced by Senator Scott Wiener and covered in a previous newsletter, would have required AI developers to test frontier models for hazardous capabilities and take steps to mitigate catastrophic risks. (CAIS Action Fund was a co-sponsor of SB 1047.)

Newsom states that SB 1047 is not comprehensive enough. In his letter to the California Senate, the governor argued that “SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves [...]

---

Outline:

(00:18) Newsom Vetoes SB 1047

(01:55) OpenAI's o1

(06:44) AI Governance

(10:32) Links

---

First published:
October 1st, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-42-newsom-vetoes

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #41: The Next Generation of Compute Scale

Wed, 11 Sep 2024 15:31:40 GMT

Plus, Ranking Models by Susceptibility to Jailbreaking, and Machine Ethics.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

The Next Generation of Compute Scale

AI development is on the cusp of a dramatic expansion in compute scale. Recent developments across multiple fronts—from chip manufacturing to power infrastructure—point to a future where AI models may dwarf today's largest systems. In this story, we examine key developments and their implications for the future of AI compute.

xAI and Tesla are building massive AI clusters. Elon Musk's xAI has brought its Memphis supercluster—“Colossus”—online. According to Musk, the cluster has 100k Nvidia H100s, making it the largest supercomputer in the world. Moreover, xAI plans to add 50k H200s in the next few months. For comparison, Meta's Llama 3 was trained on 16k H100s.

Meanwhile, Tesla's “Gigafactory Texas” is expanding to house an AI supercluster. Tesla's Gigafactory supercomputer [...]

---

Outline:

(00:18) The Next Generation of Compute Scale

(04:36) Ranking Models by Susceptibility to Jailbreaking

(06:07) Machine Ethics

---

First published:
September 11th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-41-the-next

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #40: California AI Legislation

Wed, 21 Aug 2024 16:08:50 GMT

Plus, NVIDIA Delays Chip Production, and Do AI Safety Benchmarks Actually Measure Safety?.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

SB 1047, the Most-Discussed California AI Legislation

California's Senate Bill 1047 has sparked discussion over AI regulation. While state bills often fly under the radar, SB 1047 has garnered attention due to California's unique position in the tech landscape. If passed, SB 1047 would apply to all companies performing business in the state, potentially setting a precedent for AI governance more broadly.

This newsletter examines the current state of the bill, which has had various amendments in response to feedback from various stakeholders. We'll cover recent debates surrounding the bill, support from AI experts, opposition from the tech industry, and public opinion based on polling.

The bill mandates safety protocols, testing procedures, and reporting requirements for covered AI models. The bill was [...]

---

Outline:

(00:18) SB 1047, the Most-Discussed California AI Legislation

(04:38) NVIDIA Delays Chip Production

(06:49) Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?

(10:22) Links

---

First published:
August 21st, 2024

Source:
https://newsletter.safe.ai/p/aisn-40-california-ai-legislation

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #39: Implications of a Trump Administration for AI Policy

Mon, 29 Jul 2024 00:00:00 GMT

Plus, Safety Engineering Overview.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Implications of a Trump administration for AI policy

Trump named Ohio Senator J.D. Vance—an AI regulation skeptic—as his pick for vice president. This choice sheds light on the AI policy landscape under a future Trump administration. In this story, we cover: (1) Vance's views on AI policy, (2) views of key players in the administration, such as Trump's party, donors, and allies, and (3) why AI safety should remain bipartisan.

Vance has pushed for reducing AI regulations and making AI weights open. At a recent Senate hearing, Vance accused Big Tech companies of overstating risks from AI in order to justify regulations to stifle competition. This led tech policy experts to expect that Vance would favor looser AI regulations.

However, Vance has also praised Lina Khan, Chair of the Federal Trade [...]

---

Outline:

(00:18) Implications of a Trump administration for AI policy

(04:57) Safety Engineering

(08:49) Links

---

First published:
July 29th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-39-implications

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #38: Supreme Court Decision Could Limit Federal Ability to Regulate AI

Tue, 09 Jul 2024 00:00:00 GMT

Plus, “Circuit Breakers” for AI systems, and updates on China's AI industry.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Supreme Court Decision Could Limit Federal Ability to Regulate AI

In a recent decision, the Supreme Court overruled the 1984 precedent Chevron v. Natural Resources Defence Council. In this story, we discuss the decision's implications for regulating AI.

Chevron allowed agencies to flexibly apply expertise when regulating. The “Chevron doctrine” had required courts to defer to a federal agency's interpretation of a statute in the case that that statute was ambiguous and the agency's interpretation was reasonable. Its elimination curtails federal agencies’ ability to regulate—including, as this article from LawAI explains, their ability to regulate AI.

The Chevron doctrine expanded federal agencies’ ability to regulate in at least two ways. First, agencies could draw on their technical expertise to interpret ambiguous statutes [...]

---

First published:
July 9th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-38-supreme-court

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AISN #37: US Launches Antitrust Investigations

Tue, 18 Jun 2024 15:00:35 GMT

US Launches Antitrust Investigations

The U.S. Government has launched antitrust investigations into Nvidia, OpenAI, and Microsoft. The U.S. Department of Justice (DOJ) and Federal Trade Commission (FTC) have agreed to investigate potential antitrust violations by the three companies, the New York Times reported. The DOJ will lead the investigation into Nvidia while the FTC will focus on OpenAI and Microsoft.

Antitrust investigations are conducted by government agencies to determine whether companies are engaging in anticompetitive practices that may harm consumers and stifle competition.

Nvidia investigated for GPU dominance. The New York Times reports that concerns have been raised about Nvidia's dominance in the GPU market, “including how the company's software locks [...]

---

Outline:

(00:10) US Launches Antitrust Investigations

(02:58) Recent Criticisms of OpenAI and Anthropic

(05:40) Situational Awareness

(09:14) Links

---

First published:
June 18th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-37-us-launches

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #36: Voluntary Commitments are Insufficient

Thu, 30 May 2024 14:30:17 GMT

Voluntary Commitments are Insufficient

AI companies agree to RSPs in Seoul. Following the second AI Global Summit held in Seoul, the UK and Republic of Korea governments announced that 16 major technology organizations, including Amazon, Google, Meta, Microsoft, OpenAI, and xAI have agreed to a new set of Frontier AI Safety Commitments.

Some commitments from the agreement include:

Assessing risks posed by AI models and systems throughout the AI lifecycle.
Setting thresholds for severe risks, defining when a model or system would pose intolerable risk if not adequately mitigated.
Keeping risks within defined thresholds, such as by modifying system behaviors and implementing robust security controls.
Potentially halting development or deployment if risks cannot be sufficiently mitigated.

These commitments [...]

---

Outline:

(00:03) Voluntary Commitments are Insufficient

(02:45) Senate AI Policy Roadmap

(05:18) Chapter 1: Overview of Catastrophic Risks

(07:56) Links

---

First published:
May 30th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-35-voluntary

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #35: Lobbying on AI Regulation

Thu, 16 May 2024 14:30:52 GMT

OpenAI and Google Announce New Multimodal Models

In the current paradigm of AI development, there are long delays between the release of successive models. Progress is largely driven by increases in computing power, and training models with more computing power requires building large new data centers.

More than a year after the release of GPT-4, OpenAI has yet to release GPT-4.5 or GPT-5, which would presumably be trained on 10x or 100x more compute than GPT-4, respectively. These models might be released over the next year or two, and could represent large spikes in AI capabilities.

But OpenAI did announce a new model last week, called GPT-4o. The “o” stands for “omni,” referring to the fact that the model can use text, images, videos [...]

---

Outline:

(00:03) OpenAI and Google Announce New Multimodal Models

(02:36) The Surge in AI Lobbying

(05:29) How Should Copyright Law Apply to AI Training Data?

(10:10) Links

---

First published:
May 16th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-35-lobbying

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #34: New Military AI Systems

Wed, 01 May 2024 17:01:01 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

AI Labs Fail to Uphold Safety Commitments to UK AI Safety Institute

In November, leading AI labs committed to sharing their models before deployment to be tested by the UK AI Safety Institute. But reporting from Politico shows that these commitments have fallen through.

OpenAI, Anthropic, and Meta have all failed to share their models with the UK AISI before deployment. Only Google DeepMind, headquartered in London, has given pre-deployment access to UK AISI.

Anthropic released the most powerful publicly available language model, Claude 3, without any window for pre-release testing by the UK AISI. When asked for comment, Anthropic co-founder Jack Clark said, “Pre-deployment testing is a nice idea but very difficult to implement.”

When asked about their concerns with pre-deployment testing [...]

---

Outline:

(00:03) AI Labs Fail to Uphold Safety Commitments to UK AI Safety Institute

(02:17) New Bipartisan AI Policy Proposals in the US Senate

(06:35) Military AI in Israel and the US

(11:44) New Online Course on AI Safety from CAIS

(12:38) Links

---

First published:
May 1st, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-34-new-military

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #33: Reassessing AI and Biorisk

Thu, 11 Apr 2024 18:01:03 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

This week, we cover:

Consolidation in the corporate AI landscape, as smaller startups join forces with larger funders.
Several countries have announced new investments in AI, including Singapore, Canada, and Saudi Arabia.
Congress's budget for 2024 provides some but not all of the requested funding for AI policy. The White House's 2025 proposal makes more ambitious requests for AI funding.
How will AI affect biological weapons risk? We reexamine this question in light of new experiments from RAND, OpenAI, and others.

AI Startups Seek Support From Large Financial Backers

As AI development demands ever-increasing compute resources, only well-resourced developers can compete at the frontier. In practice, this means that AI startups must either partner with the world's [...]

---

Outline:

(00:45) AI Startups Seek Support From Large Financial Backers

(03:47) National AI Investments

(05:16) Federal Spending on AI

(08:35) An Updated Assessment of AI and Biorisk

(15:35) $250K in Prizes: SafeBench Competition Announcement

(16:08) Links

---

First published:
April 11th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-33-reassessing

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #32: Measuring and Reducing Hazardous Knowledge in LLMs

Thu, 07 Mar 2024 16:00:40 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Measuring and Reducing Hazardous Knowledge

The recent White House Executive Order on Artificial Intelligence highlights risks of LLMs in facilitating the development of bioweapons, chemical weapons, and cyberweapons.

To help measure these dangerous capabilities, CAIS has partnered with Scale AI to create WMDP: the Weapons of Mass Destruction Proxy, an open source benchmark with more than 4,000 multiple choice questions that serve as proxies for hazardous knowledge across biology, chemistry, and cyber.

This benchmark not only helps the world understand the relative dual-use capabilities of different LLMs, but it also creates a path forward for model builders to remove harmful information from their models through machine unlearning techniques.

Measuring hazardous knowledge in bio, chem, and cyber. Current evaluations of dangerous AI capabilities have [...]

---

Outline:

(00:03) Measuring and Reducing Hazardous Knowledge

(04:35) Language models are getting better at forecasting

(07:51) Proposals for Private Regulatory Markets

(14:25) Links

---

First published:
March 7th, 2024

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-32-measuring

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #31: A New AI Policy Bill in California

Wed, 21 Feb 2024 19:00:24 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

This week, we’ll discuss:

A new proposed AI bill in California which requires frontier AI developers to adopt safety and security protocols, and clarifies that developers bear legal liability if their AI systems cause unreasonable risks or critical harms to public safety.
Precedents for AI governance from healthcare and biosecurity.
The EU AI Act and job opportunities at their enforcement agency, the AI Office.

A New Bill on AI Policy in California

Several leading AI companies have public plans for how they’ll invest in safety and security as they develop more dangerous AI systems. A new bill in California's state legislature would codify this practice as a legal requirement, and clarify the legal liability faced by developers [...]

---

Outline:

(00:33) A New Bill on AI Policy in California

(04:38) Precedents for AI Policy: Healthcare and Biosecurity

(07:56) Enforcing the EU AI Act

(08:55) Links

---

First published:
February 21st, 2024

Source:
https://newsletter.safe.ai/p/aisn-31-a-new-ai-policy-bill-in-california

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #30: Investments in Compute and Military AI

Wed, 24 Jan 2024 17:00:18 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Compute Investments Continue To Grow

Pausing AI development has been proposed as a policy for ensuring safety. For example, an open letter last year from the Future of Life Institute called for a six-month pause on training AI systems more powerful than GPT-4.

But one interesting fact about frontier AI development is that it comes with natural pauses that can last many months or years. After releasing a frontier model, it takes time for AI developers to construct new compute clusters with larger numbers of more advanced computer chips. The supply of compute is currently unable to keep up with demand, meaning some AI developers cannot buy enough chips for their needs.

This explains why OpenAI was reportedly limited by GPUs last year. [...]

---

Outline:

(00:06) Compute Investments Continue To Grow

(03:48) Developments in Military AI

(07:19) Japan and Singapore Support AI Safety

(08:57) Links

---

First published:
January 24th, 2024

Source:
https://newsletter.safe.ai/p/aisn-30-investments-in-compute-and

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #29: Progress on the EU AI Act

Thu, 04 Jan 2024 16:00:37 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

A Provisional Agreement on the EU AI Act

On December 8th, the EU Parliament, Council, and Commission reached a provisional agreement on the EU AI Act. The agreement regulates the deployment of AI in high risk applications such as hiring and credit pricing, and it bans private companies from building and deploying AI for unacceptable applications such as social credit scoring and individualized predictive policing.

Despite lobbying by some AI startups against regulation of foundation models, the agreement contains risk assessment and mitigation requirements for all general purpose AI systems. Specific requirements apply to AI systems trained with >1025 FLOP such as Google's Gemini and OpenAI's GPT-4.

Minimum basic transparency requirements for all GPAI. The provisional agreement regulates foundation models — using the [...]

---

Outline:

(00:06) A Provisional Agreement on the EU AI Act

(04:55) Questions about Research Standards in AI Safety

(06:48) The New York Times sues OpenAI and Microsoft for Copyright Infringement

(10:34) Links

---

First published:
January 4th, 2024

Source:
https://newsletter.safe.ai/p/aisn-29-progress-on-the-eu-ai-act

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

The Landscape of US AI Legislation

Fri, 29 Dec 2023 03:00:49 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

This week we’re looking closely at AI legislative efforts in the United States, including:

Senator Schumer's AI Insight Forum
The Blumenthal-Hawley framework for AI governance
Agencies proposed to govern digital platforms
State and local laws against AI surveillance
The National AI Research Resource (NAIRR)

Senator Schumer's AI Insight Forum

The CEOs of more than a dozen major AI companies gathered in Washington on Wednesday for a hearing with the Senate. Organized by Democratic Majority Leader Chuck Schumer and a bipartisan group of Senators, this was the first of many hearings in their AI Insight Forum.

After the hearing, Senator Schumer said, “I asked everyone in the room, ‘Is government needed to play a role in regulating AI?’ and [...]

---

Outline:

(00:30) Senator Schumer's AI Insight Forum

(01:20) The Blumenthal-Hawley Framework

(03:09) Agencies Proposed to Govern Digital Platforms

(04:46) Deepfakes and Watermarking Legislation

(06:12) State and Local Laws Against AI Surveillance

(06:52) National AI Research Resource (NAIRR)

(08:18) Links

---

First published:
September 19th, 2023

Source:
https://newsletter.safe.ai/p/the-landscape-of-us-ai-legislation

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #28: Center for AI Safety 2023 Year in Review

Thu, 21 Dec 2023 19:30:59 GMT

As 2023 comes to a close, we want to thank you for your continued support for AI safety. This has been a big year for AI and for the Center for AI Safety. In this special-edition newsletter, we highlight some of our most important projects from the year. Thank you for being part of our community and our work.

Center for AI Safety's 2023 Year in Review

The Center for AI Safety (CAIS) is on a mission to reduce societal-scale risks from AI. We believe this requires research and regulation. These both need to happen quickly (due to unknown timelines on AI progress) and in tandem (because either one is insufficient on its own). To achieve this, we pursue three pillars of work: research, field-building, and advocacy.

Research

CAIS conducts both technical and conceptual research on AI safety. We pursue multiple overlapping strategies which can be layered together [...]

---

Outline:

(00:27) Center for AI Safety's 2023 Year in Review

(00:56) Research

(03:37) Field-Building

(07:35) Advocacy

(10:04) Looking Ahead

(10:23) Support Our Work

---

First published:
December 21st, 2023

Source:
https://newsletter.safe.ai/p/aisn-28-center-for-ai-safety-2023

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #27: Defensive Accelerationism

Thu, 07 Dec 2023 16:00:24 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Defensive Accelerationism

Vitalik Buterin, the creator of Ethereum, recently wrote an essay on the risks and opportunities of AI and other technologies. He responds to Marc Andreessen's manifesto on techno-optimism and the growth of the effective accelerationism (e/acc) movement, and offers a more nuanced perspective.

Technology is often great for humanity, the essay argues, but AI could be an exception to that rule. Rather than giving governments control of AI so they can protect us, Buterin argues that we should build defensive technologies that provide security against catastrophic risks in a decentralized society. Cybersecurity, biosecurity, resilient physical infrastructure, and a robust information ecosystem are some of the technologies Buterin believes we should build to protect ourselves from AI risks.

Technology has risks, but [...]

---

Outline:

(00:06) Defensive Accelerationism

(03:55) Retrospective on the OpenAI Board Saga

(07:58) Klobuchar and Thune's “light-touch” Senate bill

(10:23) Links

---

First published:
December 7th, 2023

Source:
https://newsletter.safe.ai/p/aisn-27-defensive-accelerationism

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #26: National Institutions for AI Safety

Wed, 15 Nov 2023 16:00:20 GMT

Also, Results From the UK Summit, and New Releases From OpenAI and xAI.

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

This week's key stories include:

The UK, US, and Singapore have announced national AI safety institutions.
The UK AI Safety Summit concluded with a consensus statement, the creation of an expert panel to study AI risks, and a commitment to meet again in six months.
xAI, OpenAI, and a new Chinese startup released new models this week.

UK, US, and Singapore Establish National AI Safety Institutions

Before regulating a new technology, governments often need time to gather information and consider their policy options. But during that time, the technology may diffuse through society, making it more difficult for governments to intervene. This process, termed the Collingridge Dilemma, is a fundamental challenge in technology policy.

But recently [...]

---

Outline:

(00:36) UK, US, and Singapore Establish National AI Safety Institutions

(03:53) UK Summit Ends with Consensus Statement and Future Commitments

(05:39) New Models From xAI, OpenAI, and a New Chinese Startup

(09:28) Links

---

First published:
November 15th, 2023

Source:
https://newsletter.safe.ai/p/national-institutions-for-ai-safety

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #25: White House Executive Order on AI, UK AI Safety Summit, and Progress on Voluntary Evaluations of AI Risks.

Tue, 31 Oct 2023 00:00:00 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

White House Executive Order on AI

While Congress has not voted on significant AI legislation this year, the White House has left their mark on AI policy. In June, they secured voluntary commitments on safety from leading AI companies. Now, the White House has released a new executive order on AI. It addresses a wide range of issues, and specifically targets catastrophic AI risks such as cyberattacks and biological weapons.

Companies must disclose large training runs. Under the executive order, companies that intend to train “dual-use foundation models” using significantly more computing power than GPT-4 must take several precautions. First, they must notify the White House before training begins. Then [...]

---

Outline:

(00:13) White House Executive Order on AI

(03:56) Kicking Off The UK AI Safety Summit

(06:18) Progress on Voluntary Evaluations of AI Risks

(08:52) Links

---

First published:
October 31st, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-25

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #24: Kissinger Urges US-China Cooperation on AI, China’s New AI Law, US Export Controls, International Institutions, and Open Source AI.

Wed, 18 Oct 2023 00:00:00 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

China's New AI Law, US Export Controls, and Calls for Bilateral Cooperation

China details how AI providers can fulfill their legal obligations. The Chinese government has passed several laws on AI. They’ve regulated recommendation algorithms and taken steps to mitigate the risk of deepfakes. Most recently, they issued a new law governing generative AI. It's less stringent than earlier draft version, but the law remains more comprehensive in AI regulation than any laws passed in the US, UK, or European Union.

The law creates legal obligations for AI providers to respect intellectual property rights, avoid discrimination, and uphold socialist values. But as with many AI policy proposals, these are [...]

---

Outline:

(00:15) China's New AI Law, US Export Controls, and Calls for Bilateral Cooperation

(04:58) Proposed International Institutions for AI

(08:15) Open Source AI: Risks and Opportunities

(11:25) Links

---

First published:
October 18th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-24

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #23: New OpenAI Models, News from Anthropic, and Representation Engineering.

Wed, 04 Oct 2023 00:00:00 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

OpenAI releases GPT-4 with Vision and DALL·E-3, announces Red Teaming Network

GPT-4 with vision and voice. When GPT-4 was initially announced in March, OpenAI demonstrated its ability to process and discuss images such as diagrams or photographs. This feature has now been integrated into GPT-4V. Users can now input images in addition to text, and the model will respond to both. Users can also speak to GPT-4V, and the model will respond verbally.

GPT-4V may be more vulnerable to misuse via jailbreaks and adversarial attacks. Previous research has shown that multimodal models, which can process multiple forms of input such as both text and images, are more vulnerable to adversarial attacks than text-only models. GPT-4V's System Card includes some experiments [...]

---

Outline:

(00:11) OpenAI releases GPT-4 with Vision and DALL·E-3, announces Red Teaming Network

(02:39) Writer's Guild of America Receives Protections Against AI Automation

(03:42) Anthropic receives $1.25B investment from Amazon, and announces several new policies

(06:21) Representation Engineering: A Top-Down Approach to AI Transparency

(07:57) Links

---

First published:
October 4th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-23

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #21: Google DeepMind’s GPT-4 Competitor, Military Investments in Autonomous Drones, The UK AI Safety Summit, and Case Studies in AI Policy.

Tue, 05 Sep 2023 00:00:00 GMT

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Google DeepMind’s GPT-4 Competitor

Computational power is a key driver of AI progress, and a new report suggests that Google’s upcoming GPT-4 competitor will be trained on unprecedented amounts of compute.

The model, currently named Gemini, may be trained by the end of this year with 5x more computational power than GPT-4. By the end of next year, the report projects that Google will have the ability to train a model with 20x more compute than GPT-4.

For reference, the compute difference between GPT-3 and GPT-4 was 100x. If these projections are true, Google’s new models could create a meaningful spike relative to current AI capabilities.

Google’s position [...]

---

Outline:

(00:14) Google DeepMind’s GPT-4 Competitor

(02:41) US Military Invests in Thousands of Autonomous Drones

(04:37) United Kingdom Prepares for Global AI Safety Summit

(06:15) Case Studies in AI Policy

(08:55) Links

---

First published:
September 5th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-21

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #20: LLM Proliferation, AI Deception, and Continuing Drivers of AI Capabilities.

Tue, 29 Aug 2023 00:00:00 GMT

AI Deception: Examples, Risks, Solutions

AI deception is the topic of a new paper from researchers at and affiliated with the Center for AI Safety. It surveys empirical examples of AI deception, then explores societal risks and potential solutions.

The paper defines deception as “the systematic production of false beliefs in others as a means to accomplish some outcome other than the truth.” Importantly, this definition doesn't necessarily imply that AIs have beliefs or intentions. Instead, it focuses on patterns of behavior that regularly cause false beliefs and would be considered deceptive if exhibited by humans.

Deception by Meta’s CICERO AI. Meta developed the AI system CICERO to play Diplomacy, a game where players build and betray alliances in [...]

---

Outline:

(00:11) AI Deception: Examples, Risks, Solutions

(04:35) Proliferation of Large Language Models

(09:25) Continuing Drivers of AI Capabilities

(14:30) Links

---

First published:
August 29th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-20

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

[Paper] “An Overview of Catastrophic AI Risks” by Dan Hendrycks, Mantas Mazeika and Thomas Woodside

Mon, 21 Aug 2023 09:00:01 GMT

Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose catastrophic risks. Although numerous risks have been detailed separately, there is a pressing need for a systematic discussion and illustration of the potential dangers to better inform efforts to mitigate them. This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty [...]

---

First published:
June 21st, 2023

Source:
https://arxiv.org/abs/2306.12001

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

[Paper] “X-Risk Analysis for AI Research” by Dan Hendrycks and Mantas Mazeika

Mon, 21 Aug 2023 09:00:00 GMT

Artificial intelligence (AI) has the potential to greatly improve society, but as with any powerful technology, it comes with heightened risks and responsibilities. Current AI research lacks a systematic discussion of how to manage long-tail risks from AI systems, including speculative long-term risks. Keeping in mind the potential benefits of AI, there is some concern that building ever more intelligent and powerful AI systems could eventually result in systems that are more powerful than us; some say this is like playing with fire and speculate that this could create existential risks (x-risks). To add precision and ground these discussions, we provide a guide for how to analyze AI x-risk, which consists of three parts: First, we review how systems can be made safer today, drawing on time-tested concepts from hazard analysis and systems safety that have been designed to steer large processes in safer directions. Next, we discuss strategies [...]

---

First published:
October 22nd, 2022

Source:
https://arxiv.org/abs/2206.05862

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

[Paper] “Unsolved Problems in ML Safety” by Dan Hendrycks, Nicholas Carlini, John Schulman and Jacob Steinhardt

Mon, 21 Aug 2023 09:00:00 GMT

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards (“Robustness”), identifying hazards (“Monitoring”), steering ML systems (“Alignment”), and reducing deployment hazards (“Systemic Safety”). Throughout, we clarify each problem’s motivation and provide concrete research directions.

---

First published:
June 16th, 2022

Source:
https://arxiv.org/abs/2109.13916

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #19: US-China Competition on AI Chips, Measuring Language Agent Developments, Economic Analysis of Language Model Propaganda, and White House AI Cyber Challenge.

Tue, 15 Aug 2023 00:00:00 GMT

US-China Competition on AI Chips

Modern AI systems are trained on advanced computer chips which are designed and fabricated by only a handful of companies in the world. The US and China have been competing for access to these chips for years. Last October, the Biden administration partnered with international allies to severely limit China’s access to leading AI chips.

Recently, there have been several interesting developments on AI chips. China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry. Meanwhile, the United States has struggled [...]

---

Outline:

(00:15) US-China Competition on AI Chips

(04:09) Measuring Language Agents Developments

(06:07) An Economic Analysis of Language Model Propaganda

(08:11) White House Competition Applying AI to Cybersecurity

(09:40) Links

---

First published:
August 15th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-19

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #18: Challenges of Reinforcement Learning from Human Feedback, Microsoft’s Security Breach, and Conceptual Research on AI Safety.

Tue, 08 Aug 2023 00:00:00 GMT

Challenges of Reinforcement Learning from Human Feedback

If you’ve used ChatGPT, you might’ve noticed the “thumbs up” and “thumbs down” buttons next to each of its answers. Pressing these buttons provides data that OpenAI uses to improve their models through a technique called reinforcement learning from human feedback (RLHF).

RLHF is popular for teaching models about human preferences, but it faces fundamental limitations. Different people have different preferences, but instead of modeling the diversity of human values, RLHF trains models to earn the approval of whoever happens to give feedback. Furthermore, as AI systems become more capable, they can learn to deceive human evaluators into giving undue approval.

Here we discuss a new [...]

---

Outline:

(00:13) Challenges of Reinforcement Learning from Human Feedback

(05:26) Microsoft’s Security Breach

(06:59) Conceptual Research on AI Safety

(09:25) Links

---

First published:
August 8th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-18

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #17: Automatically Circumventing LLM Guardrails, the Frontier Model Forum, and Senate Hearing on AI Oversight.

Tue, 01 Aug 2023 00:00:00 GMT

Automatically Circumventing LLM Guardrails

Large language models (LLMs) can generate hazardous information, such as step-by-step instructions on how to create a pandemic pathogen. To combat the risk of malicious use, companies typically build safety guardrails intended to prevent LLMs from misbehaving.

But these safety controls are almost useless against a new attack developed by researchers at Carnegie Mellon University and the Center for AI Safety. By studying the vulnerabilities in open source models such as Meta’s LLaMA 2, the researchers can automatically generate a nearly unlimited supply of “adversarial suffixes,” which are words and characters that cause any model’s safety controls to fail.

This discovery calls into question the fundamental limits of safety [...]

---

Outline:

(00:12) Automatically Circumventing LLM Guardrails

(05:40) AI Labs Announce the Frontier Model Forum

(07:54) Senate Hearing on AI Oversight

(14:42) Links

---

First published:
August 1st, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-17

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #16: White House Secures Voluntary Commitments from Leading AI Labs, and Lessons from Oppenheimer .

Tue, 25 Jul 2023 00:00:00 GMT

White House Unveils Voluntary Commitments to AI Safety from Leading AI Labs

Last Friday, the White House announced a series of voluntary commitments from seven of the world's premier AI labs. Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI pledged to uphold these commitments, which are non-binding and pertain only to forthcoming "frontier models" superior to currently available AI systems. The White House also notes that the Biden-Harris Administration is developing an executive order alongside these voluntary commitments.

The commitments are timely and technically well-informed, demonstrating the ability of federal policymakers to respond capably and quickly to AI risks. The Center for AI Safety supports these commitments as a precedent for cooperation on AI [...]

---

Outline:

(00:11) White House Unveils Voluntary Commitments to AI Safety from Leading AI Labs

(05:05) Lessons from Oppenheimer

(10:38) Links

---

First published:
July 25th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-16

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #15: China and the US take action to regulate AI, results from a tournament forecasting AI risk, updates on xAI’s plan, and Meta releases its open-source and commercially available Llama 2.

Wed, 19 Jul 2023 00:00:00 GMT

Both China and the US take action to regulate AI

Last week, regulators in both China and the US took aim at generative AI services. These actions show that China and the US are both concerned with AI safety. Hopefully, this is a sign they can eventually coordinate.

China’s new generative AI rules

On Thursday, China’s government released new rules governing generative AI. China’s new rules, which are set to take effect on August 15th, regulate publicly-available generative AI services. The providers of such services will be criminally liable for the content their services generate.

The rules specify illegal [...]

---

Outline:

(00:17) Both China and the US take action to regulate AI

(00:36) China’s new generative AI rules

(03:15) The FTC investigates OpenAI

(05:01) Results from a tournament forecasting AI risk

(08:18) Updates on xAI’s plan

(09:05) Meta releases Llama 2, open-source and commercially available

---

First published:
July 19th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-15

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #14: OpenAI’s ‘Superalignment’ team, Musk’s xAI launches, and developments in military AI use .

Wed, 12 Jul 2023 00:00:00 GMT

OpenAI announces a ‘superalignment’ team

On July 5th, OpenAI announced the ‘Superalignment’ team: a new research team given the goal of aligning superintelligence, and armed with 20% of OpenAI’s compute. In this story, we’ll explain and discuss the team’s strategy.

What is superintelligence? In their announcement, OpenAI distinguishes between ‘artificial general intelligence’ and ‘superintelligence.’ Briefly, ‘artificial general intelligence’ (AGI) is about breadth of performance. Generally intelligent systems perform well on a wide range of cognitive tasks. For example, humans are in many senses generally intelligent: we can learn how to drive a car, take a derivative, or play piano, even though evolution didn’t train us for those tasks. A superintelligent system would not only be [...]

---

Outline:

(00:11) OpenAI announces a ‘superalignment’ team

(03:50) Musk launches xAI

(05:12) Developments in Military AI Use

---

First published:
July 12th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-14

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #13: An interdisciplinary perspective on AI proxy failures, new competitors to ChatGPT, and prompting language models to misbehave.

Wed, 05 Jul 2023 00:00:00 GMT

Interdisciplinary Perspective on AI Proxy Failures

In this story, we discuss a recent paper on why proxy goals fail. First, we introduce proxy gaming, and then summarize the paper’s findings.

Proxy gaming is a well-documented failure mode in AI safety. For example, social media platforms use AI systems to recommend content to users. These systems are sometimes built to maximize the amount of time a user spends on the platform. The idea is that the time the user spends on the platform approximates the quality of the content being recommended. However, a user might spend even more time on a platform because they’re responding to an enraging post or interacting [...]

---

Outline:

(00:13) Interdisciplinary Perspective on AI Proxy Failures

(06:06) A Flurry of AI Fundraising and Model Releases

(12:53) Adversarial Inputs Make Chatbots Misbehave

(15:52) Links

---

First published:
July 5th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-13

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #12: Policy Proposals from NTIA’s Request for Comment, and Reconsidering Instrumental Convergence.

Tue, 27 Jun 2023 00:00:00 GMT

Policy Proposals from NTIA’s Request for Comment

The National Telecommunications and Information Administration publicly requested comments on the matter from academics, think tanks, industry leaders, and concerned citizens. They asked 34 questions and received more than 1,400 responses on how to govern AI for the public benefit. This week, we cover some of the most promising proposals found in the NTIA submissions.

Technical Proposals for Evaluating AI Safety

Several NTIA submissions focused on the technical question of how to evaluate the safety of an AI system. We review two areas of active research: red-teaming and transparency.

Red Teaming: Acting like an Adversary

Several submissions proposed government support for evaluating AIs via red teaming. In this evaluation method, a [...]

---

Outline:

(00:11) Policy Proposals from NTIA’s Request for Comment

(00:48) Technical Proposals for Evaluating AI Safety

(01:04) Red Teaming: Acting like an Adversary

(02:24) Transparency: Understanding AIs From the Inside

(03:51) Governance Proposals for Improving Safety Processes

(04:25) Requiring a License for Frontier AI Systems

(06:29) Unifying Sector-Specific Expertise and General AI Oversight

(07:51) Does Antitrust Prevent Cooperation Between AI Labs?

(08:40) Reconsidering Instrumental Convergence

(10:39) Links

---

First published:
June 27th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-12

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #11: An Overview of Catastrophic AI Risks.

Thu, 22 Jun 2023 00:00:00 GMT

An Overview of Catastrophic AI Risks

Global leaders are concerned that artificial intelligence could pose catastrophic risks. 42% of CEOs polled at the Yale CEO Summit agree that AI could destroy humanity in five to ten years. The Secretary General of the United Nations said we “must take these warnings seriously.” Amid all these frightening polls and public statements, there’s a simple question that’s worth asking: why exactly is AI such a risk?

The Center for AI Safety has released a new paper to provide a clear and comprehensive answer to this question. We detail the precise risks posed by AI, the structural dynamics making these problems so difficult to solve, and the technical, social, and political responses required to overcome this [...]

---

Outline:

(00:08) An Overview of Catastrophic AI Risks

(00:56) Malicious actors can use AIs to cause harm.

(02:18) Racing towards an AI disaster.

(04:05) Safety should be a goal, not a constraint.

(05:46) The challenge of AI control.

(07:53) Positive visions for the future of AI.

(09:02) Links

---

First published:
June 22nd, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-11

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #10: How AI could enable bioterrorism, and policymakers continue to focus on AI .

Tue, 13 Jun 2023 00:00:00 GMT

How AI could enable bioterrorism

Only a hundred years ago, no person could have single handedly destroyed humanity. Nuclear weapons changed this situation, giving the power of global annihilation to a small handful of nations with powerful militaries. Now, thanks to advances in biotechnology and AI, a much larger group of people could have the power to create a global catastrophe.

This is the upshot of a new paper from MIT titled “Can large language models democratize access to dual-use biotechnology?” The authors demonstrate that today’s language models are capable of providing detailed instructions for non-expert users about how to create pathogens that could cause a global pandemic.

Language models can help users build dangerous [...]

---

Outline:

(00:10) How AI could enable bioterrorism

(03:48) Policymakers continue to focus on AI

(05:27) Links

---

First published:
June 13th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-10

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #9: Statement on Extinction Risks, Competitive Pressures, and When Will AI Reach Human-Level? .

Tue, 06 Jun 2023 00:00:00 GMT

Top Scientists Warn of Extinction Risks from AI

Last week, hundreds of AI scientists and notable public figures signed a public statement on AI risks written by the Center for AI Safety. The statement reads:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

The statement was signed by a broad, diverse coalition. The statement represents a historic coalition of AI experts — along with philosophers, ethicists, legal scholars, economists, physicists, political scientists, pandemic scientists, nuclear scientists, and climate scientists — establishing the risk of extinction from advanced, future AI systems as one of the world’s most important problems.

The international community is [...]

---

Outline:

(00:10) Top Scientists Warn of Extinction Risks from AI

(03:35) Competitive Pressures in AI Development

(07:22) When Will AI Reach Human Level?

(12:47) Links

---

First published:
June 6th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-9

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #8: Why AI could go rogue, how to screen for AI risks, and grants for research on democratic governance of AI.

Tue, 30 May 2023 00:00:00 GMT

Yoshua Bengio makes the case for rogue AI

AI systems pose a variety of different risks. Renowned AI scientist Yoshua Bengio recently argued for one particularly concerning possibility: that advanced AI agents could pursue goals in conflict with human values.

Human intelligence has accomplished impressive feats, from flying to the moon to building nuclear weapons. But Bengio argues that across a range of important intellectual, economic, and social activities, human intelligence could be matched and even surpassed by AI.

How would advanced AIs change our world? Many technologies are tools, such as toasters and calculators, which humans use to accomplish our goals. AIs are different, Bengio says. [...]

---

Outline:

(00:11) Yoshua Bengio makes the case for rogue AI

(05:11) How to screen AIs for extreme risks

(09:12) Funding for Work on Democratic Inputs to AI

(10:43) Links

---

First published:
May 30th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-8

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #7: Disinformation, recommendations for AI labs, and Senate hearings on AI.

Tue, 23 May 2023 00:00:00 GMT

How AI enables disinformation

Yesterday, a fake photo generated by an AI tool showed an explosion at the Pentagon. The photo was falsely attributed to Bloomberg News and circulated quickly online. Within minutes, the stock market declined sharply, only to recover after it was discovered that the picture was a hoax.

This story is part of a broader trend. AIs can now generate text, audio, and images that are unnervingly similar to their naturally occurring counterparts. How will this affect our world, and what kinds of solutions are available?

The fake image generated by an AI showed an explosion at the Pentagon.

AIs can generate personalized scams. When John Podesta was the chair of Hillary Clinton’s 2016 presidential campaign [...]

---

Outline:

(00:10) How AI enables disinformation

(05:38) Governance recommendations on AI safety

(08:21) Senate hearings on AI regulation

(11:10) Links

---

First published:
May 23rd, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-7

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #6: Examples of AI safety progress, Yoshua Bengio proposes a ban on AI agents, and lessons from nuclear arms control .

Tue, 16 May 2023 00:00:00 GMT

Examples of AI safety progress

Training AIs to behave safely and beneficially is difficult. They might learn to game their reward function, deceive human oversight, or seek power. Some argue that researchers have not made much progress in addressing these problems, but here we offer a few examples of progress on AI safety.

Detecting lies in AI outputs. Language models often output false text, but a recent paper suggests they understand the truth in ways not reflected in their output. By analyzing a model’s internals, we can calculate the likelihood that a model believes a statement is true. The finding has been replicated in models that answer [...]

---

Outline:

(00:13) Examples of AI safety progress

(03:56) Yoshua Bengio proposes a ban on AI agents

(07:19) Lessons from Nuclear Arms Control for Verifying AI Treaties

(10:02) Links

---

First published:
May 16th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-6

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #5: Geoffrey Hinton speaks out on AI risk, the White House meets with AI labs, and Trojan attacks on language models.

Tue, 09 May 2023 00:00:00 GMT

Geoffrey Hinton is concerned about existential risks from AI

Geoffrey Hinton won the Turing Award for his work on AI. Now he says that part of him regrets his life’s work, as he believes that AI poses an existential threat to humanity. As Hinton puts it, “it’s quite conceivable that humanity is just a passing phase in the evolution of intelligence.”

AI is developing more rapidly than Hinton expected. In 2015, Andrew Ng argued that worrying about AI risk is like worrying about overpopulation on Mars. Geoffrey Hinton also used to believe that advanced AI was decades away, but recent progress has changed his views. Now he says [...]

---

Outline:

(00:12) Geoffrey Hinton is concerned about existential risks from AI

(02:32) White House meets with AI labs

(04:22) Trojan Attacks on Language Models

(06:51) Assorted Links

---

First published:
May 9th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-5

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #4: AI and cybersecurity, persuasive AIs, weaponization, and Hinton talks AI risks.

Tue, 02 May 2023 00:00:00 GMT

Cybersecurity Challenges in AI Safety

Meta accidentally leaks a language model to the public. Meta’s newest language model, LLaMa, was publicly leaked online against the intentions of its developers. Gradual rollout is a popular goal with new AI models, opening access to academic researchers and government officials before sharing models with anonymous internet users. Meta intended to use this strategy, but within a week of sharing the model with an approved list of researchers, an unknown person who had been given access to the model publicly posted it online.

How can AI developers selectively share their models? One inspiration could be the film industry, which places watermarks and tracking technology on “screener” copies of movies sent [...]

---

Outline:

(00:11) Cybersecurity Challenges in AI Safety

(02:48) Artificial Influence: An Analysis Of AI-Driven Persuasion

(05:37) Building Weapons with AI

(07:47) Assorted Links

---

First published:
May 2nd, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-4

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #3: AI policy proposals and a new challenger approaches.

Tue, 25 Apr 2023 00:00:00 GMT

Policy Proposals for AI Safety

Critical industries rely on the government to protect consumer safety. The FAA approves new airplane designs, the FDA tests new drugs, and the SEC and CFPB regulate risky financial instruments. Currently, there is no analogous set of regulations for AI safety.

This could soon change. President Biden and other members of Congress have recently been vocal about the risks of artificial intelligence and the need for policy solutions.

From guiding principles to enforceable laws. Previous work on AI policy such as the White House Blueprint for an AI Bill of Rights and the NIST AI Risk Management Framework has articulated guiding principles like interpretability, robustness, and privacy. But these recommendations are not enforceable – AI [...]

---

Outline:

(00:09) Policy Proposals for AI Safety

(04:19) Competitive Pressures in AI Development

---

First published:
April 25th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-3

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #2: ChaosGPT and the rise of language model agents, evolutionary pressures and AI, AI safety in the media.

Tue, 18 Apr 2023 00:00:00 GMT

ChaosGPT and the Rise of Language Agents

Chatbots like ChatGPT usually only respond to one prompt at a time, and a human user must provide a new prompt to get a new response. But an extremely popular new framework called AutoGPT automates that process. With AutoGPT, the user provides only a high-level goal, and the language model will create and execute a step-by-step plan to accomplish the goal.

AutoGPT and other language agents are still in their infancy. They struggle with long-term planning and repeat their own mistakes. Yet because they limit human oversight of AI actions, these agents are a step towards dangerous deployment of autonomous AI.

Individual bad actors [...]

---

Outline:

(00:12) ChaosGPT and the Rise of Language Agents

(02:49) Natural Selection Favors AIs over Humans

(05:17) AI Safety in the Media

---

First published:
April 18th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-2

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.

AISN #1: Public opinion on AI, plugging ChatGPT into the internet, and the economic impacts of language models..

Mon, 10 Apr 2023 00:00:00 GMT

Growing concerns about rapid AI progress

Recent advancements in AI have thrust it into the center of attention. What do people think about the risks of AI?

The American public is worried. 46% of Americans are concerned that AI will cause “the end of the human race on Earth,” according to a recent poll by YouGov. Young people are more likely to express such concerns, while there are no significant differences in responses between people of different genders or political parties. Another poll by Monmouth University found broad support for AI regulation, with 55% supporting the creation of a federal agency that governs AI similar to how the FDA approves drugs and [...]

---

Outline:

(00:12) Growing concerns about rapid AI progress

(02:53) Plugging ChatGPT into email, spreadsheets, the internet, and more

(05:35) Which jobs could be affected by language models?

---

First published:
April 10th, 2023

Source:
https://newsletter.safe.ai/p/ai-safety-newsletter-1

---

Want more? Check out our ML Safety Newsletter for technical safety research.

Narrated by TYPE III AUDIO.