The Role of AISIs in AI Governance
Summary
Rob Reich presented “The Role of AISIs in AI Governance“ on March 12, 2025 at FAR.AI Labs Seminar.
SESSION Transcript
Feel free to call me Rob if you’re comfortable with that. The one thing I ask if you’re at all a political junkie, especially here in the East Bay, you’ll recognize that I am not the former Secretary of Labor who teaches econ at Berkeley. No cutesy econ videos for me.
All right, I’m a political philosopher as Isaac said by way of introduction. My main appointment is in the Political Science Department at Stanford and I’ve been hanging out for the past seven, eight years over the engineering quad chiefly with AI people.
I have been trying in my various ways by way of giving you introduction to the work I did at the USAI Safety Institute, trying my various ways initially on campus. First, to try to understand since I had no technical training, I have a philosophical background why undergraduate students in the early 2010s began voting with their feet to leave the social sciences chiefly to some degree the humanities and begin majoring computer science and record numbers, and then worrying not because I thought that was a bad thing, I thought that was a fine thing to do. But worrying that the liberal arts orientation of the university was being, to some degree, compromised because of the somewhat vocational aspect of the technical skills that people got in the Computer Science Department.
My distant observation on campus was that people arrived at Stanford as 17 or 18-year-olds already determined to be computer science majors doing AI ML. They hopped on the computer science major bandwagon as a first semester student was basically a conveyor belt that then in their junior or senior year deposited them at a startup company or a big tech company with not much broader thinking about the policy governance aspects of the important work that they were doing.
So rather than launching a bunch of courses from my home base in philosophy or political science, I thought I would try to make a collaboration. That’s the nice way of putting it with the folks in computer science to co-teach with them.
And so I joined up with the most popular professor at Stanford. I don’t know if any of you are Stanford grads, but if you are, you’ll have known or remember Mehran Sahami, who’s currently the chair of the Computer Science Department. He teaches the intro to computer science class, the largest class on Stanford’s campus.
He and I, along with a fellow political scientist, began teaching a class on ethics, policy and technological change. That yielded the book that Isaac mentioned called, System Error. That yielded then teaching classes online to people who already worked in the tech industry. I began personally to focus more and more on AI safety and AI governance. And then that led to this appointment I had over the course of the past year at the U.S. AI Safety Institute.
I’d also mention because my – I was just asking before we got started here where everyone was from. And these different little AI safety organizations that seem to have geographically clustered over here in the East Bay with some either formal or informal connection to open philanthropy and the funding mechanisms of AI safety and/or the rationalist and effective altruist communities, which I know have a good presence here in the East Bay.
I mentioned that because, as Isaac also said, I previously wrote a whole bunch of stuff about philanthropy. I was on the board of GiveWell when it gave birth to the Open Philanthropy entity created out of GiveWell, and still do a bunch of writing, thinking, talking about effective altruism and know a good number of the people at the Open Philanthropy project.
Does FAR.AI have funding from Open Philanthropy? I think so. Yes. Right. It’s a good bet whenever one shows up. Yup.
I’ll just mention by way of addition, I have an affiliation these days with the Carnegie Endowment for International Peace, which has been running a bunch of so-called track two dialogues on AI governance, AI security, AI safety, and know a good bunch of the team down at the RAND Corporation under Jason Matheny, if any of you are familiar with their work.
So my goal with just the kind of brief remarks I want to make are, first, to signal to you that I’m not a deeply technically skilled AI safety researcher, rather someone interested in broader philosophical questions about AI to begin with, which is a familiar thing, I think, to anybody thinking about AI safety, security and governance.
And then I want to share with you a little bit about my experience in the AI Safety Institute, what its main mission has been, what it aims to accomplish more broadly, and then I’ll open it up to questions.
So let me – you possibly have cross paths with these people. I was with the U.S. AI Safety Institute from April 2024 until just this January 17th, the day before the inauguration of the new administration. I was, I think, like higher number five at the U.S. AI Safety Institute along, I was hired the same day that Paul Christiano was hired. Tony Wang, I saw on the website is an affiliate of some of our AI. He came on board sometime, I think, in the middle of the summer.
And I had lots of productive collaborations and conversations with the technical crowd. Unlike the CHIPS Act, which created a fairly significant team in the Department of Commerce under then Secretary Gina Raimondo, the U.S. AI Safety Institute, as if you paid any attention to this, as you know, never was congressionally authorized or funded.
So functionally speaking, the team had to be stood up on the basis of what is basically couch change that remained within the Department of Commerce budget. So it was always a very small staff.
One of my chief qualifications, I guess, for the position consisted in the fact that I didn’t cost the government anything. I was loaned by Stanford to the U.S. government. It’s a feature that people from other countries often find mystifying.
And I’ll confess, as a political scientist and philosopher, I, too, find it rather strange that in the United States, a law exists in which external entities can pay for people to become legal full time employees of the United States government. Often, those are universities or foundations. The Australian is baffled, baffled by this. Yes, it’s a bizarre feature of the U.S. government.
So, I was there on a full time basis between April and January. Why was the U.S. AI Safety Institute created? Let’s just cover that territory. This, I think, is fairly well known to this audience. But just in case not, the Biden-Harris administration negotiated with the CEOs of many of the companies producing frontier AI models in the summer of, let’s see, this would have been 2023 or so, spring of 2023, that the companies would provide voluntary access to the pre-deployment newest models for the purpose of testing and evaluation. Chiefly for national security sensitive concerns, biological, chemical weapons, nuclear access and cyberattacks, as well as autonomous replication and other things.
And once the executive order came out in October of 2023, the vice president, Kamala Harris, went to the very first AI summit at Bletchley Park in the UK. The UK had already announced the creation of its AI Safety Institute, and Vice President Harris announced that the U.S. would create one as well, with the main aspiration of then designing and administering the tests and evaluations that were a part of this voluntary commitment framework that had been pre-negotiated by the Biden-Harris administration with the companies.
So the vice president announces this. I think the UK summit was November 1st, October 30th, something like that, 2023. The director of the U.S. AI Safety Institute is selected, if my memory is right, in February of 2024. That person was Elizabeth Kelly, who came from the White House, then over to the Department of Commerce to lead this new team. I and Paul and several other people then are hired on board in April, and then the team grows over the course of the year.
I’ll note, to anticipate what maybe is already on your mind, to the best of my knowledge, and certainly this was true up until January 17th, despite the transition administrations here in the United States, no company has departed the voluntary commitment framework, and the vast majority of the people who were hired and worked during the Biden administration at AC remain still working there.
Okay. I’ll say one thing about my general orientation to the work I do overall, but then in particular at AC, and then give you an overview quickly of what AC’s three main pillars of work were, and just a little bit of detail about each, and then throw it open to questions about that.
So I have liked to communicate to people like you who know something about the field, as well as people who don’t know much about AI at all, that the way to communicate, at least for me, about the most interesting projects to work on, is to say that AI governance is the phrase of choice, not AI ethics, not AI policy, but rather AI governance. And the reason I think governance is a better term is that governance for me allows two important interconnected or related tracks of work.
First, governance allows one to work on law, policy, and regulation that comes distinctively from government. Those are the familiar kinds of ways of thinking about the EU AI office and all of the work that it’s doing there, the kind of traditional coercive force of government-backed law, policy, and regulation that helps to structure the field and creates binding law.
That’s perhaps obvious. AI policy or AI regulation would successfully communicate that. But I prefer AI governance because it allows us also to talk about a second track of work, which is about standards, norms, and professional responsibilities that emerge distinctively from the respective profession in question.
So in the absence of working on both tracks, I feel like the two are necessarily interrelated. And I’ll just give you one short example. A government could easily pass a law that says something like there shalt be digital watermarking. But, of course, the government is partly because of its lack of technical expertise, but also partly because it’s not the role of the lawmaker then to create the technical standard of watermarking.
Passing a law without some interaction with the professional norms and standards that exist within the expertise in the field is functionally inert or impossible to then enforce. So there has to be, in my view, some interplay between the kinds of expertise that rests in the heads of the people deep in the profession and then the policymaking community, which should have general policymaking authority rather than just specific policymaking authority, and the two intersect.
And I’ll note just by way of perhaps mild provocation for you, my own view is that the kind of developmental status of the computer science community in general and the AIML community in particular as professionals is highly immature.
AIML is an immature discipline. And I mean that as a kind of descriptive matter as well as a mildly intended insult. Descriptively, saying the obvious to those of you with technical skills, computer science only comes into existence as a field of study, something formally organized within universities in the 1950s.
AIML is really only significant in terms of people studying it and acquiring power in the world in the 2000s. And when you compare the professional norms and standards and internally felt sense of responsibilities of other longer standing fields like biomedical research or the law, for example, you’ll find a much denser footprint of norms, standards and responsibilities than you’ll find in AIML or computer science more generally.
So all of that then is to say, I thought there was an opportunity when I was given the permission to join the AI Safety Institute to imagine working on law, policy, and regulation. Although I’ll say something about why the U.S. AI Safety Institute was not a body that promulgated those things. It was meant to provide information to the policymakers that might.
But I wanted to use the perch of the U.S. AI Safety Institute as a kind of bully pulpit to try to stimulate the more rapid maturation of the AIML field as a set of professional understandings about what a responsible professional does within the field.
If anyone’s interested in hearing more about that, I’m happy to answer questions later. Okay, so AI governance is the frame for the work. We got law, policy, and regulation on the one hand, norms, standards, professional responsibilities on the other. And AC is created chiefly then to be the body that designs and administers the tests and evaluations under this voluntary commitment framework, chiefly on national security sensitive topics.
So, the Safety Institute has basically three main pillars of work. Number one is to advance the science of AI safety in all manner. This is, I’m assuming for this audience, much of your own concern, so I won’t say too much more about what that means.
But for us with inside government, that meant something like, how could you design more empirically sound tests of the broad capabilities of powerful AI models? How could you design more empirically robust and replicable tests and evaluations of particular harms or risks? And how could you then knit some of these tests and evaluations into some overarching risk management framework?
Rather than the kind of throw spaghetti at the wall practice of red teaming, which is an example of the preposterously immature standards within AIML that every company makes up a bunch of stuff and they call it red teaming and put it on their website.
If they do about what counts as having done something responsible, the goal was to get a scientific practice of capability evaluation and risk evaluation that then could be counted as a more replicable science rather than a kind of useful, but basically made up practice that has no interoperable standards or common practices across industries.
All right. Advance the science of AI safety. Number two is the task then of actually developing the tests and evaluations that would be administered. Paul Christiano is the chief technology officer, so that fell within his main ambit. And then that included, of course, all of the detailed agreements that were signed with the various companies that governed the mechanisms of access to the frontier model, the circumstances in which information about the tests and evaluations would be shared, et cetera, et cetera. So, that’s number two.
And the third pillar of work was the domestic and global dissemination of the practices of AI safety, and the coordination of broader communities to develop a field of AI safety more generally.
Practically speaking, what that meant for the 10 or so months I was at the AI Safety Institute was to help to launch the International Network of AI Safety Institutes, which took place in November, late November 2024 here in San Francisco. So just after the election, the International Network of AI Safety Institutes was launched with 10 initial government members.
I use that odd phrase government members rather than states because the EU is one of the members, along with places like Singapore, Japan, Canada, France, UK, Kenya, a few other places, which is designed to continue to advance and disseminate the work of AI safety.
I’ll say just by way of conclusion, then, that, again, to the best of my knowledge, the team at the U.S. AI Safety Institute continues to do the work that it’s been doing. I, like anyone else paying attention to the news these days here in the U.S., have been concerned initially that perhaps the Safety Institute would be cut or in some respect compromised.
So far as I can make out, there has been no deliberate decision to do so, which I take as a good sign about the importance of the work. It is a helpful feature of the Safety Institute’s work that it has been focused relatively narrowly on national security sensitive concerns. And so I want to think that it’s in the equal interest of the Trump-Vance administration to wonder about the potential of biological weapon uplift or autonomous cyberattacks.
And the Safety Institute’s technical expertise is just as useful to the Trump-Vance administration as it was to the Biden-Harris administration. And you’ll notice, of course, that the U.K. rebranded its work from AI Safety to AI Security.
So far as I can make out a descriptive, you know, a kind of branding difference without anything deeper down below. And I have my fingers crossed that the work in these two government bodies, as well as the international network, will continue in something close to the same model that existed prior to the election here in the U.S.
I’d say the thing that I’ve been concerned about from the outside now is that perhaps the blunt force approach of the cuts to probationary employees across all kinds of agencies may unintentionally disembowel the Safety Institute as well as others.
But again, to the best of my knowledge, that has not yet happened. Elizabeth Kelly, the initial director, did indeed depart. A few other people, to the best of my knowledge, also departed, but not because of deliberate cuts, rather because of their voluntary decisions, like my own, to leave in order to take up some other opportunities.
I’ll return to my ordinary position on full-time duty faculty in September and wanted to carry out some other projects, about which I’m happy to say something if you’re interested. And let me stop there and open it up to questions.
Isaac, I’ll let you run the show here unless you want me to.
Isaac: Yeah, I mean, feel free to call on people. I’ll just make sure that the mic gets to them. And yeah, and also for the people online, submit your questions. But yeah, we’ve got some time for questions. So please, yeah.
Participant: Thank you very much, Rob. I guess I want to follow up on what you just ended with. What were your reasons for leaving? Was just the kind of work not impactful? Do you think it was in a good position? Or what are you focusing on now?
Professor Rob Reich: Yeah, I did seriously entertain staying. And it was a difficult decision for me. But it came down to something as simple as the following. As I mentioned before, I was being loaned by Stanford University to the government. I was using my sabbatical to do full-time work, which I was very happy to do on behalf of the government.
But unlike the vast majority of people who have sabbatical and carry out a whole bunch of projects of their own making, I found myself wholly immersed in this other world, wholly my choice. And then with the remaining time of sabbatical between now and September, the opportunity cost of one, to some degree, the uncertainty about what would happen in the Safety Institute from whatever I could know in January, and then what other things I wanted to accomplish with my sabbatical before returning to my full-time job. I decided that for me, it was it was right to leave.
Participant: What did negotiations for the volunteer commitments look like between, I guess it wasn’t AC at that point, but do you have any sense of what they look like between like OSTP and the companies or whoever was doing it?
Professor Rob Reich: You mean the initial creation of the voluntary commitment framework? Those?
Participant: Yeah, that one mainly, and then also the ongoing ones. I was curious what the flavor is.
Professor Rob Reich: Yeah, I didn’t have any direct insight into those negotiations. But to the best of my understanding, they likely would have taken place with the White House and the companies.
Ben Buchanan was the chief AI advisor to Bruce Reed, one of the main advisors then to Biden. Ben Buchanan was at CSET before taking the role in the White House. He was one of the chief architects of the executive order as well.
And there were at least two separate announcements about these voluntary commitments to two bundles of companies that signed up for these. And although I don’t know any of the details from the internal conversations, I want to think what I’m about to say is true, not merely true back in the summer of 2023, but just as true today. It is in the self-interest of the companies to have the voluntary commitment framework. In other words, it was not a difficult negotiation to create the framework.
Why would this be the case? Because as you guys all well know, people within the companies themselves are aware of the capabilities of their then existing models and the rapid advancement of those capabilities, the possibility of the dual use or adversarial use of the models themselves.
And so, if there could be a voluntary framework in which pre-deployment access was provided and the US government conducted various tests and evaluations, the companies could say credibly, we are not grading our own homework. And we have a kind of liability, self-interest or interest to protect our potential liability should there be some AI disaster or catastrophe.
And now, in this non-coercive voluntary commitment framework, there’s at least this external body with the official imprimatur of the US government administering some tests and evaluations. And as I say, I think that incentive compatible structure remains just as true today as it did in summer of 2023.
I’ll add just in case, I think, again, you all know this, but one of the things that I often worried about was because of this incentive compatibility in the interest of the companies and having AC administer these tests and evaluations, AC could come to be seen as a safety certification agency, as if it’s putting a stamp of safety on the model itself.
And we were always at pains to say we are not a safety certification agency. We’re doing a very narrow range of tests and evaluations for very specific topics or risks, but that say, have nothing to say about the broader ordinary uses of AI and the potential liabilities they could introduce.
Participant: What do you think the relationship between the Western ACs and, for instance, Chinese labs or China AC should be like? Should Western labs, Western ACs try to get early access to DeepSeek models and that sort of thing?
Professor Rob Reich: Yup. The last question was, should Western companies try to get early access or Western governments?
Participant: Governments.
Professor Rob Reich: Yeah, this gets at what I think is in – this is among the many reasons why having people with some expertise in AI governance, the combination of some technical skill and policy or geopolitical orientation is so important.
So we reached, I think, one could reach, the AC reached an important design question about various institutions, say the creation of the International Network of Safety Institutes. One thought could be, I’m echoing now my own views here, not the official views of the Safety Institute. One thought could be, you should design an international network anchored in democracy and human rights. And you get like-minded geopolitical allies to band together and carry on a bunch of good work, which would counsel against including China in that new body.
Alternatively, you could imagine creating an international network anchored in science of AI Safety and heeding the norms of Open Science and rooting one’s practices in trying to avert or understand the most national security sensitive risks that are in the interest and principle of every country, have a body for the open sharing of the currently best empirically validated tests and evaluations of highly capable models, et cetera, et cetera. And that would suggest including any government body that wished to participate.
And you can imagine that there are good arguments in both directions, including a third possibility, which a foreign policy person might describe as deliberate strategic ambiguity. Neither one boldly stated, but some other alternative that allows in principle for kicking the can down the road and making a decision.
My own personal view is that, and this is a view rooted in my own thinking as someone who’s both taught and written a lot about democratic institutions and democratic theory, is that within democratic institutions, it’s very important to try to provide some public bodies, public institutions that are devoted to scientific practices and to some degree or another insulated from the most partisan forms of politics. Partisan forms of politics domestically and partisan forms of politics geopolitically.
So it’s a familiar thing to try to create these sorts of bodies. The Federal Reserve Board is perhaps one example, try to manage the economy. It’s tethered in obvious important ways to partisan politics, but it strives to be an independent authoritative body, expert body on other questions.
And from my point of view, imagining the emergence of new bodies, I sought to try to champion a view that there ought to be places that are unambiguously public institutions. AC was and is within the US government, but that wasn’t an instrument of partisan politics, wasn’t an instrument of geopolitical sort of dynamics.
It’s inevitable that one has to confront that. There’s no place in the world you can sit, even if you’re a private body that’s immune to geopolitics or politics generally. So I’m not trying to say you can get full insulation from it. But rather, you can promote a view that expertise should be allowed to consolidate, promulgate itself on the basis of expert knowledge, and then also then deal with the geopolitical or domestic political dynamics.
In practice, what that meant was there were opportunities to include China in track two dialogues. To the best of my knowledge, I think it’s true that Chinese AI scientists were represented at the UK summit. And it’s notable that China is not included in the international network currently. And I think it will be an interesting question how the emerging practice of the international network treats that concern going forward.
Isaac: All right, we have a couple of questions from the chat.
So Aaron asks, Framing AI safety within the context of national security has so far been more politically stable than other homes. And this is in regard to the development of professional norms and standards for computer scientists. But do you have concerns about settling on this national security framing? And how do you think this impacts the development of these professional norms? So, do you think that NIST-like standards are enough for developing these norms, or are there other things you want to see developed?
Professor Rob Reich: Good. Well, if the questioner I assume is listening could perhaps type into the question box there the alternatives to framing as national security, I’d be curious to hear what he or she has in mind.
I don’t think it’s inevitable, but I think there was strategic concern as foremost national [0:38:02] [Audio break]. Or to put it differently, I don’t necessarily point to particular interest in this route, but my observation of AI safety as a concern chiefly within a kind of an effective altruist, open philanthropy-funded orientation in 2014, 2015, 2016, far before any of this sort of broke into wider public conversation.
Of course, it was then described as X-risk, existential risk. I say this with love and affection for my EA friends in the audience, if there are any here, when accepting and promoting the aim of the company, wanting to keep power in the most charitable of descriptions, not especially politically savvy.
And, of course, that goes along with the kind of lamentable excess of confidence or hubris in the communication practices, despite the invocations of epistemic humility, an infrequency of the behavior of it in practice. You see, I have a love-hate relationship in certain ways with the EA crowd.
Okay, back to 2015, 2016, X-risk was redescribed as catastrophic risk, and clever people, perhaps it was the CivAI-related crowd, decided to get up and running a biological risk demo. And if you go to national security establishment people who may have read something about powerful AI models, but basically don’t understand much about them, and you provide them a demo, not about the effect of altruists or the extinction of humanity based upon paperclip hypotheses, but rather a demo that shows something about biological weapon capabilities and uplift, you succeed in getting their attention.
And then there becomes a bit of a path dependence from that orientation. Insofar as the AI safety community sought to become policy relevant, there appears to me have been a kind of strategy, again, I can’t point to a particular person who carried this out, but a kind of strategy to rebrand X-risk as catastrophic risk and to create various demos around national security sensitive concerns in order to introduce this to the policy community that would actually then potentially do something. And it worked.
Now, as you probably know, we’ve moved to a point in which we don’t talk so much about catastrophic risks, we talk about transformational AI that seems to have emerged into the public conversation in the past six months in order to incorporate labor market concerns and a variety of other things.
And yeah, if the questioner has alternatives, I’d be happy to react to or think through them. But I’m giving you my description of what I perceive to have happened rather than an endorsement of it. Yeah.
Isaac: So, Aaron responded, I don’t have any particular alternative that I like. NatSec seems more natural for X-risk concerns. AI ethics seems not particularly stable, but potentially broader. But NatSec seems potentially worse for things like user privacy in contrast to professional norms for other professions, like lawyers or doctors. Probably engineers are a more natural professional analogy for computer scientists are much more defense slanted.
Professor Rob Reich: Yup.
Isaac: All right. Any other questions in the room?
Participant: How awake, concerned, aware of transformative AI and the associated risks do you think the government is? And how do you expect this to change in the coming few years?
Professor Rob Reich: Yeah. What part of the government?
Participant: Yeah, I guess I’m probably most excited about the executive branch, but also, I don’t know, Congress as well, pretty interesting.
Professor Rob Reich: I think at this point in 2025, pretty aware. And if you listen to the Ben Buchanan interview with Ezra Klein four or five days ago… You did?
Participant: I did.
Professor Rob Reich: Yeah. And he was the guy in the White House. And I feel like you should invite him out here to give a version of a talk. He’d convey something, I think, of the same view that I think you’d be surprised if all of the staff of the different senators and Congress people were as aware.
But I think in the executive branch of the White House, lots of awareness. And of course, now in the new Trump administration, you’ve got a whole bunch of tech folks there as well. It remains, from my point of view, still an open question about exactly what type of policy regime they wish to promulgate. But I mean, I’m saying the obvious here again to this crowd, but Musk was in favor of the California AI safety bill. Suryam Krishnan is no dilettante in AI. David Sachs, the same thing.
So, I do think that there’s a pretty widely understood set of concerns about transformational AI. The open question is how to manage the complicated politics internal to the current Republican Party, but then of course more broadly with Congress. And if you have thoughts about that, I’m happy to react to it. But, yeah.
Participant: I guess, it seems fairly likely that stuff happens very quickly in the next five years. And it seems to me that the government is just radically unprepared for this. How would you expect them to–
Professor Rob Reich: Just the government? I mean, society is radically [0:44:12] [Crosstalk]
Participant: And society. Yeah. I guess the government seems like maybe the most, one of the things that is maybe plausibly capable of reacting like kind of quickly via the executive branch.
Professor Rob Reich: Yes. Okay. I mean, I agree. And I’ll both say two responses to this.
Number one, if you come at this type of question, not as an AI person, a technologist, but as a political scientist looking at the history of emergent science and technology and how government tends to respond to things that have either negative externalities or potentially broad society transformative consequences, it’s a feature, not a bug of democracy that it lags behind the frontier.
It’s desirable that it do so rather than try to promulgate regulations before technology has matured and disseminated at scale. And we have reached something like scale dissemination now with AI, although the capability frontier is, of course, changing rapidly all the time. And someone who looks at the history of this, I think, should acknowledge that policy change happens in at least two simplistic models.
One is incremental small bore approach, which is the trajectory we’ve been on. I would say something like you get the GDPR, the EU AI Act, the ACs, UK, and it’s like the opening of a decade-long policy window and we’re happened to be in year two or three. So, I’m expressing cautious optimism that it continues.
The second type of approach is a catastrophe arrives and the policy community has to be seen to be doing something. 9-11 happens and you get the Patriot Act. The Tuskegee experiments on human beings happens and you have to get a response.
As I always tell people, it’s not a logical necessity that we have a catastrophe to stimulate policy change, but it does seem to be an historical pattern. So it’s perverse to wish for a catastrophe, but nevertheless, we await one.
And from my point of view, among the most important things an AI safety community can do is not merely to work on incremental policy change, but to have off the shelf ready dramatic policy and regulatory structures ready in the immediate wake of a variety of different catastrophes. And many fewer people in my judgment are working on that latter project, many more on the former.
Isaac: Cool. I saw this hand first.
Participant: Yeah, I have a follow-up question from that as well. You mentioned that a lot of policymakers are potentially aware of the transformative AI, but I’m also curious about what are the reactions to it? Is it more like, oh, it’s just some technology which can be useful for making profits and solving, and curing diseases? Or is it more like they recognize the duality of the technology and the threats that come along with it as well?
Professor Rob Reich: Yes, to both of the things you described, but in addition, other things as well. I think the awareness includes things like, if you’re in the United States, the U.S. possesses an innovation advantage. Open questions about the gap between the U.S. and China. But generally speaking, how to preserve that innovation advantage because of the connection between the technological frontier and geopolitical power.
And then the one form of accelerationism that unambiguously is going to take place in this administration is the application of AI to the military. And I don’t think I need to explain much about why, from a national security perspective, that’s of interest to people in the U.S. government.
So risks and concerns, but also potential benefits, all the stuff about applications to medicine, education, science in general, and then U.S. national security protection, as well as offensive advantage through military applications and other things. And yeah, this is, I think, pretty widely understood, for sure, in the executive branch.
Isaac: I’m going to ask one question from the chat real quick. Sawyer asks, For people not currently in positions of power but who want to work on AI governance, what areas of the field do you think most need independent work right now? I’m thinking of researchers and just general doers looking to make a positive difference.
Professor Rob Reich: Yeah. Great. This is a question which I think orients us productively on not the law, policy, and regulation side alone, but the professional norms and standards from within the field. And so I welcome that.
Let me give two responses. Number one, and I think I’m going to be singing to the choir here, but I want to share this just because I think about your wider peer group, maybe the kinds of folks you study with in undergrad in CS or AI, and how you communicate with them if indeed you do.
But we had a moment in the U.S. AI Safety Institute in the summer when we were onboarding a bunch of new people, and we had a little team retreat, and the following exchange happened. Someone said to the whole group, again, like 20, 25 people, not an especially large crowd, “I just want to say that we’re a really great group here in the commerce. We’re doing such important work, and the team is terrific. And I realize that you could have easily have gone into industry and made much more money. And so I just want to thank you for signing up and for joining this important work.”
And I raised my hand, and I said, “I would like to rephrase. We are working in the distinctively authoritative and legitimate institutions of democratic government in the most powerful country in the world, on the most consequential technology of the 21st century. Like what, you preferred to go work for a shitty startup? Like, put your goddamn big boy pants on. Let’s go!”
And you guys have all made choices similar to this, so this is not directed at you. But imagine directing it at the 90% of your classmates in undergraduate, you know, your undergraduate years, and what are they thinking is a version of the thought.
Okay. That’s the singing to the choir. Now, a second response to the question. I’ve come to the view, well, if I ask myself the question, what are the levers of intervention within the professionalization of AIML developers and people, like, starting with undergraduate education but then radiating outward wherever people happen to find work?
In my judgment, one of the most consequential opportunities to have some influential effect is by trying to change the internal understandings and incentives and practices of the professional societies where reputations are established. So, NeurIPS, ICML, the other, you know, kind of professional bodies like that.
And, you know, repeating things that I’m guessing you guys know, if you, like, look at the 2024, 2023 papers at NeurIPS, and you do an abstract search that includes safety or security, you come up with less than 10% of the papers and posters, and the overwhelming majority have something to do with the capability frontier.
That’s, I want to describe in economic terms as, like, a terrible allocation of human talent. It reflects the disciplinary immaturity of the profession to be as disproportionately interested in pushing the capability frontier forward without commensurate time and talent and investment in safety and security.
So you could try to do, like, Schmidt Science’s open philanthropy style things and have prizes and various ways of funding, you know, the Berkeley constellation of outfits in which you’re all engaged, and those are good things to do. But where is the central – what are the central bodies reputations and status are assigned, the reputational economies? It’s the professional societies or among the various places of the professional societies.
And so I think having an opportunity to try to change how those are practiced or operated is a valuable point of potential intervention. You might say, what do I have in mind? Well, try to get elected to the board of NeurIPS or ICML and get involved with the governance of those bodies and see what type of change is possible within it.
You’re not going to get the Department of Commerce to tell any of these bodies what to do, and nor should it. It would be a mistake to have NeurIPS be responsive to a government body. But that’s where professional norms and standards are generated and observed and upheld, and so that’s where someone deep in the technical weeds of this can make a big difference.
Participant: As someone coming from a more economic background, I see what you’re describing at both a local and international level as fundamentally a coordination problem.
In terms of development of standards in a very, very, probably unprecedentedly high-velocity field, you’re trying to come to some, ideally, to use economic language, Pareto-efficient Nash equilibrium when you potentially have everyone working at odds with each other. Because to take a perhaps crude example, there is no professional standards body for some random person to build a unicorn startup in AI, developing a defense-adjacent capability that has no regard for safety whatsoever.
And then at an international level, if, say, you exclude the [0:54:46] [Audio break] of the Chinese biologist who, I think, used CRISPR to edit two twins to make them resistant to HIV. And then when the rest of the world was appalled, he had no idea why, because the professional standards had evolved, at least in his context, so differently.
How do you prevent the development of two entirely separate worlds of professional standards when you focus on something like only including democratic societies, if the risk of that is to then break down the possibilities of coordination so much that you end up with something analogous to an AI arms race at a much greater scale than you otherwise might have expected?
Professor Rob Reich: Good. This gives me a chance to address again the question you asked about China and Chinese scientists being involved and present at various forms of AI safety dialogue and practice. I want to run with your example of this Chinese biomedical researcher, because I think this is an instructive parallel for the professional immaturity of AIML as compared to biomedical research. And so everything you just described is right about the worry of these two worlds emerging.
However, this particular Chinese biomedical researcher had done postdoctoral training at Stanford University, and there was an inquiry after he used CRISPR shortly after his postdoc that eventually found no fault or coordination planning from the lab at Stanford.
And tellingly to me, he was an active contributor to not merely Chinese biomedical science, but global biomedical science in this frontier area of CRISPR, and the story I often tell about that is, I think, illuminating for AIML as a very sharp contrast. So you tell me if you have a different version of this.
But the story that I often tell is Jennifer Doudna, the local Nobel Prize winner for her role in discovering CRISPR, has this discovery. And a few months later, she writes in her autobiography that she thought it was all upside. And I imagine all the agricultural changes, the kind of potential novel approaches to repair of various genetic anomalies within humans. And she has a nightmare in which Hitler comes to her and says, “I’m very interested in your discovery, professor.”
And in the morning, I think she says, she calls the other leading biomedical researchers in this particular frontier area, and they have an emergency meeting. And they publish an article six months later that calls for a voluntary ban on the use of CRISPR on human embryos or human beings, until we can have a broader social discussion and understanding about the guardrails that need to be put in place so that we can harvest the extraordinary benefits while steering away from these obvious potential harms as well.
And when this Chinese biomedical researcher did announce this use of CRISPR, what happened? He was excommunicated from the practice of science. No conference would allow him to speak. No journal would publish his papers. And eventually, the Chinese government imprisoned him, although for separate reasons.
I often say, can you think of an AIML developer who didn’t break the law but violated a norm widely understood within their respective community and suffered a professional cost for it? I’m waiting for an answer to that question. That’s an example of the immaturity of the professional standards.
Now, the question you asked is not just about the immaturity but about the possibility of two rival systems of norms and standards evolving. And I don’t have anything particularly novel or interesting to say to that other than if our geopolitics makes much less likely venues in which AI safety can be discussed as a scientific practice on a global basis, which would allow for the emergence of widely shared norms and standards, then we have to do everything we can outside of government to ensure that there are such places for those dialogues to happen.
And we have to try to insist upon the scientific basis of this inquiry with the norms of Open Science underpinning it, rather than announcing ourselves as first and foremost interested in geopolitics or policy.
Again, that’s why my preferred phrase of AI governance, I think, is relevant here. Because governance allows us to talk about scientific practices and standards and norms, not just partisan or geopolitical politics.
Isaac: Okay, that brings us exactly to time, unfortunately. I know there’s more questions, sorry, we couldn’t get to those. But everyone join me in thanking Professor Rob Reich.
Professor Rob Reich: Thanks. I’ll hang out here for a bit if anybody wants to keep talking, but no expectation, of course, that you do.
