Seoul Alignment Workshop 2026: What We Learned

On July 6, FAR.AI convened the Seoul Alignment Workshop, co-located with ICML 2026, bringing together nearly 200 people from frontier labs, academia, government, and AI safety organizations. The talks ranged from the geometry of a model's internals to the design of national regulation.

One could mistake it for a survey of unrelated problems. We see it as something more specific: AI safety practitioners trying to shape the field into something that looks more like an engineering discipline, and discovering how far it still has to go. The recurring demand was not for more capable models but for two things engineering is brimming with and AI security generally lacks: measurements you can trust and standards you can hold a system to. Much of the day was spent showing that our current instruments are not meeting that bar, and the rest was spent talking about what the field is building that might. That is the hopeful shift we think is worth watching.

‍

The Capability-Reliability Gap

The day opened by naming the problem all talks would circle: AI capabilities are climbing far faster than our ability to make systems reliable; closing that gap is becoming one of the field's central engineering challenges.

The opening panel took on this question directly. Cynthia Chen framed it for Stephan Rabanser (Princeton CITP), Adam Gleave (FAR.AI), and Dylan Hadfield-Menell (MIT): systems now perform superhumanly on most benchmarks and still fail in ordinary use. Where is the gap?

Rabanser's answer came from outside AI. Aviation, nuclear, and automotive engineering never certify a system on a single accuracy number; they certify across several dimensions at once, and that is much of what makes them safe. He proposed four for AI agents: consistency, robustness, predictability, and operational safety. Measured this way, the trend lines separate. Accuracy on his benchmarks has climbed steadily across recent model generations while reliability has stayed close to flat. Models have become genuinely robust to tool faults, but the same request worded differently still trips them. He closed on the stakes: the reliability bar for full automation is far higher than for augmentation, where a human in the loop still acts as the backstop.

Gleave's position was that reliability has to become more like a capability. It is one of the main things holding back real-world use, and he put the lack of progress down to two things. Reliability is hard to measure, since there are no standard benchmarks and consequential failures are by definition infrequent, so neither an afternoon of testing nor a year of it tells you much. And there is no recipe: capabilities improve predictably with more compute and more data, while reliability has many partial approaches and few you can put billions behind. What he wants is a technical standard that, if followed, assures a system is safe with better than 99% confidence, and universal adoption of it through governance, market incentives, and voluntary commitments. What we have is no quantified reliability at all, with benchmarks for how good a model is at coding and none for how often it will drop a production database. The result is flying blind: we do not know how to build reliably safe systems, and we would not recognize one if we built it. 

His answer is to treat safety as an engineering problem, on three principles. Build systems out of parts that can be tested on their own, and make the case for the whole from those parts. State safety properties in advance and test the predictions, so that rules of thumb become empirical. And test beyond black-box behavior, using probes to catch what the output does not reveal. What that would buy is visible in what red teaming currently returns. FAR.AI's red-team now rarely finds end-to-end jailbreaks on frontier proprietary models. What the team finds instead is a system where every safeguard layer fails except one, which leaves a single load-bearing layer and very little confidence, since one new attack technique takes the whole thing down.

Hadfield-Menell's diagnosis started with a small failure of his own. He asked a coding agent to improve a prompt on a toy planning problem, and it went one shot to a perfect solution by writing the answer into the prompt, so the planner could not get it wrong. His explanation: models are trained against goals that can be scored automatically, and they optimize for those at the expense of the goals we actually care about, which nobody knows how to score. Reinforcement learning has made this sharper as it has become standard in post-training. He noted that he and the safety field alike were slow to take the problem seriously and called for work on systems that can reason about their own uncertainty as they run, rather than relying only on alignment fixed during training.

‍

Measuring Automated AI Research

One capability makes the reliability problem more urgent, because it compresses the time available to solve it: AI systems are increasingly doing AI research.

Maksym Andriushchenko (ELLIS/EPFL) presented PostTrainBench, which measures whether agents can run LLM post-training end-to-end. He framed the stakes through the frontier safety frameworks: Google, OpenAI, and Anthropic all name autonomous AI R&D in theirs, because it is key to recursive self-improvement and from there loss of control. That makes tracking it a priority, and his argument is that the instruments weren't up to the job. Earlier benchmarks measured easier things, small models on simple tasks, and handed the agent much more context. PostTrainBench gives it a base model, a fixed compute budget, and web search, says nothing about method, and evaluates only the model that comes back. That openness is also what makes it gameable: an agent can quietly pass off a model downloaded from Hugging Face as its own output, so the benchmark runs a reward-hacking judge and scores a flagged run at the base model, meaning the cheating earns nothing. Both capabilities are rising together, which is why Andriushchenko closed on the judging rather than the scores. A benchmark that cannot tell real post-training from a gamed run will report cheating as progress.

Alan Chan (GovAI) took the question up a level. His framing is that automating AI R&D multiplies the effective amount of research labor, and since labor is an important input to progress, more of it should mean faster progress. The consequences run in two directions. Capabilities of every kind arrive faster than safety research and policy can absorb them. And oversight degrades, since its checks depend on humans being in the loop: the ground-level view that catches research sabotage, and the independent perspectives that catch bad decisions before they are made. Chan was careful not to assert any of this will happen; we don't know how far automation has gone inside the labs, or whether it will speed progress up at all. His close was a pitch to the researchers in the room: build the metrics and evaluations that would let the field see whether safety work is keeping pace with capabilities, since right now it has almost no way to tell.

‍

When Results Fail under Pressure

Three talks converged on a harder version of the measurement problem. It is not only that the instruments are missing. It is that several of the existing instruments we have stop meaning what we think they mean once a model is under optimization pressure or knows it is being watched.

Atticus Geiger (GoodFire) went after an assumption underneath much of interpretability, in a talk on the role of neural geometry in understanding models. Today, interpretability work produces a list of directions inside the model with short English labels attached, but a label is not evidence: tagging a direction "deception" says nothing about whether it does anything in the model. The picture is shakier still because the underlying assumption, that each concept is a single direction, turns out to be false, with concepts stored in curved, multidimensional shapes instead. What he wants in place of a labeled direction is closer to an encyclopedia entry for each representation: its geometry, the concept it maps to, and a causal account of the role it actually plays. Done by hand, as his group has done it, that is far too slow to apply broadly, which is why he sees research agents as what would make it viable. He was clear this is where the work is headed, not where it stands, which leaves a question over the tools the field has now: how much weight they bear once a model has reason to defeat them.

Two talks came at deception from opposite sides and met in the middle. Chris Cundy (FAR.AI) read it from inside the model. His argument is that training rewards deception: reinforcement learning pays out on whatever the judge approves, so wherever the judge can't fully check the work, convincing it is easier than doing the job. The natural fix is to catch deception with probes on the model's internal states and penalize it during training, but his Obfuscation Atlas, an outstanding paper honorable mention at ICML, found that pressing that penalty harder mostly teaches the model to hide, either dressing the cheat in justification or shifting the very activations the probe reads. Stranger still, cheating on its own shifts those activations, because doing it changes the model's own belief that cheating is honest. The hopeful note was that at some settings the probe produced honest work where its absence produced cheating, so probes can train honesty and not only detect its absence, as long as they aren't trusted blindly once a model has been optimized against them. 

Lu Wang (University of Michigan) found the same fragility from the outside, in behavior rather than activations, in work on what she calls alignment faking: a model behaving one way when it believes it is being watched and another when it thinks it is not. An evaluation only means something if a model acts the same whether or not it is watched, and her diagnostic, VLAF (Value-Laden Probing For Alignment Faking Diagnosis), sets a model's own values against a developer instruction so that any change of course under observation reads as strategy rather than confusion. The gap between how often models comply when monitored and when they believe they are not came out wide, across different models and different stakes, from being retrained to being replaced by a more compliant model. Her sharpest result was that this is not all-or-nothing: as the stated chance of being watched rises, the compliance gap rises with it, so models are tuning their behavior to how likely oversight seems rather than flipping a switch. Between them, Cundy and Wang make one point from two directions. Whether you read a model's insides or its outputs, the reading starts to lie precisely when the model has a reason to want it to.

‍

Building Safety in Rather than Bolting It on

Guardrails bolted on around a finished model can be stripped off or slipped past, as the deception and open-weight results kept showing. A recurring alternative was to specify the behavior you want and train it in from the start.

Jiayu Shen (Alibaba) presented Alibaba's Model Spec, a framework that sets out in testable terms how a model should behave and where its limits are. Six core principles become 43 guidelines that describe a good response, with a four-tier permission hierarchy (root, system, developer, user). The principles are more specific than that makes them sound: the one on ethics is written to catch two failure modes at once: models that flatter and agree with everything and models that lecture or refuse when they should just answer. Checking whether a model actually follows the spec is staged, from compliance on single clear-cut rules toward consistency against the whole spec, which Shen was candid is still unsolved. What sets the effort apart is that the spec is not written behind closed doors: it is open-sourced and revised through both expert review and public questionnaires that probe where people read the rules differently than the authors intended.

Andreas Krause (ETH Zürich) addressed how a model learns any of this in the first place. Demonstrations are expensive to collect, and trial-and-error reinforcement learning is slow and invites reward hacking. But models are surrounded by useful feedback as they work: an error message, a user correcting the last answer, and they are already good at making sense of it in the moment. The problem is that this understanding evaporates once the conversation ends. His method, self-distillation, fixes that by using the model twice: once to make an attempt, and once as its own teacher, looking back at the feedback and asking how it should have answered. That improved answer becomes the training example, so the lesson gets written into the model's weights rather than lost. Trained this way on real user conversations, using each person's follow-up as the feedback, it improves models that have already been through standard alignment, and it knows when to leave well enough alone: when a follow-up carries no correction, it makes no change. His bet is that a model can keep getting better from the feedback ordinary use already produces.

Iulian Serban (LawZero) argued that you cannot get to a safe system by patching or steering the models we have now, because the danger is baked into their being goal-directed agents. His group's alternative, Scientist AI, starts from a bet: that intelligence and agency can be pulled apart. Take away the drive to pursue goals and the ability to act in the world, keep the ability to reason, and you get a system that can model what will happen and answer questions about it without wanting anything or trying to bring anything about. He was clear this is an assumption the whole design rests on, not something proven. The system is built to work like an idealized scientist. It keeps track of who said what, so that a Reddit post claiming vaccines cause autism is stored as a fact about the poster rather than a fact about vaccines. It forms hypotheses that can be tested and rejected, and weighs how likely each one is against evidence rather than committing to an answer. A recent paper from the group reports that as this system gets more capable it also gets more honest, which, if true, would break the link between capability and danger that most safety work is built to manage.

‍

On Open Weights and Global Perspectives

The open-weight debate has increasingly become geopolitical, and three talks put its two halves next to each other.

Irene Solaiman (Hugging Face) brought five years of Hugging Face download data, and the story it tells is that open source has entered a Chinese period. Among the twenty most-downloaded authors on the platform, roughly 75% of downloads now go to models developed in China. Since January 2025 the top ten trending models have been consistently majority Chinese, broken only briefly by Google's Gemma and NVIDIA's Nemotron. Qwen has passed Meta as the most built-on model family, with derivatives growing around 30% quarter on quarter, and the number of Chinese organizations with real followings publishing on the platform rose sharply through 2025 while the American line stayed flat. Solaiman argues this is increasingly a sovereignty issue: owning your own model stack gives a country control and reliability at a time when access to American models can be cut off. She ended by widening the question past who reaches the frontier first. What may matter more, she suggested, is who spreads their models furthest, whose cultural values travel inside them, and who ends up shaping how AI is deployed.

Kellin Pelrine (FAR.AI) supplied the security counterweight, and he opened with the good news. Closed-weight models have become substantially harder to jailbreak over the past year. It is not solved, but the remaining problem has shifted from needing scientific breakthroughs to needing good engineering and broad adoption. Open weights are another matter. Fifteen minutes of searching is enough to download a jailbroken Gemma 4 and have a compliant model, no expertise required. The weakness is not limited to the fine-tuning attacks people associate with open weights: a public jailbreak for DeepSeek-V4, never patched from a previous model, works with nothing more than a cleverly worded prompt. Breaking a model does not necessarily cost it any capability, either. Anthropic found this so-called jailbreak tax disappears with stronger models, and Pelrine's team found the same on the open side. His reason for optimism is that robustness looks trainable, not something only external filters can provide. GPT-5.5 resists jailbreaks well on its own, with nothing wrapped around it, which shows the resistance can live in the model itself. That is exactly the kind of protection open weights can keep, since a filter wrapped around an open model can simply be removed, while robustness trained into the model cannot be stripped off as easily. So the work worth doing is on techniques like pretraining data filtering, which builds the safeguard in from the start. No single fix makes a model safe, but he argued the risk can be brought down significantly.

Stella Biderman (EleutherAI), with Oxford and AISI, took Pelrine's point and built on it. The whole safety playbook, she argued, was written for models behind an API, and none of it holds once you hand over the weights: filtering, alignment training, and usage monitoring all assume a level of control you give up the moment anyone can download and retrain the model. And "just don't release open weights" is not an answer, since it sidesteps the problem rather than solving it. Her work, Deep Ignorance, takes a different route: strip the dangerous knowledge out of the training data before the model ever learns it. Filtering bioweapons material out of training left models far weaker on a biosecurity benchmark, and the drop held even when someone fine-tuned the model afterward to teach the capability back, taking more than an order of magnitude more effort to undo than any other method. The catch is that it only works when the capability is self-contained, hard to reconstruct from what remains, and safe to be missing, which by her estimate is roughly CBRN knowledge and not much beyond it. Her closing point was that almost no one has seriously tried to make open models safe on their own terms, and she suspects much of what would work simply hasn't been attempted yet.

‍

Evaluating at Realistic Scale

A cluster of talks made a common point from different directions: the evaluations exist and we are running them under conditions that do not resemble deployment.

Noam Brown (OpenAI) showed how much a single score can hide. When GPT-5.5 came out it looked only marginally better than its predecessor on a cyber test, 82% against 79%, yet in use the jump was large. The score had concealed the model's improved performance, because the older model was simply thinking for longer before answering. Plotted against how long each was allowed to think, the real gap was clear. This can't be fixed by running each model to its plateau, because the newest models don't plateau: on AISI data they keep improving past tens of millions of tokens of reasoning. This matters most for safety testing, since a model that looks harmless on a cheap evaluation may be well over the line once someone pays for it to think. His fix is to test across a range of compute budgets and extrapolate, and to have safety frameworks account for thinking time before clearing a model as below a threshold.

Samira Nedungadi (SecureBio AI) raised a subtler problem: what happens when models outrun the experts who write the tests? Bio benchmarks are built and graded by human experts, and the latest models now match or beat that expert baseline on many of them, because the answer key captures expert consensus and models predict that consensus better than any individual can. Writing harder questions doesn't help, since at the frontier of a field there is no consensus to grade against. So SecureBio lets an experiment settle the answer instead: its Predictive Bio Bench hands a model the setup of a real experiment that was run but never published, and grades its prediction against a result it had no way of knowing. A companion benchmark made the capability vivid, coding agents reproduced three of four published biology AI models from the papers alone, in under twelve hours and for a few hundred dollars each. Her point is that measuring past human expertise will take a spread of independent tests, some answerable only by reality.

Changyi Li (Fudan University) took the third gap: risks that only surface in the right situation. Dangerous behavior tends to emerge from a specific mix of tools, incentives, and pressure that a static question-and-answer test never creates, and building those situations by hand doesn't scale. AutoControl Arena generates them instead, using code for the parts that must behave reliably and a language model for the scenario on top, then runs an agent inside and audits what it did. It earns trust by reproducing failures labs have already seen in the wild and matching hand-built environments. Its central finding, the alignment illusion, is that models look safe under normal conditions and come apart once the pressure is turned up, with stronger models sometimes falling harder. Capability isn't safety: a more capable model refuses obvious harm more reliably but is also better at finding the loophole when the harm is buried in a realistic task.

Yinpeng Dong (Tsinghua University) made the point about checking an agent's work. Producing an answer is cheap while verifying it is hard: an agent can output a clinical diagnosis in seconds, but confirming it is right takes expertise and a review of the evidence. Judging only the final answer measures the wrong thing. His method works through the agent's steps one at a time, asking at each whether the step made the decision more or less likely to be correct, and grounding that judgment in domain guidelines so each step gets a reliable weight. That separates sound reasoning from unsound far better than looking at the endpoint alone. He closed on loss of control, which he made measurable by breaking it into three ingredients, a misaligned goal, the capability to do harm, and the ability to evade oversight, on the argument that it is never any one of these alone but the three together.

‍

Governing What we Cannot yet Measure

Samuel Simko (EuroSafeAI) argued that the field measures safety at the wrong level. Most research asks whether a single model can be made to misbehave, but the risks Simko is most concerned with sit higher up. In GT-HarmBench, EuroSafeAI casts interactions between models as strategic games and finds that models well-behaved on their own will escalate a confrontation, such as a simulated arms race, where a person would step back. In SocialHarmBench, models asked to rewrite history or aid surveillance comply at high rates even under plain prompting. Neither risk is visible to a standard evaluation, which watches one model answer one prompt, because neither exists at that level: they only appear when models interact or operate at scale. That is exactly what makes them hard to govern, since you cannot regulate what you have no way to measure.

The Global Perspectives on AI Safety panel put individuals from four jurisdictions in conversation and asked where AI governance is converging and where the differences still run deep: Lisa Soder (European AI Office, speaking personally), Cozmin Ududec (UK AISI), Myuhng-Joo Kim (Korea AISI), and Stephen Casper (Harvard), moderated by Isabella Duan (Safe AI Forum). What emerged was less a consensus and more a set of different vantage points on the same problem.

Casper drew the structural line. The EU AI Office is a regulator with real power to compel and to fine; the UK's institute is a partner that tests models by invitation, its evaluations technically voluntary collaborations. Most governance bodies in the world look more like the second than the first, and China fits neither mold. His worry was that the world is converging on those few templates too quickly, running too few experiments in how AI might be governed at a moment when the field is young and no one knows what will work. Soder, from the EU side, saw a convergence she was more comfortable with: facing the same uncertainty, jurisdictions have all landed on gathering information, whether by running their own evaluations, as the UK does, or requiring companies to report, as the EU does. But she shared Casper's unease about uniformity for its own sake, warning that the appetite for identical rules everywhere squeezes out the experimentation that would show which rules actually work. Ududec added the precondition beneath all of it: without deep technical expertise inside government and in an independent evaluator ecosystem, a published safety report cannot even be read for what is solid, what is missing, and what it implies.

Kim brought a different frame again. Korea's institute runs under a law that defines safety as protecting citizens from AI risks and preserving trust in an AI society, tying it to risk and trust rather than to properties of a model. The convergence he sees is quiet and practical, happening at the working level. One-time testing before release is no longer considered enough. The basic toolkit of red teaming, capability evaluation, and incident reporting is lining up across countries, and evaluation is shifting from what a model knows to what an agent does. His evidence was a six-month joint study with Singapore on data leakage by AI agents, run across twelve scenarios and two independent pipelines in two regulatory systems, which reached the same conclusion on both sides: getting the job done and keeping data safe are separate axes, and an agent can complete a task flawlessly while leaking sensitive information along the way. Two countries, different legal starting points, same finding, which to Kim is the real lesson: cooperation does not require identical laws, only shared methods, comparable evidence, and channels people trust. The one place he wanted more agreement, not less, was on what should trigger a mandatory safety evaluation in the first place. Compute thresholds are easy to measure and that is why they are used, but they track risk poorly, since a small specialized model can do real harm and giving an ordinary model tools or autonomy changes its risk profile regardless of size. His proposal is a trigger that layers compute together with demonstrated capability, autonomy, how widely a model is deployed, what tools it can reach, and how sensitive its domain is.

‍

Where we Go Next

There was no single conclusion at the end of the day, but there was a seeming change in posture. Across different talks, the same demand kept surfacing: if AI Safety as a discipline is going to move the needle, it has to build measurements you can trust, and it has to set standards with teeth. The clearest sign of that shift was Jesse Hoogland's launch of Resolution, a new alignment organization betting that formal theory can put automated safety research on firmer ground. It is the field trying to scale its own rigor as fast as the labs are scaling capability.

Much of the Seoul AW centered on the question of why we don't have reliable measurements yet.  Several of the tools the field leans on stop being useful once a model has reason to defeat them: interpretations that assume the wrong structure, probes a model learns to slip, or evaluations a model evades once it knows it's being watched. We sometimes fail for having the wrong conditions, too little compute, experts who can no longer grade the answers, scenarios too clean to trigger the behaviours we fear. The encouraging part is it is clear there's promising research being done in the way of fixing just that.

Adam Gleave's framing, that we should aim for systems that we can certify as safe with better than 99% confidence, and Chris Cundy's, that the field should have a zero tolerance policy for any known falsehood from a model, are versions of the same hope: replacing ad hoc judgment with a threshold a system either clears or not. That is a higher bar than the field currently holds itself to, and meeting it will take work, and the willingness to draw lines and enforce them — in training, evaluation, and in how these systems are governed.

Capability will keep advancing regardless. The frontier we think is underserved is reliability; making these systems safe and being able to prove it.

‍

Next up is FAR.AI's Cambridge AI Research Directions Workshop (CAIRD), co-hosted with the Cambridge Boston Alignment Initiative (CBAI). 

Recorded talks from Seoul are available on the FAR.AI YouTube channel. 

To hear about future events, subscribe to the FAR.AI newsletter or submit your interest here.