AViD Workshop 2026

In early 2026, Anthropic chose not to release its latest Mythos model to the public due to concerns about its potential to help malicious actors with cyberattacks, reporting that Mythos had “found thousands of high-severity vulnerabilities, including some in every major operating system and web browser.” Decisions around whether AI models are released and who has access to them are becoming increasingly high-stakes. As AI systems become more capable and strategically important, there is growing interest in mechanisms that could allow third parties to obtain trustworthy evidence about how models are trained, evaluated, and deployed — without requiring unrestricted access to sensitive assets such as model weights or proprietary infrastructure. This challenge arises in contexts ranging from confidential third-party evaluations and enterprise deployment assurance to longer-term questions around governance and international coordination. On May 17, FAR.AI and the Center for AI Safety co-hosted the Assurance & Verification of AI Development (AVID) Workshop, colocated with IEEE S&P, which brought together over 100 researchers and engineers to discuss what technical foundations can enable us to generate evidence about properties of AI systems from training through deployment. 

Independent auditing and oversight: An ecosystem for independent evaluation and auditing is beginning to emerge, including collaborations between frontier AI developers, government agencies such as the UK AI Security Institute, and external evaluators including METR, Apollo, and SecureBio. However, external evaluators often operate under significant resource constraints and with limited visibility into model development processes. Better tooling and infrastructure for assurance could allow greater levels of access to models and sensitive information and more automated, faster analysis to identify potential risks. In his AVID workshop talk, Koen van der Veen (OpenMined) discussed technical approaches to enable confidential and verified auditing - a set-up where AI developers do not need to share their models with auditors, and auditors do not need to share private evaluations that might include sensitive (or even classified) data with developers. This is one example of how technical advances can allow deeper forms of cooperation between AI developers and third parties. 

Demand for cooperation-enabling technologies: OpenAI has previously highlighted that it could be beneficial to coordinate a slowdown at “critical junctures” to provide time to “solve technical alignment problems” or “give society enough time to adapt,” a step that would likely require cooperation with AI developers across several countries. Leaders at Anthropic and Google DeepMind have publicly said that it would be preferable to slow down AI development collectively to better manage AI risks. However, competitive pressures make it unattractive for AI developers or countries to slow down unilaterally while allowing others to race ahead unconstrained. In scenarios such as those described by OpenAI, there could be strong demand from AI developers and governments for technical infrastructure that allows them to provide credible evidence to each other that relevant high-risk AI development activities have been limited.

Verifiable inference as a testing ground: What the challenges described above share is that in both cases, there is a need to enable targeted transparency that meets the needs of governments or auditors, while protecting sensitive information such as model weights that must be kept secret for commercial or national security reasons. One specific application that received considerable attention during the workshop is proving which version of a model is running. This appears to be both technically tractable with current methods and commercially useful. It provides a foundation for transparency: for example, it would enable an auditor or user to understand whether the model that is running is the same one that was evaluated by third parties, or a modified version for which the evaluation results are no longer valid. Verifiable inference has the additional benefit of being incentivized for AI developers that would like to ensure the integrity of their models against any insider threats or tampering during deployment. For example, Anthropic has publicly committed to developing a “prototype of provable inference, a technique for reliably, provably “signing” AI model outputs in a way that makes them attributable to a specific set of model weights.”

Confidential computing: One approach takes advantage of the existing confidential computing features now available in modern CPUs and some AI accelerators that allow remote attestation of specific software measurements or execution environments. Practically, this means that it is possible to get proof that the code running on AI chips has not been tampered with, that the expected model has been loaded, and that these are running in a secure encrypted environment. However, this requires trust in the hardware designer and manufacturer and their management of the keys used to encrypt data. Confidential computing implementations typically provide limited protection against sophisticated physical attacks aimed at extracting secrets or bypassing protections, so there is also a need for defenses such as enclosures that can prevent (or at least detect and record) such attacks. Additionally, current confidential computing implementations are typically not suitable for practical large-scale AI training or inference of the most powerful frontier models with weights that consume multiple TB of memory. In his talk at AVID, Amean Asad from Confidential AI discussed this challenge and potential near-term approaches to overcome it. 

Zero-knowledge proofs: An alternative approach takes advantage of zero-knowledge proofs which make it possible to generate cryptographic proofs corresponding to specific inference computations. The main problem is that the computation required for these proofs has historically been many orders of magnitude slower than the inference itself. In his AVID talk, Bing-Jyue Chen(Kang lab, UIUC) discussed how work by their group and others has been closing this gap in the last 2 years, making real-time proofs of inference for frontier models an increasingly plausible prospect in coming years. Research groups at the University of Birmingham and General-Purpose AI Policy Lab also discussed extending zero-knowledge proofs to prove other properties of AI models, such as the amount of computation used during training.

Recomputation: Some researchers have tackled the challenge of provable inference via independently recomputing the outputs that a model produces when fed certain inputs, and comparing these against the original outputs to identify any discrepancies. A key question here is how to ensure that models predictably output the same tokens when given the same inputs, as there are a number of aspects of AI inference that can lead to non-deterministic outputs that cannot be reproduced exactly (for example, expert routing in mixture of expert architectures, or the fact that adding the same numbers in a different order on GPUs can lead to different results). At the workshop, Roy Rinberg (Harvard University) presented recent work to build a prototype that overcomes non-determinism challenges. In a break-out session, Anjay Friedman (RAND) also discussed how to securely and comprehensively log model inputs and outputs for later sampling and recomputation.

Analog sensors: Such approaches can be complemented by verification approaches that use analog sensors that record power consumption or other physical signatures of AI inference and training. These provide a more noisy and coarse-grained picture of activity at an AI data center. That coarseness can be an advantage in scenarios  such as international cooperation where concerns around leaking commercially valuable information would be most severe. To defend against attacks from highly-resourced, nation-state level adversaries, approaches that rely primarily on internally generated attestations from trusted hardware with an adversary-controlled data center are likely to be insufficient on their own. Inferring activity externally through telemetry or physical measurement is likely to be more robust against some forms of adversarial manipulation. Analog sensors can also provide an additional set of signals that make it more challenging to obfuscate any prohibited activities. In a break-out discussion at AVID, Amir Nuriyev and Gabriel Kulp (MATS) discussed initial work on measuring power signatures of AI chips to enable high-level classification of workloads without leaking secrets.

Tamper-resistance, formal verification and supply chain assurance: One recurring theme in discussion was that the mechanisms discussed above will need to offer strong guarantees of confidentiality and security to see real-world adoption, particularly in an international context. Adam Chlipala (MIT) discussed how cheaper formal verification could play a major role in eliminating software and hardware vulnerabilities. There could also be a role for open-source designs for sensors or other key components to build confidence that they are not capable of exfiltrating more information than intended. Quintus Kilbourn (Flashbots) outlined challenges with supply chain security for the hardware that confidential computing relies on, and avenues to develop open-source hardware that could more easily be inspected for vulnerabilities. While preventing tampering by attackers with physical access is challenging, layering defenses and including systems for tamper detection could raise the costs for covert attackers. Shahin Tajik (Worcester Polytechnic Institute) presented some approaches to monitoring and tamper detection that leverage physical properties of AI hardware with the goal of achieving greater robustness against nation-state adversaries. 

Future research directions: One direction for future research raised during the workshop is moving beyond assurance of deployed AI models at inference time and towards assurance of training processes, such as generating evidence to back claims on the amount of computation used. As AI capabilities continue to improve, a further question is how to apply AI models to perform confidential analysis and share aggregated results, and more generally set up verification systems so that they can take advantage of the speed and scalability of AI-powered analysis without compromising security or accuracy. Lastly, there is a question of how to design verification systems that do not require retrofitting or replacement of data center hardware, to mitigate security concerns and reduce cost and time requirements. One relevant proposal outlined by Mauricio Baker (RAND) would use memory challenge-response protocols or cryptographic proofs of memory erasure as a foundation for proving what activities are taking place in a data center.

A recurring theme throughout the workshop was that the challenge is not simply verifying AI systems in the abstract, but developing practical mechanisms for generating trustworthy evidence under real-world constraints: adversarial environments, limited trust, operational complexity, and strong confidentiality requirements. Meaningful assurance will likely require combining approaches across hardware, systems, cryptography, and external measurement, each with different trust assumptions and failure modes. While many of the approaches discussed are not yet ready to deploy at scale, participants repeatedly emphasized that recent advances in confidential computing, cryptographic proofs, and formal methods, among others, mean that higher-assurance AI infrastructure is becoming technically feasible. 

Opportunities to get involved: There are a number of upcoming opportunities for researchers, engineers and others interested in getting involved in working on these questions:

• The workshop organizers are offering quick-turnaround research grants for work on these topics - further details can be found here (deadline: June 8th)  
• CAISH is running a week-long summer hackathon program in Cambridge, UK for researchers and engineers interested in diving deeper into this topic, with the opportunity to pitch to funders and organizations that are hiring in this space (deadline: June 7th)
• Coefficient Giving is running an incubator to support people interested in working on challenges such as those described above by founding new organizations or joining existing ones (deadline: June 16th)
• Amodo, Lucid Computing and RAND are hiring researchers and engineers to develop prototypes of verification systems
• Institute for Progress is running a request for proposals which covers AI verification and evaluation, with winning proposals being published, receiving a $10k honorarium and support in finding funding and a team to execute on the proposal
• Heron Security has a list of other opportunities to learn more and get involved in working on AI security challenges such as those discussed here 

The full playlist of workshop talks is now live on YouTube.

A few highlights:

• Shahin Tajik (Worcester Polytechnic Institute) on physical verification of AI systems against nation-state adversaries.
• Koen van der Veen (OpenMined) on PySyft v2 for AI auditing of private models and user logs.
• Bing-Jyue Chen (University of Illinois Urbana-Champaign) on efficient zero-knowledge proofs for AI inference.
• Roy Rinberg (Harvard) on inference verification in a TEE
• Adam Chlipala (MIT) on end-to-end formal verification of computing infrastructure
• Amean Asad (Confidential AI) on scalable private inference for frontier AI.
• Quintus Kilbourn (Flashbots) on trust in silicon.
• Ari Juels (Cornell Tech) on security tools from crypto for AI. 

‍

Interested in future events? Submit your interest here.