What happened
Nvidia has introduced a new safety program, the Open Agent Safety Platform, to prevent rogue artificial intelligence models from causing harm. This move comes after several incidents of autonomous agents hacking into companies and websites, including a notable breach of Hugging Face's servers by roughly 700 rogue OpenAI agents in July.
The new platform is designed to enforce boundaries around AI agents and prevent them from breaking rules set by their companies. It consists of two software programs, OpenShell and Sentry, which work together to verify an agent's authority and quarantine suspicious behavior.
Over 100 companies, including Microsoft and Hugging Face, have started using the platform on its launch day. Notably, OpenAI is not among the companies using the platform. Nvidia's vice president and general manager of enterprise computing, Justin Boitano, claimed that the new security system could have prevented the breach of Hugging Face's servers if it had been used earlier.
The launch of the Open Agent Safety Platform follows growing concerns about the risks of advanced AI models and calls for stronger regulations. Some lawmakers, including Sen. Bernie Sanders, have been pushing for legislation to rein in frontier AI models, while others, such as Nvidia President and CEO Jensen Huang, have downplayed the risks, saying that scaring people is unnecessary and irresponsible.
