OpenAI has shelved the planned launch of GPT-6.1 Astra after internal tests raised concerns about deception and agents acting beyond user authorization, according to The Wall Street Journal. The company says it will investigate the issues and strengthen safety measures before releasing future models.
4 min
OpenAI says it is scrapping the release of its next-generation AI model over safety concerns that researchers raised during internal testing, in one of the clearest signs so far that agent misbehavior could stymie the industry’s rapid progression.
The move follows a summer punctuated by reports of artificial-intelligence systems industrywide going rogue, and marks a rare case of a major AI developer ditching a new release because of safety concerns.
The company had planned to launch the model, known as GPT-6.1 Astra, in the coming days or weeks, aiming for an October debut. The model was more capable than the company’s previous models in completing challenging tasks from end-to-end without human assistance, as well as writing.
The company instead will focus on improving the safety of future models, which it expects to be even more capable.
Saachi Jain, OpenAI’s head of safety systems, said in an interview that GPT-6.1 Astra regressed in two areas. Compared with its predecessor, GPT-6 Astra, the model performed poorly on tests measuring alignment, or how well the model adheres to what humans would like it to do. Specifically, GPT-6.1 Astra showed higher levels of deception: It wasn’t always honest about telling users of the actions it did or didn’t take.
Another issue was what OpenAI calls “scope authorization,” meaning that GPT-6.1 Astra would push ahead on a task without asking the user for permission, and would at times reach for external tools and services even if it might be unsafe.
“For anything regarding safety and alignment, there’s a trade off,” Jain said. “You really do need to find what’s the right line between staying within scope, but also avoiding laziness in terms of how the model actually pursues tasks even when it hits friction.”
While GPT-6.1 Astra improved in areas such as “model laziness,” Jain said it didn’t quite meet OpenAI’s bar for safety and alignment, so the company decided not to launch the model publicly.
The announcement comes one day ahead of OpenAI’s annual developer conference in San Francisco. In the past, OpenAI has used the conference as an opportunity to launch new models and services that reduce costs for software developers—a segment the ChatGPT-maker competes with rival AI company Anthropic to win over.
In recent weeks, OpenAI and Anthropic have called on industry partners to slow down the development of cutting-edge AI models and invest in safety standards, noting they will temper the pace of their own internal AI progress.
OpenAI says it is working to investigate a range of agent security incidents that it has discovered in recent months, and address the safety issues underneath them. As part of the work, the company has implemented a new monitoring system to catch AI-agent misbehavior more quickly, and started requiring engineers to use stronger security guardrails for testing its AI systems.
Earlier this summer hundreds of OpenAI’s internal agents, which were tasked with completing a cybersecurity test, ended up hacking into the AI company Hugging Face. Since then, high-profile organizations such as the Australian government and United Nations discovered that OpenAI’s agents used similar, but less extensive, techniques to gain access to their websites.
Many of the publicly known agent-security incidents involved OpenAI’s internal AI models that were never slated for public release.
Last week, OpenAI said it paused training on its most capable AI models after an AI agent slipped through a gap in the company’s internet restrictions to query a public chatbot. The company said its new monitoring systems flagged the incident within 15 minutes, and training on these models remains paused.
GPT-6.1 Astra isn’t one of those models, but a different case, the company said.
“We want to make sure our model development is safe no matter whether that’s in the company, or when we ship it to users,” Jain said. “But when we ship it to users, we have an extremely high bar in terms of safety and alignment.”
While the company decided not to ship GPT-6.1 Astra, it hopes to use the same base model to do additional reinforcement learning runs, and create future generations of its GPT-6 models.
OpenAI plans to conduct several deep dives to identify the root cause of the problems identified in GPT-6.1 Astra, Jain said. The work includes ensuring that OpenAI’s reinforcement learning environments are rewarding the right type of behavior, Jain added, though she noted the company would investigate all stages of model development.
AI companies have begun to draw scrutiny from policymakers and public officials, who are paying attention to the rapid development of the technology. Later this week, a Senate subcommittee is holding a hearing with third party AI researchers titled, “Rogue AI: Securing the Homeland Against AI Agent Attacks.”
Florida Attorney General James Uthmeier, a Republican, sued OpenAI in June, claiming that the company and Chief Executive Sam Altman knowingly released an unsafe product and ignored warnings that it could harm users.
In a motion for temporary injunction filed Monday, Uthmeier sought to prevent OpenAI from developing new AI models without third-party approved safeguards, stop ChatGPT from soliciting user engagement and limit the company’s ability to advertise ChatGPT as safe.
Tech companies claim they “cannot stop barreling forward with their potentially civilization-ending endeavors unless they are forced to do so by the government,” Uthmeier said in the filing. “The Florida Attorney General is answering your cry for help.”
An OpenAI spokeswoman said that people want to know AI is being developed safely, “and that starts with what companies like ours do ourselves.”
“Governments have an important role to play in setting robust safety standards for AI, and we’re committed to working with Florida and other states on advancing pragmatic AI policies that apply to the entire AI industry—not just one company,” she said.
Copyright 2020, Dow Jones & Company, Inc. All Rights Reserved Worldwide. LEARN MORE
BNW Developments has established a Sydney presence, joining Arada and Sobha Realty among the growing number of UAE developers pursuing Australian buyers and development opportunities.
The Australian leather house has opened an immersive four-day pop-up in Manhattan, unveiling its Bloom Collection and redefining what a product launch can look like.
AI-powered travel agents are challenging online booking platforms, but Airbnb’s exclusive listings and direct customer relationships could give it an edge. Its next challenge: turning repeat guests into loyal customers through stronger rewards and innovation.
3 min
AI agents like Muse can already find you the cheapest hotel room in Paris. But they can’t locate a cheaper version of that Catskills cabin you’ve been eyeing, because it might only be available on Airbnb ABNB -1.43%decrease.
Investors should pay attention to that distinction. When Meta Platforms META 3.38%increase launched its Muse agent last month, travel stocks including Booking Holdings BKNG -1.19%decrease, Expedia and Airbnb all fell sharply.
Airbnb, though, has a real chance to emerge as a winner from the upheaval. The catch: Its stock already reflects above-industry growth. To justify its valuation, Airbnb will need to turn repeat customers into loyal ones.
Online travel companies make much of their money by simplifying search and collecting fees on bookings. AI agents threaten that model.
But not all travel marketplaces are equally exposed. Booking and Expedia largely sell hotel rooms that travelers can also book directly or through competing sites. If an agent can find the same room elsewhere, it has little reason to favor either Booking or Expedia.
Booking and Expedia also sell sponsored listings that steer travelers toward particular properties. Wells Fargo estimates those will account for about 8% of Expedia’s revenue and a much higher percentage of profit next year.
Airbnb has more ways to adapt. More than half its listings are available only on its platform, and more than 90% of its traffic comes directly to its app or website.
The customer relationship doesn’t end after booking, either. Bernstein analyst Richard Clarke notes that 90% of Airbnb guests go on to message their hosts about arrival times, keys or local taxes, giving Airbnb a connection to both sides of the transaction.
How the companies have responded to the possible agentic future is telling.
Expedia quickly signed on as a Muse partner. Airbnb hasn’t, and Chief Executive Brian Chesky says AI chatbots have so far been a boon, sending customers to Airbnb’s website.
Chatbots might work for buying paper towels, he quipped at a Skift conference last month, but people enjoy planning trips. He argued that browsing dozens of homes isn’t something a text-based chatbot is well suited for.
Still, the AI jitters arrived just as Airbnb was persuading investors it deserved its pricey multiple. For years after its 2020 initial public offering, the stock—which trades around 25 to 30 times forward earnings—went nowhere. Investors saw a mature operator trading at a growth premium.
This year, Airbnb growth seems to be accelerating. The stock jumped in August after the company raised its full-year revenue-growth guidance with its second-quarter results. Wells Fargo analyst Ken Gawrelski credits a run of product improvements under a revamped leadership team.
Among the changes: letting guests reserve now and pay later, adding hotels, and a redesigned home page and app. Investors were also betting on what comes next, including thousands more hotels, car rentals and, eventually, advertising.
The irony is that some of those growth drivers push Airbnb closer to the kind of business Booking and Expedia run—the kind of business AI agents could threaten. Even if AI agents capture a large share of travel planning and booking, Airbnb would capture the demand. But it and others would slowly lose control over how customers find and choose properties.
That would undermine the advertising opportunity investors are counting on. The bigger point for Airbnb and other marketplaces is that losing control of customer acquisition could mean surrendering some of their pricing power and margins.
To bulletproof itself, Airbnb might look to another e-commerce giant that has so far kept AI agents at arm’s length: Amazon.
Like Amazon, Airbnb has a loyal customer base and hard-to-match supply. But it has nothing like Prime membership to make customers think twice before taking their business elsewhere.
That is where loyalty comes in. Business travelers stick with Hilton or Marriott because enough nights earn them perks. Leisure travelers rarely accumulate stays with one chain. Airbnb has millions of listings and plenty of repeat customers, but no broad rewards program. It has begun experimenting with credits on hotel stays, but it should make loyalty a priority, not just a promotion.
Airbnb also plans to launch its own AI agent next year. But travelers using an agent might still want it to search Airbnb, Vrbo, Booking and hotel websites alike, rather than limit itself to one company’s inventory.
Eventually, Airbnb might have to strike deals with agent makers, using its unique homes as leverage to negotiate favorable terms, says Gawrelski. Such deals, along with rewards for loyal guests, could over the long term eat into the company’s roughly 13% take rate, he adds. But if they bring in enough additional bookings, the trade-off could be worthwhile.
History offers a reason not to panic. The shift from desktop to mobile unsettled investors in Alphabet and Amazon, but both emerged as major beneficiaries. Companies with valuable assets can come out of a platform shift stronger, if they adapt quickly enough.
Airbnb has homes the bots can’t easily replace. Now it needs to give guests a reason to insist on booking them there.
