Preparing for the Next Worldwide Tech Outage

As tech leaders race to bring Windows systems back online after Friday’s software update by cybersecurity company CrowdStrike crashed around 8.5 million machines worldwide, experts share with CIO Journal their takeaways for preparing for the next major information technology outage.

Be familiar with how vendors develop, test and release their software

IT leaders should hold vendors deeply integrated within IT systems, such as CrowdStrike , to a “very high standard” of development, release quality and assurance, said Neil MacDonald , a Gartner vice president.

“Any security vendor has a responsibility to do extensive regression testing on all versions of Windows before an update is rolled out,” he said.

That involves asking existing vendors to explain how they write software, what testing they do and whether customers may choose how quickly to roll out an update.

“Incidents like this remind all of us in the CIO community of the importance of ensuring availability, reliability and security by prioritizing guardrails such as deployment and testing procedures and practices,” said Amy Farrow, chief information officer of IT automation and security company Infoblox.

Re-evaluate how your firm accepts software updates from ‘trusted’ vendors

While automatically accepting software updates has become the norm—and a recommended security practice—the CrowdStrike outage is a reminder to take a pause, some CIOs said.

“We still should be doing the full testing of packages and upgrades and new features,” said Paul Davis, a field chief information security officer at software development platform maker JFrog . undefined undefined Though it’s not feasible to test every update, especially for as many as hundreds of software vendors, Davis said he makes it a priority to test software patches according to their potential severity and size.

Automation, and maybe even artificial intelligence-based IT tools, can help.

“Humans are not very good at catching errors in thousands of lines of code,” said Jack Hidary, chief executive of AI and quantum company SandboxAQ. “We need AI trained to look for the interdependence of new software updates with the existing stack of software.”

Develop a disaster recovery plan

An incident rendering Windows computers unusable is similar to a natural disaster with systems knocked offline, said Gartner’s MacDonald. That’s why businesses should consider natural disaster recovery plans for maintaining the resiliency of their operations.

One way to do that is to set up a “clean room,” or an environment isolated from other systems, to use to bring critical systems back online, according to Chirag Mehta, a cybersecurity analyst at Constellation Research.

Businesses should also hold tabletop exercises to simulate risk scenarios, including IT outages and potential cyber threats, Mehta said.

Companies that back up data regularly were likely less impacted by the CrowdStrike outage, according to Victor Zyamzin, chief business officer of security company Qrator Labs. “Another suggestion for companies, and we’ve been saying that again and again for decades, is that you should have some backup procedure applied, running and regularly tested,” he said.

Review vendor and insurance contracts

For any vendor with a significant impact on company operations , MacDonald said companies can review their contracts and look for clauses indicating the vendors must provide reliable and stable software.

“That’s where you may have an advantage to say, if an update causes an outage, is there a clause in the contract that would cover that?” he said.

If it doesn’t, tech leaders can aim to negotiate a discount serving as a form of compensation at renewal time, MacDonald added.

The outage also highlights the importance of insurance in providing companies with bottom-line protection against cyber risks, said Peter Halprin, a partner with law firm Haynes Boone focused on cyber insurance.

This coverage can include protection against business income losses, such as those associated with an outage, whether caused by the insured company or a service provider, Halprin said.

Weigh the advantages and disadvantages of the various platforms

The CrowdStrike update affected only devices running Microsoft Windows-based systems , prompting fresh questions over whether enterprises should rely on Windows computers.

CrowdStrike runs on Windows devices through access to the kernel, the part of an operating system containing a computer’s core functions. That’s not the same for Apple ’s Mac operating system and Linux, which don’t allow the same level of access, said Mehta.

Some businesses have converted to Chromebooks , simple laptops developed by Alphabet -owned Google that run on the Chrome operating system . “Not all of them require deeper access to things,” Mehta said. “What are you doing on your laptop that actually requires Windows?”

Wall Street’s hottest trade has gone ice cold.

For years, it paid off to buy stocks that were rising in price—and bet against struggling shares. The momentum trade was especially profitable this year, as investors piled into hot stocks including Micron Technology, Nvidia, Advanced Micro Devices and other artificial-intelligence darlings while wagering against those likely to be hurt by the embrace of AI.

The S&P 500 Momentum Index soared 44% in the second quarter, its best quarterly performance on record, and it surged 133% over the past five years, nearly double the broad market’s performance.

Mega funds and rookie investors alike piled into the trade, some using leverage and options contracts in an effort to amplify their returns, propelling the underlying shares higher.

“It is a self-fulfilling prophecy,” said Matthew Tym, managing director at Cantor Fitzgerald, of the trade.

Suddenly, the trade is a loser. The momentum index has tumbled more than 9% since July 1, lagging behind the S&P 500’s 2.8% gain. The index—which tracks stocks in the S&P 500 based on a “momentum score”—is on track for the biggest quarterly underperformance in 25 years. July was the second-worst month for the momentum trade in around 40 years, according to Bank of America estimates; the only month worse was April 2009, in the teeth of the global financial crisis.

Hedge funds that bought momentum shares while shorting low-momentum stocks suffered even more. At the same time, a basket of the most popular stocks held by hedge funds tracked by Goldman Sachs recorded its biggest one-month underperformance in July relative to the S&P 500 in more than 20 years, according to the bank’s analysts.

Momentum trading is based on a rather simple observation: Investments that go up tend to keep outperforming; those that underperform often remain laggards. This kind of trading might seem too simple a stock-picking strategy to work. Yet it often has.

“For decades, it didn’t take a lot of sophistication to run a momentum strategy and make a decent living at it,” says Agustin Lebron, senior researcher at EquiLibre, a trading firm.

Part of the reason: It takes a while for corporate and other information to spread to various investors, so they slowly build positions, producing buying momentum.

“A huge pension fund can’t flip around its positions in a day,” says Lebron. “Behavioral biases also account for some of the effect, as well—people tend to sell their winners too early and hold losers too long.”

Fans of the strategy point to the human tendency to extrapolate from past results—and chase investment returns—noting that momentum patterns have been evident in markets for decades, even centuries. They also say that some of the worst months for momentum strategies are during longer periods of outperformance.

Some have been doing the trade by buying the strongest investments in a sector while shorting the weakest; others lean in to rising markets or asset classes. Still others use a quantitative approach or turn to banks or others who sell ways to make distinct wagers on momentum as a “tradable factor” or a “thematic basket.”

The fans remain believers. “Any strategy has disappointing periods,” says Antti Ilmanen, global co-head of the portfolio solutions group at AQR Capital Management.

The surge in Moderna and other biotech stocks helped crush the momentum trade. These shares were among the most heavily shorted in recent years, but positive news on a cancer vaccine from Moderna and Merck sent those stocks flying, crushing some quant and other hedge funds. Moderna is up around 150% so far this month.

These traders had an especially rough day on Aug. 19, which Goldman Sachs told its clients was the worst day for “systematic long-short managers” in more than two years. About half of the losses were because of momentum trades, the bank said.

Some traders have begun to short, or bet against, the very stocks that propelled the momentum trade earlier this year. Net short positions in futures tied to the Nasdaq-100 index among speculators recently climbed to some of the highest levels of the past two decades, according to data from the Commodity Futures Trading Commission.

The about-face is a sign of how markets have become more treacherous for investors, even as indexes keep climbing. Part of the issue: the recent meltdown of Situational Awareness, a hedge fund that had piled into some of the most popular momentum shares, including chip stocks. After a period of market tumult, Nvidia shares rocketed almost 9% after its earnings, showing how quickly sentiment can shift.

Some investors say the run-up in share prices driving tech stocks higher reminds them at times of the dot-com frenzy decades ago.

Mike Ogborne, the founder of San Francisco-based Ogborne Capital Management, said he has grown more cautious on technology stocks and is keeping more of his portfolio in cash than he typically does.

And he is nervous about the surge in spending by technology giants and quarterly capital expenditures that keep rising.

“It is a little bit like Cinderella and the clock striking midnight. You don’t know when midnight is going to come around,” Ogborne said. “They don’t send a memo around telling you when the capex cycle is over.”