M.B.A. Students vs. ChatGPT: Who Comes Up With More Innovative Ideas? - Kanebridge News
Share Button

M.B.A. Students vs. ChatGPT: Who Comes Up With More Innovative Ideas?

We put humans and AI to the test. The results weren’t even close.

By CHRISTIAN TERWIESCH
Thu, Sep 14, 2023 9:01amGrey Clock 4 min

How good is AI in generating new ideas?

The conventional wisdom has been not very good. Identifying opportunities for new ventures, generating a solution for an unmet need, or naming a new company are unstructured tasks that seem ill-suited for algorithms. Yet recent advances in AI, and specifically the advent of large language models like ChatGPT, are challenging these assumptions.

We have taught innovation, entrepreneurship and product design for many years. For the first assignment in our innovation courses at the Wharton School, we ask students to generate a dozen or so ideas for a new product or service. As a result, we have heard several thousand new venture ideas pitched by undergraduate students, M.B.A. students and seasoned executives. Some of these ideas are awesome, some are awful, and, as you would expect, most are somewhere in the middle.

The library of ideas, though, allowed us to set up a simple competition to judge who is better at generating innovative ideas: the human or the machine.

In this competition, which we ran together with our colleagues Lennart Meincke and Karan Girotra, humanity was represented by a pool of 200 randomly selected ideas from our Wharton students. The machines were represented by ChatGPT4, which we instructed to generate 100 ideas with otherwise identical instructions as given to the students: “generate an idea for a new product or service appealing to college students that could be made available for $50 or less.”

In addition to this vanilla prompt, we also asked ChatGPT for another 100 ideas after providing a handful of examples of successful ideas from past courses (in other words, a trained GPT group), providing us with a total sample of 400 ideas.

Collapsible laundry hamper, dorm-room chef kit, ergonomic cushion for hard classroom seats, and hundreds more ideas miraculously spewed from a laptop.

How to compare

The academic literature on ideation postulates three dimensions of creative performance: the quantity of ideas, the average quality of ideas, and the number of truly exceptional ideas.

First, on the number of ideas per unit of time: Not surprisingly, ChatGPT easily outperforms us humans on that dimension. Generating 200 ideas the old-fashioned way requires days of human work, while ChatGPT can spit out 200 ideas with about an hour of supervision.

Next, to assess the quality of the ideas, we market tested them. Specifically, we took each of the 400 ideas and put them in front of a survey panel of customers in the target market via an online purchase-intent survey. The question we asked was: “How likely would you be to purchase based on this concept if it were available to you?” The possible responses ranged from definitely wouldn’t purchase to definitely would purchase.

The responses can be translated into a purchase probability using simple market-research techniques. The average purchase probability of a human-generated idea was 40%, that of vanilla GPT-4 was 47%, and that of GPT-4 seeded with good ideas was 49%. In short, ChatGPT isn’t only faster but also on average better at idea generation.

Still, when you’re looking for great ideas, averages can be misleading. In innovation, it’s the exceptional ideas that matter: Most managers would prefer one idea that is brilliant and nine ideas that are flops over 10 decent ideas, even if the average quality of the latter option might be higher. To capture this perspective, we investigated only the subset of the best ideas in our pool—specifically the top 10%. Of these 40 ideas, five were generated by students and 35 were created by ChatGPT (15 from the vanilla ChatGPT set and 20 from the pre trained ChatGPT set). Once again, ChatGPT came out on top.

What it means

We believe that the 35-to-5 victory of the machine in generating exceptional ideas (not to mention the dramatically lower production costs) has substantial implications for how we think about creativity and innovation.

First, generative AI has brought a new source of ideas to the world. Not using this source would be a sin. It doesn’t matter if you are working on a pitch for your local business-plan competition or if you are seeking a cure for cancer—every innovator should develop the habit of complementing his or her own ideas with the ones created by technology. Ideation will always have an element of randomness to it, and so we cannot guarantee that your idea will get an A+, but there is no excuse left if you get a C.

Second, the bottleneck for the early phases of the innovation process in organisations now shifts from generating ideas to evaluating ideas. Using a large language model, an innovator can produce a spreadsheet articulating hundreds of ideas, which likely include a few blockbusters. This abundance then demands an effective selection mechanism to find the needles in the haystack.

To date, these models appear to perform no better than any single expert in their ability to predict commercial viability. Using a sample of a dozen or so independent evaluations from potential customers in the target market—a wisdom of crowds approach—remains the best strategy. Fortunately, screening ideas using a purchase intent survey of customers in the target market is relatively fast and cheap.

Finally, rather than thinking about a competition between humans and machines, we should find a way in which the two work together. This approach in which AI takes on the role of a co-pilot has already emerged in software development. For example, our human (pilot) innovator might identify an open problem. The AI (co-pilot) might then report what is known about the problem, followed by an effort in which the human and AI independently explore possible solutions, virtually guaranteeing a thorough consideration of opportunities.

The human decision maker is likely ultimately responsible for the outcome, and so will likely make the screening and selection decisions, informed by customer research and possibly by the opinion of the AI co-pilot. We predict such a human-machine collaboration will deliver better products and services to the market, and improved solutions for whatever society needs in the future.

Christian Terwiesch and Karl Ulrich are professors of operations, information and decisions at the Wharton School of the University of Pennsylvania, where Terwiesch also co-directs the Mack Institute for Innovation Management.



MOST POPULAR

What a quarter-million dollars gets you in the western capital.

Alexandre de Betak and his wife are focusing on their most personal project yet.

Related Stories
Money
To Find Winning Stocks, Investors Often Focus on the Laggards. They Shouldn’t.
By KEN SHREVE 12/06/2024
Money
Louis Vuitton Unveils Its Most Extravagant High-Jewellery Collection Ahead of Olympics
By LAURIE KAHLE 09/06/2024
Money
Sylvester Stallone Sells His Knockout Watch Collection, Including the Most Valuable Modern Timepiece Sold in Sotheby’s History
By ERIC GROSSMAN 08/06/2024

These stocks are getting hit for a reason. Instead, focus on stocks that show ‘relative strength.’ Here’s how.

By KEN SHREVE
Wed, Jun 12, 2024 4 min

A lot of investors get stock-picking wrong before they even get started: Instead of targeting the top-performing stocks in the market, they focus on the laggards—widely known companies that look as if they are on sale after a period of stock-price weakness.

But these weak performers usually are going down for good reasons, such as for deteriorating sales and earnings, market-share losses or mutual-fund managers who are unwinding positions.

Decades of Investor’s Business Daily research shows these aren’t the stocks that tend to become stock-market leaders. The stocks that reward investors with handsome gains for months or years are more often  already  the strongest price performers, usually because of outstanding earnings and sales growth and increasing fund ownership.

Of course, many investors already chase performance and pour money into winning stocks. So how can a discerning investor find the winning stocks that have more room to run?

Enter “relative strength”—the notion that strength begets more strength. Relative strength measures stocks’ recent performance relative to the overall market. Investing in stocks with high relative strength means going with the winners, rather than picking stocks in hopes of a rebound. Why bet on a last-place team when you can wager on the leader?

One of the easiest ways to identify the strongest price performers is with IBD’s Relative Strength Rating. Ranked on a scale of 1-99, a stock with an RS rating of 99 has outperformed 99% of all stocks based on 12-month price performance.

How to use the metric

To capitalise on relative strength, an investor’s search should be focused on stocks with RS ratings of at least 80.

But beware: While the goal is to buy stocks that are performing better than the overall market, stocks with the highest RS ratings aren’t  always  the best to buy. No doubt, some stocks extend rallies for years. But others will be too far into their price run-up and ready to start a longer-term price decline.

Thus, there is a limit to chasing performance. To avoid this pitfall, investors should focus on stocks that have strong relative strength but have seen a moderate price decline and are just coming out of weeks or months of trading within a limited range. This range will vary by stock, but IBD research shows that most good trading patterns can show declines of up to one-third.

Here, a relative strength line on a chart may be helpful for confirming an RS rating’s buy signal. Offered on some stock-charting tools, including IBD’s, the line is a way to visualise relative strength by comparing a stock’s price performance relative to the movement of the S&P 500 or other benchmark.

When the line is sloping upward, it means the stock is outperforming the benchmark. When it is sloping downward, the stock is lagging behind the benchmark. One reason the RS line is helpful is that the line can rise even when a stock price is falling, meaning its value is falling at a slower pace than the benchmark.

A case study

The value of relative strength could be seen in Google parent Alphabet in January 2020, when its RS rating was 89 before it started a 10-month run when the stock rose 64%. Meta Platforms ’ RS rating was 96 before the Facebook parent hit new highs in March 2023 and ran up 65% in four months. Abercrombie & Fitch , one of 2023’s best-performing stocks, had a 94 rating before it soared 342% in nine months starting in June 2023.

Those stocks weren’t flukes. In a study of the biggest stock-market winners from the early 1950s through 2008, the average RS rating of the best performers before they began their major price runs was 87.

To see relative strength in action, consider Nvidia . The chip stock was an established leader, having shot up 365% from its October 2022 low to its high of $504.48 in late August 2023.

But then it spent the next four months rangebound—giving up some ground, then gaining some back. Through this period, shares held between $392.30 and the August peak, declining no more than 22% from top to bottom.

On Jan. 8, Nvidia broke out of its trading range to new highs. The previous session, Nvidia’s RS rating was 97. And that week, the stock’s relative strength line hit new highs. The catalyst: Investors cheered the company’s update on its latest advancements in artificial intelligence.

Nvidia then rose 16% on Feb. 22 after the company said earnings for the January-ended quarter soared 486% year over year to $5.16 a share. Revenue more than tripled to $22.1 billion. It also significantly raised its earnings and revenue guidance for the quarter that was to end in April. In all, Nvidia climbed 89% from Jan. 5 to its March 7 close.

And the stock has continued to run up, surging past $1,000 a share in late May after the company exceeded that guidance for the April-ended quarter and delivered record revenue of $26 billion and record net profit of $14.88 billion.

Ken Shreve  is a senior markets writer at Investor’s Business Daily. Follow him on X  @IBD_KShreve  for more stock-market analysis and insights, or contact him at  ken.shreve@investors.com .