[ad_1]
On Thursday, OpenAI released what could effectively be considered a $200-a-month chatbot — and the AI community hasn’t quite figured out what to do with it.
The company’s new ChatGPT Pro plan gives access to “o1 pro mode,” which OpenAI says “uses more computing to get the best answers to the toughest questions.” An improved version of OpenAI’s o1 inference model, the o1 pro mode should answer science, math and programming questions more “reliably” and “comprehensively,” OpenAI says.
Almost immediately, people started asking him to draw unicorns:
I asked ChatGPT o1 Pro Mode to create an SVG image of a unicorn.
(This is the model you can access for $200 per month) pic.twitter.com/h9HwY3aYwU
– Ramy (@rammydev) December 5, 2024
And designing a “crab-based” computer:
Finally, the o1-pro has been put to its final use. pic.twitter.com/nX4JAjx71m
– Ethan Mollick (@emollick) December 6, 2024
And wax poetic on the meaning of life:
I just signed up for a $200/month OpenAI subscription.
Reply with questions to ask and I will republish them in this topic. pic.twitter.com/oTQxbPxnoP– Garrett Scott 🕳 (@thegarrettscott) December 5, 2024
But many people on the
“Has OpenAI shared any concrete examples of claims that fail in regular o1 but succeed in o1-pro?” he asked British computer scientist Simon Willison. “I want to see one concrete example that shows its benefit.”
It’s a reasonable question. After all, this is the most expensive chatbot subscription in the world. The service comes with other benefits, such as removing price caps and unlimited access to other OpenAI models. But $2,400 a year isn’t chump change, and the value proposition for the o1 pro mode in particular remains vague.
It didn’t take long to find failures. The O1 pro mode suffers from Sudoku, and is bogged down by an optical illusion joke that’s obvious to any human.
Both o1 and o1-pro fail here, probably still due to visibility limitations (same with Sudoku puzzles)https://t.co/mAVK7WxBrq pic.twitter.com/O9boSv7ZGt
– Tibor Blaho (@btibor91) December 5, 2024
OpenAI’s internal benchmarks show that o1 pro mode performs slightly better than o1 standard mode on programming and math problems:
OpenAI conducted a “more stringent” evaluation on the same criteria to showcase the consistency of the o1 pro mode: a model was deemed to have solved a question only if it got the answer correct four out of four times. But even in these tests, the improvements were not dramatic:
OpenAI’s CEO, Sam Altman, once wrote that OpenAI was on its way to… road “Towards an intelligence too cheap to be measured,” he obliged It is clear numerous times Thursday that ChatGPT Pro is not suitable for most people.
“Most users will be very happy with o1 in the (ChatGPT) Plus layer!” “Almost everyone would be better served by the free tier or the extra tier,” he said on X.
So for whom? Are there really people willing to pay $200 a month to ask gaming questions like “Write a 3-paragraph essay about strawberries without using the letter “e”“or”Solve this Math Olympiad problem“?Would they be happy to part with their hard-earned money without much assurance that the o1 standard cannot satisfactorily answer the same questions?
asked Amit Talwalkar, an associate professor of machine learning at the university Carnegie Mellon and Project Partner at Amplify Partners, his opinion. “Raising the price tenfold seems like a big risk to me,” he told TechCrunch via email. “I think we’ll have a much better sense in a few weeks as to the desirability of this job.”
Computer scientist at the University of California, Jay van den Broek, was more blunt in his assessment. “I don’t know if the price point makes sense, and if expensive thinking models will be the norm,” he told TechCrunch.
o1 “Better than most humans at most tasks” because, yes, humans exist exclusively in amnesic multi-role chat interfaces https://t.co/zbLY2BG5pQ
– Aidan McClau (@aidan_mclau) December 6, 2024
What’s ridiculous is that it’s a marketing mistake. Describing the o1 pro mode as being best at solving “the toughest problems” doesn’t tell potential customers much. And don’t Vague statements About how the model can “think longer” and show “intelligence.” As Willison points out, without specific examples of this supposedly improved capability, it’s hard to justify paying more at all, let alone ten times the price.
As far as I can tell, experts in specialized fields are the target audience. OpenAI says it plans to give a small number of medical researchers at “leading institutions” free access to ChatGPT Pro, which will include an o1 pro mode. Errors are very important in healthcare, and as Bob McGraw, former chief research officer at OpenAI, says: male On the
Been playing with o1 and o1-pro for some time.
It’s very good and a little strange. It is also not suitable for most people most of the time. You really need to have certain hard problems to solve in order to get value from them. But if you have these problems, it’s a very big problem.
– Ethan Mollick (@emollick) December 5, 2024
McGraw too pensive The o1 Pro mode is an example of what he calls “intelligence burden”: users (and perhaps model creators) don’t know how to get value from any “additional intelligence” due to the fundamental limitations of the simple text-based interface. As with other OpenAI models, the only way to interact with o1 pro mode is through ChatGPT, and — to McGrew’s point — ChatGPT isn’t perfect.
It is also true that the $200 price raises expectations. And judging by the early reception on social media, ChatGPT Pro isn’t a huge success.
[ad_2]