Has a Chinese physical AI start-up manipulated a global ranking to beat Nvidia?

A Chinese physical AI start-up’s brief claim to global dominance in robotics has run into controversy, underscoring the intense US-China competition to develop next-generation artificial intelligence and the challenges of evaluating autonomous systems. In June, Spirit AI, a Hangzhou, Zhejiang province-based firm founded in 2024, briefly overtook United States tech giant Nvidia to take the top spot on RoboArena – a global benchmark for physical AI – with its new Spirit v1.6 model, launched at the beginning of that month. However, the victory was short-lived. Just days later, the benchmark’s creators overhauled its methodology and removed the model from the official list, along with several other competitors, after finding evidence of “benchmark hacking”. Among those dropped from the rankings was a model from Chinese start-up X Square Robot, which had ranked fourth. But it was Spirit that generated the most buzz after topping the list, which it dubbed “the ‘Olympics’ of embodied intelligence in North America”, even as the benchmark itself had its own problems. RoboArena, co-developed by Nvidia and institutions including Stanford University and the University of California, Berkeley, evaluates how effectively generalist robot policies – the core software driving movement and execution – translate digital instructions into real-world actions. In a post on X in June, Pranav Atreya, a lead author of the project and a PhD student at UC Berkeley, said the team had “retroactively removed evaluations from organisations who [it] found to be engaging in benchmark manipulation”. He did not name specific companies. By Tuesday, Nvidia’s DreamZero – an action model unveiled in February – had claimed the top spot on the chart. The remainder of the top nine was dominated entirely by US players, including Google DeepMind and San Francisco, California-based robotics AI start-up Physical Intelligence, or Pi. The controversy highlights vulnerability in RoboArena’s original design, which had permitted anyone to sign up as an evaluator and conduct side-by-side comparative studies of two randomly assigned models. Because participating models were hosted on developer-owned servers rather than a centralised environment, creators could potentially game the system by powering on servers only when their own evaluators were active, increasing the odds of reviewing their own software, according to Chinese tech blog Roushen Suanfa, a name that translates to “flesh algorithm”. An evaluator account named ECUST Robot Lab conducted 58 per cent of all evaluations for Spirit v1.6, according to the blog’s analysis of over 4,600 evaluation records through June 3, the day the Spirit model was announced. It added that all the top five models, including Nvidia’s DreamZero, had at least half of their evaluations conducted by one specific account. Under the ECUST account, Spirit v1.6 achieved a 99 per cent win rate, compared with a 66 per cent win rate among independent evaluators. The evaluation records were subsequently wiped from the benchmark website, according to a review by the South China Morning Post. It remains unclear whether the ECUST account was affiliated with Spirit AI. To curb manipulation, RoboArena updated its methodology to restrict testing to vetted third-party volunteers – a move Atreya said “pre-empts any motivation to sway benchmark results”. The platform also introduced an “official” leaderboard requiring a minimum of 100 verified evaluations. On a secondary, unofficial ranking for models that have not met the threshold, Spirit v1.6 still holds the highest score, but is marked as “low sample” with just 25 recorded evaluations. Other entries, such as the MolmoAct2-droid model from Seattle-based non-profit Allen Institute for AI, were similarly excluded from the official list because of insufficient sample sizes. In a statement to the SCMP, Spirit AI said it was in communication with the RoboArena team, adding that the benchmark’s creators acknowledged the “strong performance” of the Chinese firm’s models. Quoting the benchmark team, Spirit AI maintained that its absence from the official leaderboard was mainly due to unfulfilled sample size requirements, “rather than a denial of the model’s capabilities”. Notably, Nvidia’s DreamZero, which ranked third, now tops the official list, while its model Cosmos 3, which held the second spot, has vanished from both the official and unofficial rankings. X Square Robot’s model, Wall-OSS, which ranked fourth on the old list, has also disappeared from both rankings. Nvidia, X Square and RoboArena did not respond to requests for comment. The incident underscores the challenges of creating reliable evaluation frameworks for physical AI as global tech entities race for leadership. RoboArena’s creators said in a now-deleted post on their website that a “decentralised design means individual bad actors cannot easily sway the results”, though they conceded: “However, we are not yet at this scale”. By Tuesday, the benchmark had recorded some 3,900 evaluations across fewer than 30 participating organisations.

Has a Chinese physical AI start-up manipulated a global ranking to beat Nvidia?

.css-1c6uqr6{color:inherit;font-weight:inherit;font-size:inherit;font-family:inherit;line-height:inherit;overflow-wrap:break-word;}Spirit AI’s Spirit v1.6 briefly dethroned Nvidia on the RoboArena robotics benchmark, but the model was then removed following a methodology overhaul

Spirit AI’s Spirit v1.6 briefly dethroned Nvidia on the RoboArena robotics benchmark, but the model was then removed following a methodology overhaul