Meta faces flak for using an experimental Llama 4 Maverick AI to inflate benchmark scores. This prompted an apology & policy shift, now favoring the original version which lags behind OpenAI's GPT-4o, Anthropic's Claude 3.5, & Google's Gemini 1.5. Meta explained the experimental version was optimized for dialogue and excelled in LM Arena, but its benchmark reliability is debated. Meta clarifies they test various AI models, releasing the open-source version of Llama4 #AI #Meta #Llama4 #Benchmarks

RDNA 4 × Linux im Test: Benchmarks der Radeon RX 9070 XT unter Arch Linux https://www.computerbase.de/artikel/grafikkarten/amd-radeon-rx-9070-xt-linux-test.91853/ #linux #benchmarks #amd

Announcing the OpenAI Pioneers Program https://openai.com/index/openai-pioneers-program/ #AI #evals #benchmarks

Hallo schlaues Fediverse, ich tauche gerade in ein völlig absurdes #Rabbithole und mein M1 Macbook hat dank #LlmStudio und #Ollama seinen Lüfter wiederentdeckt… Aktuell ist lokal bei #LLM mit 8-12b Schluss (32GB Ram). Gibt es irgendwo #Benchmarks die mir bitte ausreden, dass das mit einem M4 >48GB RAM drastisch besser wird? Oder wäre was ganz anderes schlauer? Oder anderes Hobby? Muss Mobil (erreichbar) sein, weil zu unsteter Lebenswandel für ein Desktop. Empfehlungen gern in den Kommentaren.

Mashable: A new AI test is outwitting OpenAI, Google models, among others. “The Arc Prize Foundation, a nonprofit that measures AGI progress, has a new benchmark that is stumping the leading AI models. The test, called ARC-AGI-2 is the second edition ARC-AGI benchmark that tests models on general intelligence by challenging them to solve visual puzzles using pattern recognition, context […]

https://rbfirehose.com/2025/03/29/mashable-a-new-ai-test-is-outwitting-openai-google-models-among-others/

The real reason AI benchmarks haven’t reflected economic impacts https://epochai.substack.com/p/the-real-reason-ai-benchmarks-havent #AI #benchmarks

Wow #google published internal #benchmarks for their new #AI model, showing numbers in comparison to other models. So this is the new way to increase #stockvalue huh? #Marketing #llm

„Solche Ansätze (der angenommenen Gleichheit der Menschen) zwingen Maschinen dazu, Menschen einförmig zu behandeln, auch wenn es legitime Unterschiede gibt", erklärt Angelina Wang den Ansatz für die neuen KI-Benchmarks.

#KI #Bias #Benchmarks

https://t3n.de/news/wir-sind-nicht-alle-gleich-wie-neue-ki-benchmarks-sprachmodellen-bei-unterschieden-helfen-sollen-1677487/

Ah, the classic tale of a coder thinking #SIMD would make their code fly , only to discover it trips over its own feet . Our hero's memory seems as patchy as their #benchmarks, but fear not, the valuable lesson here is clear: #optimization is just a synonym for #headache.
https://genna.win/blog/convolution-simd/ #coding #woes #lessons #HackerNews #ngated

Nice overview of benchmarks/leaderboards for the various AI models.
https://scale.com/leaderboard
#AI #leaderboard #benchmarks

How would you interview an AI, to give it a job? https://www.strangeloopcanon.com/p/how-would-you-interview-an-ai-to #AI #evals #benchmarks

New Essay

"The Intelligent AI Coin: A Thought Experiment"

Open Access here: https://seanfobbe.com/posts/2025-02-21_intelligent-ai-coin-thought-experiment/

Recent years have seen a concerning trend towards normalizing decisionmaking by Large Language Models (LLM), including in the adoption of legislation, the writing of judicial opinions and the routine administration of the rule of law. AI agents acting on behalf of human principals are supposed to lead us into a new age of productivity and convenience. The eloquence of AI-generated text and the narrative of super-human intelligence invite us to trust these systems more than we have trusted any human or algorithm ever before.

It is difficult to know whether a machine is actually intelligent because of problems with construct validity, plagiarism, reproducibility and transferability in AI benchmarks. Most people will either have to personally evaluate the usefulness of AI tools against the benchmark of their own lived experience or be forced to trust an expert.

To explain this conundrum I propose the Intelligent AI Coin Thought Experiment and discuss four objections: the restriction of agents to low-value decisions, making AI decisionmakers open source, adding a human-in-the-loop and the general limits of trust in human agents.

@histodons @politicalscience

Erste #Benchmarks zur Leistung der #Radeon #RX9070 (XT) in #Spielen sind durchgesickert. Die Zahlen stammen dabei wohl direkt von #AMD und versprechen eine deutliche Steigerung verglichen mit der #RX7900GRE. https://winfuture.de/news,149067.html?utm_source=Mastodon&utm_medium=ManualStatus&utm_campaign=SocialMedia

AI
xAI vs OpenAI: Benchmark Controversy

OpenAI says xAI misled the public by omitting key "cons@64" scores in Grok 3 benchmarks.

Grok 3 Reasoning Beta actually underperforms OpenAI’s o3-mini-high when measured at first attempt (@1).

Experts say AI benchmarks often lack transparency, omitting crucial cost and efficiency data.

#AI #xAI #OpenAI #Grok3 #Benchmarks

In ersten #Benchmarks können die kommenden #Radeon-#Grafikkarten von #AMD nicht überzeugen. Die #RX9070 XT bleibt deutlich hinter den Erwartungen zurück und bewegt sich auf dem Niveau einer #RTX4070 Super. https://winfuture.de/news,149037.html?utm_source=Mastodon&utm_medium=ManualStatus&utm_campaign=SocialMedia

Guess what? Linux faster than Windows increasingly | '25 Gaming benchmarks

Enjoy the show!

https://youtu.be/D45AknAsIPw

Avui toca aprendre a fer #benchmarks amb #jmh #java #programming