Comparative Analysis of AI Reasoning Architectures in Chess: O1 vs R1 Performance Evaluation

Muppa, Ravi Kishore (2025) Comparative Analysis of AI Reasoning Architectures in Chess: O1 vs R1 Performance Evaluation. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (472kB) | Preview

Abstract

This work shows the systematic analysis of specialized AI models' chess playing performance as a testbed using chess, and a comparison of the R1 model from DeepSeek and the O1 model from OpenAI. This work fills the gap in the quantitative comparison of various architectural styles on chess playing performance and move quality. Comprehensive CPL (Centipawn Loss) evaluation metric analyses of 120 games played from 30 strategically distinct chess startup positions and 2,400 moves were conducted. Our evaluation framework comprised direct model comparisons, analysis of self-play consistency, and performance comparisons on varying position types (tactical, strategic, and endgame positions). Our results show that O1's hybrid of supervised learning and reinforcement learning style results in significantly better performing chess playing than R1's pure reinforcement learning architecture, achieving 25.6 versus 87.0 average CPL (3.5 times better performing). O1 showed higher consistency (75% lower standard deviation), higher overall performance on all position types, and an 85% victory rate on one-vs-one competition. Statistical testing upheld these differences as strongly significant (p < 0.001). This research discovers that design decisions of architectures significantly influence move quality and chess playing ability where O1 consistently performed over a range of position complexity while R1 experienced severe performance loss on complex positions. From the experiment on this research, we have empirical proof that an integration of learning on master games and reinforcement learning results in an overall better outcome than pure self-play methods of building AI programs on chess-related domains.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Subhnil, Shubham UNSPECIFIED
Subjects:	Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence G Geography. Anthropology. Recreation > GV Recreation Leisure > Games and Amusements
Divisions:	School of Computing > Master of Science in Data Analytics
Depositing User:	Ciara O'Brien
Date Deposited:	01 Jul 2026 11:44
Last Modified:	01 Jul 2026 11:44
URI:	https://norma.ncirl.ie/id/eprint/9438

Actions (login required)

View Item