My position is that, at the end of the day, LLMs just shuffle zeros and ones to some very smart algorithms developed by human programmers on vast amounts of training data. Using various stochastic strategies such as gradient descent for next word prediction on neural networks with billions of parameters and trillions of tokens — these algorithms result in imperfect representations of the training data, even if the training data is accurate, real and unbiased. To ensure that these AI machines do not fabricate, hallucinate and descend into madness, they must be closely supervised by human experts. These AI machines provide little (if any) insight into the nature of human understanding of the world, and the nature of consciousness and intentionality required for that understanding. … An independent study from researchers at the Lassonde School of Engineering, York University, found major flaws in the SWE-benchmark, significantly lowering GPT4o’s actual performance (from 18.83% to 3.83% accuracy). Many of the AI machine fixes are described as suspicious, including “cheating” and tests so weak that even an incorrect solution can pass the tests. These Al machines are not good at solving software issues on the kind of work that you might give a junior programmer.
Phenomenal deep dive into how AI is not ‘reasoning’ as much as it is cleverly stacking subsequent evaluation tasks on top of each other. Also a note here on the effectiveness of SWE-Bench which tends to overstate real world performance.
I am a believe though, AI is generating great starting subroutines in the same way a lathe creates a great starting spindle.
Quote Citation: Jonathan, “LLMs: The Illusion of Thinking – JSO”, September 7, 2025, https://jso.eecs.yorku.ca/2025/09/07/llms-the-illusion-of-thinking/