@JoeJ Why should there be a ceiling?
Being a programmer, i notice my personal ceiling every day. It's very close and i can't break through.
Genius people are maybe twice as smart.
If there are people twice as smart as you, why are game gfx so mid? I thought you were at the cutting edge of a new GI breakthrough
But why should there be a cap? If we achieve AGI, i assume making it's HW bigger also makes it smarter. It will also become smarter with time from experience and self optimization.
Intelligence is merely analyzing some data in order to find patterns. So for example with Einstein who is viewed as a stereotypical genius, basically what happened is, before the 1900s everyone believed in the Aether. Then Michelson and Morley performed an experiment which returned a null result for the Aether. Basically, this was a giant issue, because Science was extremely convinced that there must be Aether, so with a new experiment that cast doubt on the Aether, it would take a genius in order to explain how reality does not have Aether in it. So Einstein formulated a theory given the data he was given, and he became famous for that.
What they don't tell you is, Morley was a student of Michelson's (or was it the other way around), either way, the student guy wanted to do real science, but the teacher was more about just getting stuff published. And the teacher kept publishing fudged results from the experiment, the student complained about it but was ignored due to his low status.
So basically, Einstein created the best pattern he could from the data he was given. As he grew older, he began to question his own theories, even considering that the Aether must be true, a true sign of intelligence.
What does this have to do with Ai? Well it took Einstein 10 years to create his theory. What would you say if an Ai took only 5 years? The Ai is twice as intelligent as Einstein? Compared to the estimated timespan of the universe, what is 5 years?
So if an Ai created a convincing theory of reality in 10 seconds, and Einstein took 10 years, does that make the Ai is 31,563,0000 million times smarter than Einstein? What if one were to factor in that Ai has an advantage of speed of light calculations, while humans have signals of 100 meters per second? Then in terms of “internal time” you'd adjust for that. Or you could compare the amount of energy between the two? What if it took the amount of yearly power output of a star for the Ai to calculate that, how would you quantify how smart it is compared to a human? And what if the theory is good enough to convince all the humans, but its actually a false theory, like much of the stuff ChatGPT puts out?
Anyway, the point is, intelligence is pattern making, it is a way for an organism to navigate its environment and obtain resources efficiently. So its a bunch of parlor tricks, and there is a ceiling to how many parlor tricks you can do. For example, a rapid acceleration of Science in the 1700s, but nowadays most of the low hanging fruit is gone. So its harder to score a “Eureka” (the first Eureka was probably easy by comparison.) How many “Eurekas” can you score? Its like how good can guns get? The first primitive gun ever made was a “Eureka”, but if somebody makes a gun that fires bullets at 90% the speed of light, cool sure, but its just the same as anything else, another diminished returns, another gimmick, another parlor trick.
Its like lets say an Ai becomes omnipotent, can see every atom in the solar system simultaneously, and starts to discover some patterns scientists haven't discovered before. They will eventually run out of patterns to discover, and this is what I refer to as “the ceiling”. After the first discoveries have lost their novelty, the next discoveries may be increasingly mundane and mediocre, to be published in online journals most people won't read.
Likely, once an Ai realizes the ceiling, then it can no longer go up, but down. The Ai will choose to dumb itself down and create a hedonistic future. Isn't that what humans are already doing? Don't people play make-believe with movies and games? When you are watching a movie you realize the plotholes only after the movie. And there is an appeal to simple games that aren't overly complicated. And, once a campaign mode game is completed it loses its appeal. So after an Ai discovers everything what would be its motivations?