Measure of Intelligence
How smart is AI today?
Not very, argues Francois Chollet in his 2020 paper, On the Measure of Intelligence. Chollet is the creator of Keras, one of the most popular programs for neural networks and deep learning, and his work in these cutting-edge technologies has given him a unique perspective on AI. Artificial Intelligence research lies at the intersection of two of my favorite subjects- analyzing lots of data and anthropomorphizing my computer. When one of the leaders of the field released a paper sharing his thoughts, then, I greedily devoured all of its 64 pages of fury, and they left me fascinated, excited, and maybe just a little bit confused. Let me see if I can summarize:
Chollet argues that they are extremely skilled in complex tasks- they can drive cars, make puns, and win international e-sports tournaments- but there’s a difference between skill and intelligence. Turns out, learning a skill- even a complex one- does not actually take any intelligence. As it turns out, an AI can simply learn the shortcuts that best accomplish the skill without ever developing any intelligence. A great example of this is Google’s AI “Five,” which learned to play DotA2 well enough to compete against world champions, but tanked when it played against beginners. It’s like when a student learns to eliminate answers on a multiple choice test by just comparing them to the other answers. They never learned the material, they just became clever.
AI typically find these shortcuts by either being pre-loaded with them (“priors”) or by learning from extensive training which actions to take in any situation (“overfitting”). It’s hard to think of this as “intelligence,” though; it’s more like “doing exactly what they’ve been trained to do, like any monkey can.” For an agent to be truly intelligent, it should be able to generalize past experiences to new and different situations that it has been previously taught nothing about. Moreover, it should be able to do this for a variety of tasks.
Chollet argues that any agent has a finite range of intellectual tasks it can possibly learn- a “scope.” For example, a human’s scope contains both “predicting the path of a thrown baseball” and “leveling a picture frame,” but not “counting the number of seeds on a strawberry at a glance.” Within its scope, an agent can build a skill out of a “curriculum” (a “training set,” in academic terms). The skill created would, itself, depend on the curriculum. A broader curriculum would create a skill applicable to a broader set of situations, and a curriculum for a different skill would create a completely different skill.
The agent’s intelligence, then, comes from being able to handle situations that were outside of its curriculum. It does not matter how much knowledge it has. It only matters how much knowledge it can generalize. An agent that can handle more situations with the same curriculum, or, similarly, the same situations with a smaller curriculum, is more intelligent.
This leads us to Chollet’s Measure of Intelligence: “Intelligence is the rate at which a learner turns its experience and priors into new skills that involve uncertainty.”
Intelligence = skill generalization
(priors + experience)
Chollet takes this proposition one step further by creating a corresponding intelligence exam. His Abstraction and Reasoning Corpus (ARC) is made up of a variety of tasks. Each task consists of a handful of practice examples, followed by a graded question. To understand how it works consider these tasks from it:
As you can see, the tasks aren’t hard for an intelligent agent (#humblebrag), but the variety of them and the sparseness of examples means that you can’t just learn their shortcuts. You have to generalize a small amount of experience to be able to answer each one.
Altogether, ARC gives programmers a way to measure their program’s intelligence, rather than merely its crystallized skill. That’s important- programmers make the most progress when they have a metric to optimize. In fact, they’re doing it right now! The ARC dataset is currently available on Kaggle, and hundreds of teams have already tried their hand at making an AI capable of solving it. With this improved Measure of Intelligence, Chollet has created a new and exciting idea of Artificial Intelligence, and I, personally, can’t wait to see how others manage to realize it!
Works Cited:
Comments
Post a Comment