ChatGPT taught how to play Red Dead Redemption 2

A research team from China and Singapore recently presented a paper describing the concept of general computer control (GCC) for artificial intelligence. As part of their experiments, the scientists trained an AI model running on OpenAI GPT-4V to play Red Dead Redemption 2, a complex video game with a rich and varied virtual world. This project aims to explore the possibilities of creating advanced artificial general intelligence (AGI).

To interact with a computer and interpret visual and audio signals, a special system called CRADLE, including six modules, was developed. The game was chosen because it was challenging and action-packed, providing a rich learning environment. The AI ​​was trained to control the character using a mouse and keyboard, learning the game’s mechanics from scratch and facing various challenges such as navigating the world and completing missions.

During the course of the study, the AI ​​was able to successfully complete a number of objectives in the game, although some missions, such as Protect Dutch with intense firefights and Search House with the need to explore a complex environment, were beyond its limits. This project shows significant promise for AI in learning and adapting to complex tasks in dynamic and unpredictable environments.

Based on the results of the study, a weak point of the entire system was identified – this is the GPT-4V machine vision system. It turned out that this AI model cannot cope with processing all visual information. In particular, there is a problem with understanding the mini-maps, with the perception of obstacles in the game environment and other elements. So far, the capabilities of GPT-4V are insufficient for accurate spatial-visual recognition.