NVIDIA launches Cosmos 3 for robotics and autonomous systems

At the technology conference GTC Taipei, the NVIDIA company officially announced the large-scale launch of its innovative model of artificial intelligence Cosmos 3. The developers call the new product the world’s first fully open omnimodel capable of deep visual-oriented analysis of the surrounding world. The architectural features of the system allow it to process input data and output high-fidelity output in a variety of different modalities, such as standard text output, generated realistic images, simulated video clips, and synchronized real-world background sound.

Modern drones and robotic devices regularly face the problem of fragmentary understanding of surrounding objects due to insufficient databases or incomplete virtual modeling in conventional simulators. NVIDIA Cosmos 3 intelligent omnimodel is designed specifically as an effective brain for complex physical artificial intelligence (Physical AI). The new architecture is able to “natively” perceive the physical space it enters and generate movement commands based on a predicted simulation of the dynamic response of surrounding things.

Technical specifications

Cosmos 3 AI is based on a unique hybrid engineering solution: the system organically combines a transformer for cognitive reasoning and an expert transformer for direct audio-visual generation. The internal structure of Cosmos 3 works according to the following two-phase scenario:

• Evaluation of spatio-temporal relations: First, the omnimodel defragments complex physical laws of interaction, calculates mass, kinematics of movement and force vector;
• Expert Rendering: Having received an accurate simulation plan, the second transformer proceeds directly to processing a detailed video behavior trajectory and sets of control instructions.

The integration of similar modules allows you to speed up the execution of logical conclusions several times using parallel computing systems without the risk of losing the natural accuracy of the interaction.

For what and in what fields will the new neural network be useful

Technical engineers of NVIDIA distinguish three key conceptual scenarios of practical implementation:

• Visual-Linguistic Model (VLM): A universal cognitive system that thinks in all classic human modalities;
• Simulation world model (World Model): A virtual physical environment-polygon that simulates reality for running in autonomous devices within the safe limits of a virtual server;
• Fine Premarking Environment: A scalable reference solution for training more point or commercial intelligent models.

Varieties of the omnimodel

To create a variable working infrastructure, the company introduced the Cosmos 3 system at once in several separate technical editions:

• Cosmos 3 Super: Ultra-edition, aimed at the difficult stage of the final post-training of commercial autopilots and unmanned special equipment of an industrial scale. Guarantees the absolute reliability of each generated movement at the physical level;
• Cosmos 3 Nano: Efficient real-time processing chip. Implements ultra-fast response according to the principle of operation of the robot in small fractions of a second with a minimum of energy consumption;
• Cosmos 3 Edge: A promising version of the neural network, which is in the stage of pre-release readiness. It is designed to be deployed directly in local blocks of peripheral chips without the need to use external cloud platforms.