The AI startup xAI, led by billionaire Elon Musk, continues to expand its ambitious AI projects, currently working on “real-world models,” a new generation of AI systems designed to understand and interact with the physical world rather than just analyzing text and images.

This project places Musk’s company in direct competition with tech giants like Meta and Google, which are also striving to develop systems capable of learning from the real world and building new worlds that simulate it. Although the idea resembles the concept of Artificial General Intelligence (AGI), the Financial Times report did not explicitly mention the term.

According to the report, xAI has recruited top researchers from Nvidia in recent months to support its efforts in building these models. “Real-world models” rely on training AI using videos and data from robots to understand real-world dynamics such as motion, physics, and cause and effect—skills that current text-based models like ChatGPT or Grok do not possess.

Unlike popular AI tools today that predict words or pixels, “real-world models” aim to understand the behavior of objects in three-dimensional environments, such as how a ball bounces, how light reflects, or how a robot moves inside a crowded room. These models intend to give AI a physical intuition closer to that of humans.

The newspaper quoted two informed sources saying that xAI is currently developing models that can be used in video games, enabling AI to create fully interactive and immersive 3D environments. One source indicated that the same technology could be used in the future for robots to teach them how to understand and design real spaces.

To accelerate this direction, xAI hired former Nvidia researchers Zeeshan Patel and Ethan He, specialists in building real-world models. Nvidia was among the first companies in this field through its Omniverse platform, which allows developers to simulate highly realistic digital worlds.

Elon Musk confirmed via the X platform (formerly Twitter) that his company plans to launch an “amazing AI-produced video game” before the end of next year. Musk hinted at the project last year, and recent hires indicate that work on the game has already entered an advanced stage.

This week, xAI also released its latest image and video generation model with “massive updates,” making it freely available to users. The company is currently expanding its specialized team called the “Omni Team,” tasked with creating “magical AI experiences beyond text,” including images, video, and audio. Salaries for this team range from $180,000 to $440,000 annually, while a job ad for a “video game trainer” offers $45 to $100 per hour to train the Grok system on game design.

Nvidia believes the real-world models market could be worth as much as the entire global economy, given its potential applications in robotics, industrial automation, and autonomous vehicles.

However, despite the great enthusiasm, the challenges remain enormous; training these systems requires massive amounts of data and huge computing power, in addition to precise modeling of physics laws and human behavior—something even the most advanced labs have yet to achieve.

On the other hand, gaming industry experts express skepticism about AI’s ability to solve the sector’s problems. Michael Dos, Head of Publishing at Larian Studios, developer of Baldur’s Gate 3, said, “AI will not solve the biggest problem in games, which is leadership and vision,” adding that the industry does not need “mathematically generated or psychologically trained gameplay loops,” but worlds that express human passion and attract real interaction.

With this project, Elon Musk bets that “real-world models” will be the boldest step to integrate AI with both the real and digital worlds. If successful, this vision could radically change how humans and machines interact with reality; if it fails, it will remain an exciting new chapter in Musk’s future scientific experiments.