GenCeption from Google DeepMind recently proposed a new approach to building foundation models for visual data. The key idea is to take a pretrained diffusion (generative) model, freeze its autoencoder weights (if it’s a latent diffusion model), and fine-tune the diffusion model’s Diffusion Transformer (DiT) architecture for both dense and sparse visual representation tasks using learnable task and video tokens. Importantly, dense downstream tasks can reuse the model’s frozen decoder weights (if present), treating new representation learning/prediction tasks as a visual reconstruction objective.
This approach currently provides both high performance and wide coverage of tasks spanning computer vision, suggesting that, though a denoising objective, its pretrained DiT architecture has learned highly informative and broadly applicable representations of continuous (visual) data. This suggests that computer vision, perhaps along with other (non-discrete) data domains, is beginning to discover the axes of neural scaling that are most effective beyond the next-token prediction paradigm prevalent in natural language processing (NLP). Scientific applications based on this idea are likely to follow suit, though the nuances of what it takes to train such an informative generative model in different scientific domains will almost certainly require extra consideration.
Can large language models (LLMs) deduce the physical laws of novel simulated worlds through text-based interaction alone? If so, would this convince us that there is something inherently informative that LLMs are learning beyond distributions for next-token prediction? With the new DiscoverPhysics benchmark from Princeton, we can begin to (partially) address these questions.
DiscoverPhysics has a convenient leaderboard to track community progress on this task of uncovering novel physical laws, along with open-source code accompanying the benchmark. Interestingly, with the release of Anthropic’s Claude 5 Fable model, the leaderboard appears to have a resolute leader in terms of per-world accuracy as well as explainability. Does this suggest that larger models are simply more likely to find better (and more interpretable) solutions to physical phenomena than smaller models (re: epiplexity)?
What are the implications of neural scaling for different scientific applications? Do we simply need larger models pretrained on whatever data, be it text, images, or the like? Or do we first need to find the right architecture to scale (as for DeepMind’s GenCeption model)?