Researchers at Tufts University have presented an alternative way of measuring epiplexity (i.e., learnable, structural information excluding random noise) to optimize algorithms for representational discovery. The key idea is to minimize the spectral description length of a (neural network) encoder’s feature representations, as a loss function in and of itself, which is surprisingly differentiable and modestly efficient using PyTorch operations such as torch.linalg.solve_triangular() and torch.linalg.svdvals().
This learnable novelty loss function can also be understood as maximizing the log-determinant of the regularized Fisher information matrix, whereby over time training is inducing “intelligence” by concentrating more Fisher information across representational dimensions that yield the most geometric volume per parameter. Imagine this process as maximizing the size of a Fisher ellipsoid in information space (thanks, @Mohamad Yassine, for this helpful analogy - https://www.alphaxiv.org/abs/2607.18433). Thankfully, the source code accompanying this learnable novelty paper can be found on GitHub.
The idea of accelerating representation learning within generative models has received widespread attention in mainstream areas of machine learning (think REPA). However, this idea has recently begun surfacing in the realm of AI for science, in particular in molecular machine learning.
LENSEs is arguably one of the first compelling AI for science methods along this research direction. The idea is simple: take the embeddings of a pretrained molecular encoding model and teach a diffusion model to mimic these embeddings (and their respective inputs/outputs) while simultaneously denoising atomistic data. In doing so, one can jointly accelerate the diffusion model’s convergence speed and improve its sample quality. This idea will likely show up in other areas of AI for science in the near future.
What do you think of representation alignment and learnable novelty and their role in scientific machine learning? Is their application straightforward in most computational settings, or will they require significant customization to achieve strong results in relevant research domains?