AI-Based Metric for the Scientific Text Novelty

Abstract

The study introduces an information-theoretic approach for the quantitative assessment of unpredictability and, by extension, the originality of ideas. Two metrics are proposed, both of which are potentially useful for analyzing scientific or engineering texts.

The first metric, Semantic Gain, quantifies the amount of new knowledge contained in a text relative to the knowledge embedded in a baseline model. The second metric captures informational unexpectedness (or surprisal score), which is grounded in information entropy and measures how surprising a given text is to a well-trained language model.

Both metrics are applied to the abstracts of peer-reviewed engineering articles, and the results demonstrate that, despite being derived from different computational approaches and model architectures, the two metrics exhibit a strong linear correlation. This correlation may be interpreted as evidence of their mutual consistency and conceptual validity.