Literature Review & Framework

The Empowerment of Science of Science by Large Language Models

By Guoqiang Liang, Mengxuan Li, Zhihao Zhang, Gege Lin, Shuo Zhang, Zhihao Zhang • October 2025

TL;DR

"Traditional Science of Science (SciSci) relies heavily on structural metadata like citation links, ignoring the rich semantic content inside papers. This paper introduces a comprehensive blueprint for using Large Language Models (LLMs) to transform how we perceive, evaluate, and forecast scientific breakthroughs."

Science of Science (SciSci) AI4Science Knowledge Graphs AI Agents

Why This Matters

The field of Science of Science (SciSci) aims to quantitatively understand, map, and predict how scientific knowledge is created, structured, and disseminated. For decades, researchers relied on citation counts, co-citation networks, and basic word frequencies.

However, these methods suffer from a massive limitation: they treat papers as black boxes. A citation link tells us *that* paper A cited paper B, but it doesn't explain *why* (e.g., did they build upon it, or criticize its methodology?). Traditional natural language processing (NLP) tools couldn't grasp the deep semantic nuance hidden inside full-text PDFs.

By introducing LLMs into the SciSci workflow, we can move from simple structural citation tracking to rich, semantic-aware knowledge synthesis.

The Big Idea

The authors propose that LLMs should not just be used as writing assistants, but as core engines of scientific discovery. By combining LLMs with external tools, vector databases, and multi-agent systems, SciSci can transform across three key dimensions:

Scientific Perception

Extracting Subject-Action-Object (SAO) triplets to build semantic knowledge graphs directly from full-text PDFs.

Scientific Evaluation

Deploying LLM-based autonomous AI agents equipped with memory and tools to evaluate research impact and peer reviews.

Scientific Forecasting

Constructing multilayer networks to pinpoint emerging research fronts and predict upcoming technological convergences.

Interactive: The Evolution of SciSci Paradigms

Click on each era to see how data structures and analytical capabilities evolved over time.

Traditional Bibliometrics (1950s - 1990s)

Focused on basic statistics like citation counts, h-index, and journal impact factors. Relationships were modeled as simple static links without deeper context.

Key Data Source Science Citation Index (SCI), Web of Science

How It Works

The transition to LLM-empowered SciSci is driven by specific technical workflows. The paper breaks these down into five fundamental layers of user-facing technologies:

1. Prompt Engineering

Level 1

Guiding pre-trained models using structured inputs like Chain-of-Thought (CoT) or Tree-of-Thoughts (ToT) to extract structured knowledge without fine-tuning.

2. Retrieval-Augmented Generation

Level 2

Connecting LLMs to external academic databases (like OpenAlex, PubMed) to prevent hallucinations and ground answers in empirical scientific literature.

3. Fine-Tuning

Level 3

Adapting model weights using Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA/qLoRA on domain-specific corpora (e.g., medical papers or physics abstracts).

4. Tool Learning & Agents

Level 4

Equipping models with APIs, code execution tools, and memory structures to form autonomous agents capable of evaluating research and mapping trends.

The Mathematical Engine: Self-Attention

At the core of these workflows is the Transformer’s self-attention mechanism, which maps high-dimensional vector spaces to compute semantic relationships:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) \times V$$

Interactive: SAO Semantic Triplet Extractor

Live Demo

Traditional co-word analysis only links keywords. LLMs extract Subject-Action-Object (SAO) structures to capture how concepts interact. Select a scientific statement below to see how it is parsed into a semantic node graph.

Select a Paper Abstract Sentence:

Extracted SAO Triplet

Subject: Improved GNN

Action: enhances

Object: paper classification accuracy

*Note: LLMs extract these semantic triples from unstructured text, mapping them into a high-dimensional vector space.

LLM Benchmarks for SciSci Tasks

To construct complex multilayer networks for research front forecasting, the authors benchmarked several modern LLMs. They evaluated them across input/output cost, speed, context length, and topic relevance.

Interactive Model Ranker

Adjust weights to prioritize Cost vs. Speed for your SciSci pipeline.

DeepSeek-V3

GPT-4o

Cost Sensitivity 50%

Speed / Throughput 50%

Based on these empirical tests, the authors selected DeepSeek-V3 as the optimal model for multilayer network construction. It achieved comparable topic relevance to GPT-4o but at a tiny fraction of the API cost ($\approx \$0.12$ vs $\$8.50$ per million tokens).

Limitations & Open Questions

The Hallucination Barrier

LLMs can generate highly confident, yet completely fabricated citations. While RAG systems reduce this, absolute precision is still hard to guarantee in multi-hop scientific reasoning.

Bias & Metric Redesign

If LLMs are used to evaluate papers, their training biases could reinforce existing scientific hegemonies, favoring established research structures over highly disruptive, unconventional ideas.

Glossary

Science of Science (SciSci)

An interdisciplinary field that uses quantitative methods to study science itself—analyzing scientists, publications, journals, and policy to find patterns in discovery.

Subject-Action-Object (SAO) Triplet

A semantic structure extracted from text where a Subject performs an Action on an Object. This mapping reveals functional relationships rather than simple word associations.

Retrieval-Augmented Generation (RAG)

A technique that enhances LLM accuracy by retrieving relevant documents from an external database and feeding them directly into the prompt context.

BibTeX Citation

@article{liang2025empowerment,
  title={The Empowerment of Science of Science by Large Language Models: New Tools and Methods},
  author={Liang, Guoqiang and Li, Mengxuan and Zhang, Zhihao and Lin, Gege and Zhang, Shuo and Zhang, Zhihao},
  journal={ChinaXiv Preprint},
  volume={202510.00072},
  year={2025}
}