Benchmarking university program curricula via alignment with industry needs using text analytics : a comparison between traditional text analytics methods and large language models
Dvorkin, Anton (2025)
Pro gradu -tutkielma
2025
School of Business and Management, Kauppatieteet
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe2025052654544
https://urn.fi/URN:NBN:fi-fe2025052654544
Tiivistelmä
Benchmarking is considered one of the most effective processes for assessment and improvement in academia. This thesis presents a method for benchmarking university curricula and individual courses based on their alignment with job market needs. Two approaches are taken: Natural Language Processing (NLP) techniques, including frequency-based methods, static embeddings, and contextual embeddings from Large Language Models (LLMs).
Two different job advertisement datasets are compared to university curricula to explore the feasibility of using small job ad samples (30-100 postings) instead of large ones (>1,000) for reliable inference. The results highlight the importance of selecting an appropriate embedding model, regardless of its benchmark test scores, and the necessity of validating its performance using ground truth datasets.
The framework developed is of practical relevance for university decision-makers providing them with a tool for independent course evaluations using large language models which can reduce biases in assessing curriculum alignment with job market needs. In the case study of Business Analytics programs at two Finnish universities, the tool identified 12 out of 32 courses as least aligned with market demands. Although the case study focuses on Business Analytics programs, the proposed approach can be transferred across academic disciplines.
Two different job advertisement datasets are compared to university curricula to explore the feasibility of using small job ad samples (30-100 postings) instead of large ones (>1,000) for reliable inference. The results highlight the importance of selecting an appropriate embedding model, regardless of its benchmark test scores, and the necessity of validating its performance using ground truth datasets.
The framework developed is of practical relevance for university decision-makers providing them with a tool for independent course evaluations using large language models which can reduce biases in assessing curriculum alignment with job market needs. In the case study of Business Analytics programs at two Finnish universities, the tool identified 12 out of 32 courses as least aligned with market demands. Although the case study focuses on Business Analytics programs, the proposed approach can be transferred across academic disciplines.