Doktorandské kolokvium KAI - Jozef Kubík (20.11.2023)

v pondelok 20.11.2023 o 13:10 hod. v miestnosti I/9

---

Prednášajúci: Jozef Kubík

Názov: Efficient fine-tuning of Bert models in Slovak language

Termín: 20.11.2023, 13:10 hod., I/9

Abstrakt:
The popularity of creating large language models has been incredibly rising. Most modern LLMs offer great accuracy in many different text-based tasks but are limited by a huge number of data required to not only pre-train but also fine-tune these models. This problem only deepens in models trained on data from low and mid-resource languages, such as Slovak. In our work, we examine the fine-tuning process of two such models and try to enhance it by connecting to Epinet network to create Epistemic neural network, a relatively new concept that helps the model to detect its own uncertainty to make better decisions in the long run. In the presentation, we will show not only results based on the classic fine-tuning of such novel network but also try to use methods of Active learning to lower the data requirements while retaining similar results.

 

 

---