• Digital Humanities and Cultural Heritage
• Publications close
• National Health and Medical Researc... close
• figshare close
• arXiv.org e-Print Archive close

Relying on large pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) for encoding and adding a simple prediction layer has led to impressive performance in many clinical natural language processing (NLP) tasks. In this work, we present a novel extension to the Transformer architecture, by incorporating signature transform with the self-attention model. This architecture is added between embedding and prediction layers. Experiments on a new Swedish prescription data show the proposed architecture to be superior in two of the three information extraction tasks, comparing to baseline models. Finally, we evaluate two different embedding approaches between applying Multilingual BERT and translating the Swedish text to English then encode with a BERT model pretrained on clinical notes.

Additional file 1: Table S1. Definitions of systematic reviews which explicitly quoted specific organizations/checklists/criteria. A table that contains definitions of systematic reviews, extracted from analyzed data sources, in which the authors hav explicitly quoted specific organizations, or checklists or criteria. Table S2. All references that were used to support definition of systematic review, or inclusion/exclusion criteria that could be used as a proxy for a definition. A list of references that autohrs of analyzed information sources have used in their manuscript to support either a definition of systematic review, or inclusion criteria, or exclusion criteria that could be used as a proxy indicator of a systematic review definition

Additional file 2: Table S3. Definitions of a systematic review from nine manuscripts with the most commonly used combination of categories. A file containing verbatim extracted definitions that were used in nine manuscripts that had the most commonly used combination of categories of systematic review definitions