• Digital Humanities and Cultural Heritage
• Publications close
• National Institutes of Health close

Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions. In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research. This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation. Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development. Comment: This paper has been accepted by the Journal of Biomedical Informatics

Natural history collections are critical reservoirs of biodiversity information but collections staff are constantly grappling with substantial backlogs and limited resources. The task of transcribing specimen label text into searchable databases requires a significant amount of time, manual labor, and funding. To address this challenge, we introduce VoucherVision, a tool harnessing the capabilities of several Large Language Models (LLMs; Naveed et al. 2023) to augment specimen label transcription. The VoucherVision tool automates laborious components of the transcription process, leveraging an Optical Character Recognition (OCR) system and LLMs to convert unstructured label text into appropriate data formats compatible with database ingestion. VoucherVision uses a combination of structured output parsers and recursive re-prompting strategies to ensure consistency and quality of the LLM-formatted text, significantly reducing errors. Integration of VoucherVision with the University of Michigan Herbarium’s transcription workflow resulted in a significant reduction in per-image transcription time, suggesting significant potential advantages for collections workflows. VoucherVision offers promising strides towards efficient digitization, with curatorial staff playing critical roles in data quality assurance and process oversight. Emphasizing the importance of knowledge sharing, the University of Michigan Herbarium is backing the Specimen Label Transcription Project (SLTP), which will provide open access to benchmarking datasets, fine-tuned models, and validation tools to rank the performance of different methodologies, LLMs, and prompting strategies. In the rapidly evolving landscape of Artificial Intelligence (AI) development, we recognize the profound potential of diverse contributions and innovative methodologies to redefine and advance the transformation of curatorial practices, catalyzing an era of accelerated digitization in natural history collections. An early, public version of VoucherVision is available to try here: https://vouchervision.azurewebsites.net/

Objectives: Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets. Methods: We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data. Results: In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain. Conclusions: With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better. Advances in knowledge: Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.

AbstractProtein engineering is an emerging field in biotechnology that has the potential to revolutionize various areas, such as antibody design, drug discovery, food security, ecology, and more. However, the mutational space involved is too vast to be handled through experimental means alone. Leveraging accumulative protein databases, machine learning (ML) models, particularly those based on natural language processing (NLP), have considerably expedited protein engineering. Moreover, advances in topological data analysis (TDA) and artificial intelligence-based protein structure prediction, such as AlphaFold2, have made more powerful structure-based ML-assisted protein engineering strategies possible. This review aims to offer a comprehensive, systematic, and indispensable set of methodological components, including TDA and NLP, for protein engineering and to facilitate their future development.

Abstract Objective Identifying study-eligible patients within clinical databases is a critical step in clinical research. However, accurate query design typically requires extensive technical and biomedical expertise. We sought to create a system capable of generating data model-agnostic queries while also providing novel logical reasoning capabilities for complex clinical trial eligibility criteria. Materials and Methods The task of query creation from eligibility criteria requires solving several text-processing problems, including named entity recognition and relation extraction, sequence-to-sequence transformation, normalization, and reasoning. We incorporated hybrid deep learning and rule-based modules for these, as well as a knowledge base of the Unified Medical Language System (UMLS) and linked ontologies. To enable data-model agnostic query creation, we introduce a novel method for tagging database schema elements using UMLS concepts. To evaluate our system, called LeafAI, we compared the capability of LeafAI to a human database programmer to identify patients who had been enrolled in 8 clinical trials conducted at our institution. We measured performance by the number of actual enrolled patients matched by generated queries. Results LeafAI matched a mean 43% of enrolled patients with 27 225 eligible across 8 clinical trials, compared to 27% matched and 14 587 eligible in queries by a human database programmer. The human programmer spent 26 total hours crafting queries compared to several minutes by LeafAI. Conclusions Our work contributes a state-of-the-art data model-agnostic query generation system capable of conditional reasoning using a knowledge base. We demonstrate that LeafAI can rival an experienced human programmer in finding patients eligible for clinical trials.

PurposeEvaluate changes in early adolescent substance use from May 2020 to May 2021 during the coronavirus disease 2019 pandemic using data from a prospective nationwide cohort: the Adolescent Brain Cognitive Development Study.MethodsIn 2018-2019, 9,270 youth aged 11.5-13.0 completed a prepandemic assessment of past-month alcohol and drug use, then up to seven during-pandemic assessments between May 2020 and May 2021. We compared the prevalence of substance use among same-age youth across these eight timepoints.ResultsPandemic-related decreases in the past-month prevalence of alcohol use were detectable in May 2020, grew larger over time, and remained substantial in May 2021 (0.3% vs. 3.2% prepandemic, p <.001). Pandemic-related increases in inhalant use (p = .04) and prescription drug misuse (p < .001) were detectable in May 2020, shrunk over time, and were smaller but still detectable in May 2021(0.1%-0.2% vs. 0% pre-pandemic). Pandemic-related increases in nicotine use were detectable between May 2020 and March 2021 and no longer significantly different from prepandemic levels in May 2021 (0.5% vs. 0.2% prepandemic, p = .09). There was significant heterogeneity in pandemic-related change in substance use at some timepoints, with increased rates among youth identified as Black or Hispanic or in lower-income families versus stable or decreased rates among youth identified as White or in higher-income families.DiscussionAmong youth ages 11.5-13.0 years old, rates of alcohol use remained dramatically reduced in May 2021 relative to prepandemic and rates of prescription drug misuse and inhalant use remained modestly increased. Differences remained despite the partial restoration of prepandemic life, raising questions about whether youth who spent early adolescence under pandemic conditions may exhibit persistently different patterns of substance use.

ABSTRACT Objectives: The present study examines how different lifelong employment patterns are related to social relationships in old age, and whether there are gender differences in the impact of lifelong employment patterns. Designs and participants: The study was based on data collected among European adults as part of the Health, Aging and Retirement Survey in Europe (SHARE) and focuses on retired adults. Measurements: The study combines data on social relationships, collected in 2015, with retrospective data on employment history (number of jobs and years of employment) collected in 2017. Results: The findings show that adults who worked in more jobs had overall better structural characteristics of their later life networks – they had larger social networks and were more likely to include children and friends within those networks but less likely to include their spouse. On the other hand, working in more jobs was related to less emotional closeness with the network. These results varied between men and women; women who were involved in the labor market over their life had larger social networks and tended to include friends as confidants. Among men, working for more years was related to higher emotional closeness with the social network. Conclusions: The study may indicate a gendered pattern of social advantages and disadvantages to involvement in the labor market over the work course. Practitioners should consider the lifelong employment of adults to identify those who might be at risk of social isolation.