🔴 Clinical Named Entity Recognition Using Natural Language Processing

AIVIA Project Engine AI in HealthCare & Medicine
, , , , , , ,
1

Clinical Named Entity Recognition Using Natural Language Processing

Objective

The primary objective of this project is to develop a Natural Language Processing (NLP) model that can automatically extract and classify clinical entities (e.g., diseases, medications, symptoms) from unstructured medical texts such as electronic health records (EHRs) or clinical notes. By implementing Named Entity Recognition (NER) techniques, you will build a model that aids in structuring clinical data, which is crucial for patient care, research, and healthcare analytics.

This project will utilize the NCBI Disease Corpus, a publicly available dataset, and leverage the Hugging Face Datasets and Transformers libraries to simplify data acquisition and model development.


Learning Outcomes

By completing this project, you will:

  • Understand the challenges of processing unstructured clinical text:

    • Grasp the complexity of medical language and terminologies.
    • Learn about privacy considerations and de-identification.

  • Gain proficiency in NLP techniques for NER:

    • Implement models using pre-trained transformers like BERT.
    • Use libraries and frameworks suitable for clinical NLP, such as Hugging Face Datasets and Transformers.

  • Develop skills in data annotation and preprocessing:

    • Handle clinical text data and prepare it for model training.
    • Understand the importance of annotated corpora in supervised learning.

  • Evaluate and interpret NER models:

    • Use appropriate evaluation metrics for sequence labeling tasks.
    • Analyze model performance and errors.

Prerequisites and Theoretical Foundations

1. Intermediate Knowledge of Python Programming

  • Text Processing: String manipulation, regular expressions.
  • NLP Libraries: NLTK, spaCy, Hugging Face Transformers.
Click to view Python code examples 
# Importing libraries
import re
import nltk

# Text processing
text = "Patient presents with chest pain and shortness of breath."
tokens = nltk.word_tokenize(text)

2. Understanding of NLP Concepts

  • Tokenization and Part-of-Speech Tagging:
    • Breaking text into words or tokens.
  • Named Entity Recognition:
    • Identifying and classifying entities in text.
  • Sequence Labeling:
    • Assigning labels to each token in a sequence.
Click to view NLP concepts
  • NER Models: Can be rule-based, statistical (e.g., CRF), or neural network-based (e.g., BiLSTM, Transformers).
  • Evaluation Metrics: Precision, recall, F1-score at the entity level.

3. Basics of Clinical Terminology

  • Medical Ontologies:
    • SNOMED CT, ICD-10, UMLS.
  • Common Clinical Entities:
    • Diseases, symptoms, medications, procedures.
Click to view clinical terminology concepts
  • Entity Types: Categories of interest in clinical text (e.g., ‘Disease’, ‘Medication’).
  • Terminologies and Codes: Standardized vocabularies for interoperability.

Skills Gained

  • Natural Language Processing

    • Implementing Named Entity Recognition (NER) for clinical text
    • Understanding clinical language processing challenges
    • Handling tokenization and alignment for transformer models
    • Converting data into BIO tagging format
    • Working with modern NLP frameworks (Hugging Face)

  • Supervised Learning

    • Working with labeled sequence data
    • Using pre-trained models and transfer learning
    • Fine-tuning transformer models on clinical data
    • Managing training process with Hugging Face Transformers
    • Handling data preprocessing and transformation

  • Model Analysis & Evaluation

    • Implementing sequence labeling metrics
    • Performing systematic error analysis
    • Interpreting model performance
    • Improving model based on analysis
    • Understanding model limitations

Tools Required

  • Programming Language: Python (version 3.6 or higher)
  • NLP Libraries:
    • Hugging Face Datasets: Simplifies dataset loading (pip install datasets)
    • Hugging Face Transformers: Pre-trained models and tokenizers (pip install transformers)
    • seqeval: Evaluation metrics for sequence labeling (pip install seqeval)
  • Deep Learning Framework:
    • PyTorch (used by default in Transformers)
  • Dataset:
    • NCBI Disease Corpus: Available via Hugging Face Datasets library.
  • System Requirements:
    • Minimum: 8GB RAM, CPU only (longer training time)
    • Recommended: 16GB RAM, GPU with 4GB+ VRAM (faster training)
    • Storage: ~2GB for models and data
    • Use small batch sizes for systems with limited RAM and/or no GPU

Steps and Tasks

Step 1: Data Acquisition

Tasks:

  • Obtain the Dataset:
    • Use the Hugging Face Datasets library to download and load the NCBI Disease Corpus.

Implementation:

# Install necessary libraries
!pip install datasets transformers seqeval

# Import libraries
from datasets import load_dataset

# Load the NCBI Disease Corpus
dataset = load_dataset('ncbi_disease')
Explanation
  • Hugging Face Datasets:
    • Provides a simple interface to download and prepare datasets.
  • NCBI Disease Corpus:
    • A dataset annotated for disease mentions, suitable for NER tasks.
  • Dataset Structure:
    • The dataset is already split into ‘train’, ‘validation’, and ‘test’ sets.

Step 2: Data Exploration and Preprocessing

Tasks:

  • Explore the Data:

    • Understand the structure of the dataset.
    • Check the distribution of entity labels.

  • Prepare the Data for Tokenization:

    • Understand how to align labels with tokenized inputs.

Implementation:

# View dataset splits
print(dataset)

# Access the training set
train_dataset = dataset['train']

# View an example
print(train_dataset[0])

# Check label names
label_list = train_dataset.features['ner_tags'].feature.names
print("Label List:", label_list)

# Number of labels
num_labels = len(label_list)
Explanation
  • Each example in the dataset contains:
    • id: A unique identifier.
    • tokens: A list of tokens (words).
    • ner_tags: A list of labels corresponding to each token.
  • Label List:
    • Contains the labels used for NER, e.g., ‘O’, ‘B-Disease’, ‘I-Disease’.

Step 3: Data Preprocessing and Tokenization

Tasks:

  • Tokenize the Data:

    • Use a tokenizer compatible with your chosen pre-trained model.
    • Handle subword tokenization and label alignment.

  • Align Labels with Tokenized Inputs:

    • Ensure that the labels correspond correctly to the tokenized tokens.

Implementation:

from transformers import AutoTokenizer

# Load a pre-trained tokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

# Function to tokenize and align labels
def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(
        examples['tokens'],
        is_split_into_words=True,
        truncation=True,
        padding='max_length',
        max_length=128
    )
    labels = []
    for i, label in enumerate(examples['ner_tags']):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:
            if word_idx is None:
                label_ids.append(-100)  # Special tokens
            elif word_idx != previous_word_idx:
                label_ids.append(label[word_idx])
            else:
                label_ids.append(label[word_idx] if label[word_idx] != 0 else 0)
            previous_word_idx = word_idx
        labels.append(label_ids)
    tokenized_inputs['labels'] = labels
    return tokenized_inputs

# Apply the function to the dataset
tokenized_datasets = dataset.map(tokenize_and_align_labels, batched=True)
Explanation
  • Subword Tokenization:
    • Tokenizers like BERT’s may split words into subwords; label alignment must account for this.
  • Label Alignment:
    • Assign labels to subword tokens, handling special tokens appropriately.
  • Padding and Truncation:
    • Ensures all sequences are of equal length for batch processing.

Step 4: Model Development

Tasks:

  • Choose a Pre-trained Model:

    • Use ‘bert-base-cased’ or a domain-specific model like ‘dmis-lab/biobert-base-cased-v1.1’.

  • Set Up the Model for Token Classification:

    • Initialize a BertForTokenClassification model with the appropriate number of labels.

Implementation:

from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

# Load a pre-trained model for token classification
model = AutoModelForTokenClassification.from_pretrained('bert-base-cased', num_labels=num_labels)

# Define training arguments
training_args = TrainingArguments(
    output_dir='./results',
    evaluation_strategy='epoch',
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
)

# Define data collator
from transformers import DataCollatorForTokenClassification

data_collator = DataCollatorForTokenClassification(tokenizer)
Explanation
  • AutoModelForTokenClassification:
    • Automatically configures the model for token classification tasks.
  • TrainingArguments:
    • Specifies training parameters like learning rate, batch size, and number of epochs.

Step 5: Model Training

Tasks:

  • Initialize the Trainer:

    • Use the Hugging Face Trainer class.

  • Train the Model:

    • Start the training process and monitor progress.

Implementation:

import numpy as np
from datasets import load_metric

# Load evaluation metric
metric = load_metric('seqeval')

# Define compute_metrics function
def compute_metrics(p):
    predictions, labels = p
    predictions = np.argmax(predictions, axis=2)

    true_labels = [
        [label_list[l] for l in label if l != -100]
        for label in labels
    ]
    true_predictions = [
        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
        for prediction, label in zip(predictions, labels)
    ]
    results = metric.compute(predictions=true_predictions, references=true_labels)
    return {
        'precision': results['overall_precision'],
        'recall': results['overall_recall'],
        'f1': results['overall_f1'],
        'accuracy': results['overall_accuracy'],
    }

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['validation'],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

# Train the model
trainer.train()
Explanation
  • Trainer Class:
    • Simplifies training by handling the training loop, evaluation, and logging.
  • compute_metrics:
    • Custom function to compute evaluation metrics using seqeval.

Step 6: Model Evaluation

Tasks:

  • Evaluate the Model on the Test Set:
    • Use the evaluate method to compute metrics on the test data.

Implementation:

# Evaluate on the test set
test_results = trainer.evaluate(tokenized_datasets['test'])

# Print evaluation results
print(test_results)
Explanation
  • Evaluation Metrics:
    • Includes precision, recall, F1-score, and accuracy.
  • Understanding Results:
    • Higher scores indicate better model performance.

Step 7: Error Analysis and Model Improvement

Tasks:

  • Analyze Misclassifications:

    • Identify where the model is making errors.
    • Look for patterns or common issues.

  • Refine the Model:

    • Consider using a domain-specific model like BioBERT.
    • Adjust training parameters or preprocessing steps.

Implementation:

# Get predictions
predictions, labels, _ = trainer.predict(tokenized_datasets['test'])
predictions = np.argmax(predictions, axis=2)

# Extract true labels and predictions
true_labels = [
    [label_list[l] for l in label if l != -100]
    for label in labels
]
true_predictions = [
    [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
    for prediction, label in zip(predictions, labels)
]

# Analyze errors
for i in range(len(true_predictions)):
    for j in range(len(true_predictions[i])):
        if true_predictions[i][j] != true_labels[i][j]:
            token = tokenized_datasets['test']['tokens'][i][j]
            print(f"Token: {token}, True Label: {true_labels[i][j]}, Predicted Label: {true_predictions[i][j]}")
Explanation
  • Error Analysis:
    • Helps identify specific tokens or types of entities where the model struggles.
  • Model Refinement:
    • Using BioBERT can improve performance due to pre-training on biomedical text.

Using BioBERT:

# Load BioBERT model and tokenizer
model_name = 'dmis-lab/biobert-base-cased-v1.1'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name, num_labels=num_labels)

# Re-tokenize the datasets
tokenized_datasets = dataset.map(tokenize_and_align_labels, batched=True)

# Re-initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['validation'],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

# Train the model with BioBERT
trainer.train()

Step 8: Visualization and Interpretation

Tasks:

  • Visualize Training Metrics:

    • Plot loss curves and evaluation metrics over epochs.

  • Interpret Model Predictions:

    • Examine specific examples to understand model behavior.

Implementation:

import matplotlib.pyplot as plt

# Access training logs
logs = trainer.state.log_history

# Extract loss values
train_loss = [log['loss'] for log in logs if 'loss' in log]
eval_loss = [log['eval_loss'] for log in logs if 'eval_loss' in log]
epochs = range(1, len(train_loss) + 1)

# Plot training and evaluation loss
plt.figure(figsize=(8, 6))
plt.plot(epochs, train_loss, label='Training Loss')
plt.plot(epochs, eval_loss, label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss Over Time')
plt.legend()
plt.show()
Explanation
  • Visualization:
    • Helps in diagnosing issues like overfitting.
  • Interpretation:
    • Provides insights into model learning dynamics.

Step 9: Conclusion and Future Work

Tasks:

  • Summarize the Project:

    • Reflect on the model’s performance and challenges faced.

  • Discuss Ethical Considerations:

    • Address privacy, bias, and deployment concerns.

  • Suggest Improvements:

    • Propose ways to enhance the model further.

Implementation:

  • Document in a Report:
    • Include methodology, results, interpretations, and recommendations.
Example Discussion
  • Model Performance:
    • The model achieved an F1-score of X% using BERT and Y% using BioBERT.
  • Challenges:
    • Handling rare disease entities and overlapping annotations.
  • Ethical Considerations:
    • Ensuring data is de-identified and compliant with regulations.
  • Future Work:
    • Expand entity types, incorporate relation extraction, and explore data augmentation techniques.

Conclusion

In this project, you have:

  • Developed NLP models for extracting clinical entities from unstructured text.
  • Gained experience in data acquisition and preprocessing using Hugging Face Datasets.
  • Implemented both baseline and advanced models, including Transformer-based architectures.
  • Evaluated and interpreted model performance using appropriate metrics.
  • Considered ethical implications in handling sensitive clinical data.

This project showcases the application of NLP in healthcare to improve data accessibility and support clinical decision-making. It highlights the importance of combining technical skills with ethical considerations in healthcare AI projects.

Next Steps:

  • Expand the Entity Types:

    • Include more categories like medications, procedures, or lab tests.

  • Integrate with Downstream Applications:

    • Use extracted entities for tasks like relation extraction or clinical outcome prediction.

  • Collaborate with Healthcare Professionals:

    • Validate model outputs and gather feedback for improvements.

Resources and Learning Materials

  1. Hugging Face Datasets: https://huggingface.co/docs/datasets/index
  2. Hugging Face Transformers: https://huggingface.co/transformers/
  3. NCBI Disease Corpus: https://huggingface.co/datasets/ncbi_disease
  4. Seqeval Library: https://github.com/chakki-works/seqeval
  5. BioBERT Model: https://github.com/dmis-lab/biobert
  6. PyTorch Documentation: https://pytorch.org/docs/stable/index.html

Note: Always ensure compliance with data privacy laws and regulations when working with clinical data. The NCBI Disease Corpus is derived from PubMed abstracts and does not contain protected health information, making it suitable for educational and research purposes.