Our next meeting is with Anubhav. Please let us know whether Thursday at 6pm Pacific or Thursday at 7pm Pacific works better. We will go with whichever time slot has more votes by Wed noon Pacific:
Before joining this meeting, make sure you have followed the discussions from Week 1.
Summary from Week 1:
The majority of students voted to work on a project involving Machine Learning in the domain of Bioinformatics.
In the Bioinformatics focus meeting led by Anya, an interesting addition was proposed: incorporating network structure as a feature in the machine learning model (credit: @moneuron).
The Machine Learning focus meeting led by Sam covered the initial steps in detail. All details can be found in the post.
Github setup started by @Prasun_Sharma
A more detailed summary will be shared by end of day for those who need additional guidance. For the next meeting, all members should try and complete:
Please try out this example if you are new to ML or need a refresher: 🔸 Code Along for F-IE-1: Text Classification using NLP
The votes are exactly 50-50!
Launch Meeting - Zoom. Feel free to attend a portion of the meeting if you can’t stay for the full hour.
Data Acquisition and Initial Network Construction.
All members are expected to have completed this step. If you have any difficulties, please reply to this post and we will provide you help to catch up.
Foundational Machine Learning
This will be the main focus of this meeting.
Advanced Feature Implementation
Our next focus will be on incorporating more sophisticated network analysis techniques:
Commit Process: Based on the results shared on this forum, one team member will be designated to commit the main file. Other members can then pull this file into their branches, make additions, and merge their changes back into the main branch.
Contribution Details: Each member is responsible for clearly detailing their contributions in the commit message when committing code. It is entirely possible that subsequent commits may alter parts of the original commit.
Initial Commit: For the first piece of code, Ahmed and Aya will be invited to commit as they have successfully merged the two datasets. @Prasun_Sharma, please coordinate with @Ahmedz and @ayahashim16 to ensure their code is properly committed. Once this is done, please inform the rest of the team.
Approval Process: As leads, @Prasun_Sharma and @Moh_Saiger will be responsible for approving future commits. They will use the forum or chat channel to gather the team’s votes before finalizing any changes.
Next Steps: The upcoming code will focus on machine learning, specifically on binary classification (determining whether a link is present) and regression (predicting the probability of a link being present).
I’ve pushed the latest code to the GitHub BI-ML_Disease-Prediction_2024, specifically on the dev branch. I’ve included a step-by-step walkthrough notebook to help clarify any potential confusion. Please, take a note that the repo is private, but I hope everyone has access.
I apologize for missing the last meeting due to the time difference. Thanks to the @stemaway moderators and the team for the continuous updates and feedback
Awesome! Thanks Ahmed!
Main points from the ML meeting: Attendees, please reply with any other salient points you captured.
Visualization vs ML: Use a small STRINGdb dataset for the visualization. For the ML work, the entire Homo sapiens dataset needs to be utilized.
ML Data Balance: The main issue with the ML data is that it will be unbalanced, with significantly more negatives than positives (pointed out by @Thuraya_Ayman). This is a real challenge, so please research ways to tackle it. Addressing this in a team publication could be very valuable.
Modeling Approach: Start with logistic regression, then move to more complex models to avoid overfitting.
Threshold for Performance Metrics: The default threshold is 0.5. However, for medical data, it’s better to prioritize false positives over false negatives. Explore what this means in the context of your project—another potential topic for publication.
Multi-Classification: After each team member has selected and completed the classification for one disease, combine the data for a multi-classification problem. Anubhav will provide further guidance. This approach could yield interesting results.
Features for Initial Models: For the initial models, we are not using any features beyond the association scores. However, if the team is interested, network analysis could be conducted to add additional features. This would be a complex but very interesting study! Anya and Sam will provide further guidance.
Hi @Ahmedz,
The template repo is not to be modified. This is a repo to derive project repo from. Please reach out to your project lead to create a new project repo under your team on GitHub and push the code there. Using the formats shared to @Prasun_Sharma and @Moh_Saiger .
There’s also a readme on our mentor chains profile page. It has resources for leads but open for everyone on how to create a project repo from template and how to structure code. Let me know if you have any questions.
Hi @Samuel_bharti, Sorry for the inconvenience caused, that was on me.
No worries Ahmed! Keep pushing! Sam is there to help with questions and/or if any mistakes are made.
No worries Ahmed, here to guide you guys. I’ll fix the template.
This is a new setup of creating project repositories through a template. I’m in process of setting right permissions for everyone. We’ll have more documentation and resources for easy navigation on GitHub.
Dears, @Ahmedz and I are confused regarding building the classifier for level 1. As per our understanding, I tried to build a classifier for Lung cancer using the composed graph (which combines both the disease-gene graph and protein-protein graph ). I have assigned label 1 for each edge in the graph, and label 0 for non-edges (randomly sampled and have a size equal to the number of edges to balance the dataset), to predict whether a link exists or not. The association scores were the only feature. I tried this model using three different classifiers and two of them scored 100% (Logistic Regression scored 80.04%, Gradient Boosting scored 100%, and Random Forest scored 100%). It seems that the models overfit the training data. Can you please advise if my labelling strategy is correct or is it just because we are using one feature?
As @ayahashim16 mentioned, we performed link prediction by combining two graphs: one representing disease-gene interaction and another representing protein-protein interaction. We followed the code snippet that labels existing edges as positive samples (1) and randomly samples non-edges for negative samples (0). This approach ensures that positive and negative samples are of equal length by using random.sample(non_edges, len(edges)), resulting in a balanced dataset
I used a subset of the STRING database that matches the length of Alzheimer’s disease dataset (6000 rows). After creating and composing the graphs, I split the dataset into training, test, and validation sets. I then applied different classifiers, and the metrics evaluated on the validation set are provided below
I used the full STRING database to construct a PPI graph and combined it with the Alzheimer’s disease graph. The resulting graph was approximately 900MB in size, contained 19801 Nodes. For visualization purposes, I extracted a 1000 nodes sample and displayed it. However, for training, testing and validation, I utilized the entire graph using logistic regression
The notebook has also been pushed to GitHub on my branch, ahmed
@ayahashim16 and @Ahmedz, we have sent your query to Anubhav, we will get back to you shortly.
Hi @ayahashim16 @Ahmedz, can you try running with a bigger set of ppi edges. This is a link to the complete home sapien String-DB PPI from a previous intern: ppi.csv - Google Drive. Add code to discrad rows that may have NAN or empty values. We arer also trying out the code ourselves, will update you shortly!
Hi @ayahashim16 @Ahmedz, we believe the issue is with how the positive and negative edges are created. In the meetings, we talked about how disease to gene from OpenTarget are positive edges and disease to other genes from stringDB are negative edges (ie no known association with that disease to other genes).
However, the code snippet we share doesn’t show this distinction. It does the sampling on the full graph.
You will need to add code to create correct edges:
If you were already constraining edges correctly:
Sorry about any confusion this may have caused. Please share this information with your entire team. And let us know if you have any questions.
Dear, Thanks for your response. I have tried the models for the Lung Cancer disease, and considered the labels as follows based on your recommendation:
I only used the association score as a feature and I have set it to zero when the gene has no association with the disease. But unfortunately, I have faced the same issue and the model overfits the training data. I think the model learns to assign a negative label when the association score is equal to zero and to assign a positive label when the association score is greater than zero. Please advise.
Hi @ayahashim16 You analysis looks correct, can you remove the association score and use network features instead. There were some suggestions in the original code snippets, and you could also try something like this:
node2vec = Node2Vec(G, dimensions=64, walk_length=30, num_walks=200)
model = node2vec.fit(window=10, min_count=1)
def get_edge_features(edge):
return model.wv[edge[0]] * model.wv[edge[1]]
X = [get_edge_features(edge) for edge in positive_edges + negative_edges]
y = [1] * len(positive_edges) + [0] * len(negative_edges)
Let me also ping Anya/Sam to see if we can add biological features.
On emore thing: Apply stronger regularization in your model to prevent overfitting. For example, in logistic regression:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(C=0.01, penalty='l2', solver='liblinear')
model.fit(X_train, y_train)
Let me know what you find!
Firstly, a big thank you to @ayahashim16 and @Ahmedz for sharing their findings.
def simple_node_embedding(G, dim=64):
embeddings = {}
for node in G.nodes():
degree = G.degree(node)
neighbor_degrees = [G.degree(n) for n in G.neighbors(node)]
avg_neighbor_degree = np.mean(neighbor_degrees) if neighbor_degrees else 0
embedding = np.zeros(dim)
embedding[0] = degree
embedding[1] = avg_neighbor_degree
embedding[2:] = np.random.randn(dim-2)
embeddings[node] = embedding / np.linalg.norm(embedding)
return embeddings
# Cap the number of PPI interactions
if len(ppi_ot) > 100000:
ppi_filtered = ppi_ot.sample(n=100000, random_state=42)
else:
additional_needed = 10000 - len(ppi_ot)
ppi_non_ot = ppi_df[~((ppi_df['GeneName1'].isin(ot_genes)) | (ppi_df['GeneName2'].isin(ot_genes)))]
additional_sample = ppi_non_ot.sample(n=min(additional_needed, len(ppi_non_ot)), random_state=42)
ppi_filtered = pd.concat([ppi_ot, additional_sample])
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
n_estimators=50, # Fewer trees
max_depth=10, # Limit depth
min_samples_split=10, # Larger splits
min_samples_leaf=5, # Larger leaf nodes
random_state=42
)
rf.fit(X_train, y_train)
Use separate tests sets and cross-validation to further help prevent overfitting: (Cross-validation helps assess how well your model generalizes across different subsets of the data)
X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
def cross_validate_and_evaluate(model, X_train_val, y_train_val, X_test, y_test, model_name, cv=5): # Perform cross-validation cv_scores = cross_val_score(model, X_train_val, y_train_val, cv=cv, scoring=‘f1’)
print(f"\n{model_name} Cross-Validation Results:") print(f"Mean F1-score: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})") # Train on full training set and evaluate on test set model.fit(X_train_val, y_train_val) y_pred = model.predict(X_test)
Play around with model parameters and amount of data and see what you can get!!
And this is a rough analysis: