Final Assessment
Name: Vishaal Ranjan
Team: ML Team 8 (Bertinator)
Overview of Things Learned:
Technical Area:
-
I learned how to create web crawlers from scratch using Scrapy and BeautifulSoup
-
I also learnt about the API calls that are made in the websites that use infinite scrolling and how to scrape the data for such pages
-
Performed data cleaning and analysis on the scraped data from the forums
-
Concatenated the data from all the forums and used regular expressions to remove HTML tags and some unnecessary characters
-
Used the BERT model in order to predict the forum for any given post
-
Achieved an accuracy of 97% while through the BERT model while using data from 5 different forums
-
Learned how to deploy the model on AWS through the use of Lambda and Serverless
Tools Used: Python, Pandas, Scrapy, BeautifulSoup, BERT, AWS, Git, Colab, Visual Studio Code
Soft Skills: Learned how to Collaborate with teammates from different time zones, solved technical issues by interacting with the team and task leads and coming up with a solution, communicated my learnings effectively to the whole team
Achievement Highlights
- Successfully scraped the airline forum through Scrapy (which included 36000+ entries and 13 features)
- Successfully joined all the dataframes together after cleaning the data and handling the null values
- Researched and got a better understanding of the BERT model through which I was able to create the custom model for our usecase
- Made use of Slack and Github in order to collaborate and share code
- Learned about model deployment on AWS through the use of its services like Lambda and Serverless
List of Meetings attended
- Attended all the team meetings that have been organized so far (2 meetings/week)
- Git Webinar
- Scraping with BeautifulSoup and Scrapy Webinar
- Recommender Systems Webinar
- Weekly meetings with Technical Lead Maleeha