🟢 WebScrapper DataDive

AIVIA Project Engine Software Engineering
, , , , ,
1

DataDive: Web Scraping and Exploratory Data Analysis


Objective

In this project, you will journey into the world of data collection and analysis, beginning with web scraping, progressing into data processing, and culminating in exploratory data analysis (EDA). Your playground is the real-world dataset from the website Startup Jobs. The mission: to collect, clean, analyze, and visualize this data, transforming it from raw, unstructured information into insightful knowledge. You will utilize this data to generate conclusions that demystify its significance to the everyday individual.


Learning Outcome

Upon successful completion of this project, you will:

  • Acquire proficiency in Python programming.
  • Understand the ethics and methodologies of web scraping and data collection.
  • Gain hands-on experience in using robust web scraping tools such as Selenium and BeautifulSoup.
  • Become conversant with data science libraries like Pandas.
  • Develop your skills in data cleaning, text processing, and EDA.
  • Learn how to utilize visualization libraries for data representation.
  • Understand and apply statistical methodologies like Term Frequency-Inverse Document Frequency (TF-IDF).
  • Gain a foundational understanding of AI models, including their capabilities, limitations, and ethical considerations.

Steps and Tasks

1. Python Basics

Before diving in, it’s important to have a solid foundation in Python. If you’re just starting out, get up to speed with a fast-paced resource like Codecademy’s Python Course.

2. Introduction to Web Scraping

Kickstart your journey by understanding the ethics of web scraping and the methodologies for data collection. Learn how to use Selenium and BeautifulSoup with the official resources like BeautifulSoup tutorial and Selenium Documentation.

Looking for a more tutorial based approach to learning these libraries? try watching some of this BeautifulSoup tutorial and this Selenium tutorial

To apply your new found knowledge, lets try scraping data from Startup Jobs. Using this site, we can scrape information about hundreds of job postings and place them in a CSV file in a organized manner. Every job posting will be represented as a single row, with the each column in this row delimiting an attribute. When gathering this data, be sure to stick to a specific theme of jobs (Ex. Machine Learning), it can help narrow down analysis later.

Some key attributes to scrape:

  • The job title
  • The company hiring
  • Job type (Part time, full time, etc)
  • Job location
  • Associated tags with the job
  • The job description

Since startup jobs does have a dynamic aspect to it, you can combine the in-depth search functionality of BeautifulSoup with the web interaction ability of Selenium. In this case, you would use selenium to conduct tasks such as clicking and scrolling, while using BeautifulSoup to search HTML code from the webpage supplied by Selenium

To save all of our scrapped data to a CSV file, we can temporarily store data in a dictionary, (where the key is a column header, and the value being a list of the attributes) and use the Pandas Python library to make a CSV file. If you’re new to Pandas, utilize the Pandas Documentation

3. Data Cleaning and Exploratory Data Analysis (EDA)

Data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset .

After scraping StartUp Jobs, we are left with uncleaned data, which may have errors hindering our work in the future. With your freshly scraped data in hand, clean it using the Pandas Python library. Here is an example of basic data cleaning on a different dataset

Some possible things to consider while cleaning your data:

  • Removal of duplicate job postings (exact same job titles by the same company)
  • Removal of job postings in different languages
  • Removal of job postings with NA as attributes (matters for some attributes, for others not as much)

EDA (Exploratory Data Analysis) an analysis approach that identifies general patterns in the data . These patterns include outliers and features of the data that might be unexpected.

Develop code to help in the process of exploring and identifying patterns in the data we have scraped.

Some ways to explore the data:

  • Develop an algorithm which identifies the most common and significant words in the descriptions
  • Find the number of various attributes, such as how many jobs are onsite vs remote

4. Data Visualization

Once you’ve gleaned insights from your data, it’s time to communicate them visually. Leverage graphing libraries like Matplotlib or Seaborn to create compelling visualizations that highlight your data’s key takeaways.

There are many different types of graphs and plots you can make with Python with the data provided! However, remember that the best visualizations of data are simple and able to understood by beginners.

Here is a tutorial to get started using Matplotlib and Seaborn

5. Statistical Analysis

Deepen your analysis by applying statistical methodologies like TF-IDF to your data. You can learn more about these techniques in resources like MonkeyLearn’s guide on TF-IDF.

6. GPT API’s for Data Cleaning

Test out the GPT API for data cleaning tasks. Reflect on the advantages and drawbacks of employing AI in your data processing workflow.

Never used GPT API’s before? Here is some documentation to get you started .


Need Extra Help?

If you find yourself facing challenges, the following code snippets might come in handy. They offer a glimpse into the basic operations involved in web scraping, data cleaning, and EDA. Remember, real-world data requires more rigorous analysis and might necessitate advanced data cleaning steps such as removing duplicate entries or outliers, normalizing text data, etc. As you progress, make it a point to explore multiple aspects of your data and identify relationships between different variables.

1. Web Scraping with BeautifulSoup 
import requests
from bs4 import BeautifulSoup

# URL of the page you want to scrape
url = "https://startup.jobs/"

# Send a GET request to the webpage
response = requests.get(url)

# Parse the HTML content of the page with BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')

# Find job titles
titles = soup.find_all('h5', class_='job-listing-title')

# Find job locations
locations = soup.find_all('span', class_='job-listing-location')

# Print each job title and location
for title, location in zip(titles, locations):
    print(f'Title: {title.text.strip()}')
    print(f'Location: {location.text.strip()}')
    print("\n")

2. Data Cleaning with Pandas 
import pandas as pd

# Load your data
df = pd.read_csv('your_data.csv')

# Remove rows with missing values
df_cleaned = df.dropna()

# Save the cleaned data back into a CSV file
df_cleaned.to_csv('cleaned_data.csv', index=False)

3. Data Visualization with Pandas and Matplotlib 
import matplotlib.pyplot as plt

# Load your cleaned data
df = pd.read_csv('cleaned_data.csv')

# Get a basic statistical summary of the data
print(df.describe())

# Visualize the distribution of job locations
df['Location'].value_counts().plot(kind='bar')
plt.title('Distribution of Job Locations')
plt.ylabel('Count')
plt.xlabel('Location')
plt.show()