Data Gathering
This page talks about the data gathering techniques that have been used to collect raw data that would be further used in developing the ML model.
|
|
|---|
Data has been collected via 4 methods:
1 Data gathering using Python API
Tweepy is a Python library for accessing the Twitter Developer API. You need to sign up for a Twitter Developer Account (Twitter setup page will guide you in the processhe API. Once you have access to the API use the step-by-step guide to create an app and project. Remember to copy the keys in a txt file on your local machine.
Twitter is one of the most widely used social networks. For many organizations and people, having a great Twitter presence is a key factor to keeping their audience engaged. Part of having a great Twitter presence involves keeping your account active with new tweets and retweets, following interesting accounts, and quickly replying to your followers’ messages.
Here Tweepy is introduced as a tool to access Twitter data in a fairly easy way with Python. There are different types of data we can collect, with the obvious focus on the “tweet” object. Once we have collected some data, the possibilities in terms of analytics applications are endless. One of the major applications Tweepy is used for is extracting tweets for sentiment or emotion analysis. The emotion of the user can be obtained from the tweets by tokenizing each word and applying machine learning algorithms on that data. Such emotion or sentiment detection is used worldwide and will be broadly used in the future.
We have used tweepy to extract tweets related to salary and employment. And then further the text data is used for analysis to predict the names of the firms based upon statements in the past.
2 Data gathering using R API
NEWSDATA.IO provides API that generates news from 20,000 sources. It provides users to search by query, country, language, category and domain. The data from API displays the status of the request, total number of results and the news results. The news results contain title of the news, link to the article where it was posted, source of the article, keywords i.e. list of keywords related to the article, creator i.e. autohr(s) of the article, image_url, video_url, description, published Date i.e. date when the article was published and the full content.
Newsdata.io’s news API is developed to provide live and historical global news from thousands of sources with exceptional response. You can retrieve top stories based on country, as well as search all news data and filter by category, language, source, publish date, and more. Newsdata.io API provides headlines, images, and other article metadata from a range of popular news sources in JSON architecture with an API Key.
3 Web Scraping Levels.fyi
Levels.fyi is an online platform that helps in comparing career levels and compensation packages across different companies. We would be scraping data from this website for our model.
If you work in one of the MAANG(Meta/Apple/Amazon/Netflix/Google) companies, you’ve likely heard of site levels.fyi. For those who haven’t, Levels makes it easy to compare and contrast different career levels across different companies, and is generally considered more accurate in terms of actual tech salaries relative to other compensation sites like glassdoor.com. Levels.fyi has a great interface and its visualizations make it easy to compare salary bands across companies, however, for this project data has been analyzed in a way that pre-aggregated views on the site doesn’t allow.
4 Raw Data Gathering
Raw data has been gathered from multiple sources and hosted on Github.