Week 5B
Getting Data Part 1: Working with APIs
Oct 1, 2020
Week #5 Agenda¶
Last time:
- Introduction to APIs
- Pulling census data and shape files using Python
Today:
- API Example: Lead poisoning in Philadelphia
- Using the Twitter API
- Plotting geocoded tweets
- Word frequencies
- Sentiment analysis
import geopandas as gpd
import pandas as pd
import numpy as np
from shapely.geometry import Point
from matplotlib import pyplot as plt
import seaborn as sns
import hvplot.pandas
import holoviews as hv
hv.extension("bokeh")
%matplotlib inline
import carto2gpd
import cenpy
Last time: Querying the census with cenpy¶
First, let's initialize a connection to the 2018 5-year ACS dataset
acs = cenpy.remote.APIConnection("ACSDT5Y2018")
# Set the map service for pulling geometries
acs.set_mapservice("tigerWMS_ACS2018")
Exercise: lead poisoning in Philadelphia¶
Let's use demographic census data in Philadelphia by census tract and compare to a dataset of childhood lead poisoning.
Step 1. Download the demographic data for Philadelphia¶
- We are going to be examining the percent of the population that identifies as black in each census tract, so we will need:
- Total population: 'B03002_001E'
- Non-Hispanic, Black population: 'B03002_004E'
- You'll want to use the state --> county --> tract hierarchy , using the
*operator to get all tracts in Philadelphia county - Remember PA has a FIPS code of "42" and Philadelphia County is "101"
philly_demo_tract = acs.query(
cols=["NAME", "B03002_001E", "B03002_004E"],
geo_unit="tract:*",
geo_filter={"state" : "42",
"county" : "101"},
)
Step 2. Download and merge in the census tract geometries¶
# Census tracts are the 9th layer (index 8 starting from 0)
acs.mapservice.layers[8]
# Use SQL to return geometries only for Philadelphia County in PA
where_clause = "STATE = 42 AND COUNTY = 101"
# Query for census tracts
philly_census_tracts = acs.mapservice.layers[8].query(where=where_clause)
philly_census_tracts.head(n=1)
philly_demo_tract.head(n=1)
philly_demo_tract = philly_census_tracts.merge(
philly_demo_tract,
left_on=["STATE", "COUNTY", "TRACT"],
right_on=["state", "county", "tract"],
)
Step 3. Calculate the black percentage¶
Add a new column to your data called percent_black.
Important: Make sure you convert the data to floats!
for col in ['B03002_001E', 'B03002_004E']:
philly_demo_tract[col] = philly_demo_tract[col].astype(float)
philly_demo_tract['percent_black'] = 100 * philly_demo_tract['B03002_004E'] / philly_demo_tract['B03002_001E']
Step 4. Query CARTO to get the childhood lead levels by census tract¶
- The API documentation for this data is here: https://cityofphiladelphia.github.io/carto-api-explorer/#child_blood_lead_levels_by_ct
- You can use the
carto2gpdpackage- You'll need the API endpoint for CARTO and the name of the table
table_name = 'child_blood_lead_levels_by_ct'
lead_levels = carto2gpd.get("https://phl.carto.com/api/v2/sql", table_name)
lead_levels.head()
Step 5. Remove census tracts with missing lead measurements¶
See the .dropna() function and the subset= keyword.
lead_levels = lead_levels.dropna(subset=['perc_5plus'])
Step 6. Merge the demographic and lead level data frames¶
- From the lead data, we only need the 'census_tract' and 'perc_5plus'. Before merging, trim your data to only these columns.
- You can perform the merge by comparing the
census_tractandGEOIDfields - Remember: when merging, the left data frame should be the GeoDataFrame — use
GeoDataFrame.merge(...)
# Trim the lead levels data
lead_levels_trimmed = lead_levels[['census_tract', 'perc_5plus']]
# Merge into the demographic data
merged = philly_demo_tract.merge(lead_levels_trimmed,
how='left',
left_on='GEOID',
right_on='census_tract')
Step 7. Trim to the columns we need¶
We only need the 'geometry', 'percent_black', and 'perc_5plus', and 'NAME' columns
merged = merged[['NAME_x', 'geometry', 'percent_black', 'perc_5plus']]
merged.head()
Step 8. Plot the results¶
Make two plots:
- A two panel, side-by-side chart showing a choropleth of the lead levels and the percent black
- A scatter plot showing the percentage
You can make these using hvplot or geopandas/matplotlib — whichever you prefer!
# Lead levels plot
img1 = merged.hvplot(geo=True,
crs=3857,
c='perc_5plus',
width=500,
height=400,
cmap='viridis',
title='Lead Levels')
# Percent black
img2 = merged.hvplot(geo=True,
crs=3857,
c='percent_black',
width=500,
height=400,
cmap='viridis',
title='% Black')
img1 + img2
cols = ['perc_5plus', 'percent_black']
merged[cols].hvplot.scatter(x=cols[0], y=cols[1])
Step 9. Use seaborn to plot a 2d density map¶
In the previous plots, it's still hard to see the relationship. Use the kdeplot() function in seaborn to better visualize the relationship.
You will need to remove any NaN entries first.
You should see two peaks in the distribution clearly now!
fig, ax = plt.subplots(figsize=(8,6))
X = merged.dropna()
sns.kdeplot(X['perc_5plus'], X['percent_black'], ax=ax)
API Example #2: the Twitter API¶
Twitter provides a rich source of information, but the challenge is how to extract the information from semi-structured data.
Semi-structured data¶
Data that contains some elements that cannot be easily consumed by computers
Examples: human-readable text, audio, images, etc
Key challenges¶
- Text mining: analyzing blocks of text to extract the relevant pieces of information
- Natural language processing (NLP): programming computers to process and analyze human languages
- Sentiment analysis: analyzing blocks of text to derive the attitude or emotional state of the person
First: Getting an API key¶
Step 1: Make a Twitter account¶
Step 2: Apply for Developer access¶
You will need to apply for a Developer Account, answering a few questions, and then confirm your email address.
Once you submit you'll application, you'll need to wait for approval...usually this happens immediately, but there can sometimes be a short delay
Sample answer¶
Needs to be at least 100 characters
- I'm using Twitter's API to perform a sentiment analysis as part of a class teaching Python. I will be interacting with the API using the Python package tweepy.
- I plan to analyze tweets to understand topic sentiments.
- I will not be interacting with Twitter users as part of
- I will not be displaying Twitter content off of Twitter.
Step 3: Create a new app¶
Step 4: Create your API keys¶
In the "Keys and Tokens" section, generate new access tokens.
You will need the Consumer API keys and access tokens to use the Twitter API.
We'll be using tweepy to search recent tweets¶
The standard, free API let's you search tweets from the last 7 days
For more information, see the Twitter Developer docs
Tweepy: a Python interface to Twitter¶
import tweepy as tw
Define your API keys¶
# INPUT YOUR API AND ACCESS KEYS HERE
api_key = ""
api_key_secret = ''
access_token = ''
access_token_secret = ''
auth = tw.OAuthHandler(api_key, api_key_secret)
auth.set_access_token(access_token, access_token_secret)
api = tw.API(auth, wait_on_rate_limit=True)
Rate Limits¶
Be careful: With the free API, you are allowed 15 API calls every 15 minutes.
See the Rate Limits documentation for more details.
What does wait_on_rate_limit do?¶
If you run into a rate limit while pulling tweets, this will tell tweepy to wait 15 minutes until it can continue.
Unfortunately, you need to sign up (and pay) for the premium API to avoid these rate limits.
How to find out how many requests you've made?¶
data = api.rate_limit_status()
data
data['resources']['search']
Tip: converting a time stamp to a date¶
import datetime
datetime.datetime.fromtimestamp(1601594736)
Several different APIs available¶
Including user tweets, mentions, searching keywords, favoriting, direct messages, and more...
See the Tweepy API documentation
You can also tweet (if you want!)¶
You can post tweets using the update_status() function. For example:
tweet = 'Hello World! @PennMusa'
api.update_status(tweet)
We'll focus on the search API¶
We'll use a tweepy.Cursor object to query the API.
# collect tweets related to the phillies
search_words = "#phillies"
# initialize the cursor
cursor = tw.Cursor(api.search,
q=search_words,
lang="en",
tweet_mode='extended')
cursor
Next, specify how many tweets we want¶
Use the Cursor.items() function:
# select 5 tweets
tweets = cursor.items(5)
tweets
Python iterators¶
As the name suggests, iterators need to be iterated over to return objects. In our case, we can use a for loop to iterate through the tweets that we pulled from the API.
# Iterate on tweets
for tweet in tweets:
print(tweet.full_text)
The concept of "paging"¶
Unfortunately, there is no way to search for tweets between specific dates.
The API pulls tweets from the most recent page of the search result, and then grabs tweets from the previous page, and so on, to return the requested number of tweets.
Customizing our search query¶
The API documentation has examples of different query string use cases
Examples¶
Let's remove retweets¶
new_search = search_words + " -filter:retweets"
Get a new tweets using our new search query:
cursor = tw.Cursor(api.search,
q=new_search,
lang="en",
tweet_mode='extended')
tweets = cursor.items(5)
Did it work?¶
for tweet in tweets:
print(tweet.full_text)
How to save the tweets?¶
Create a list of the tweets using Python's inline syntax
# select the next 10 tweets
tweets = [t for t in cursor.items(10)]
print(len(tweets))
A wealth of information is available¶
Beyond the text of the tweet, things like favorite and retweet counts are available. You can also extract info on the user behind the tweet.
first_tweet = tweets[0]
first_tweet.full_text
first_tweet.favorite_count
first_tweet.created_at
first_tweet.retweet_count
first_tweet.user.description
first_tweet.user.followers_count
Let's extract screen names and locations¶
A fraction of tweets have locations associated with user profiles, giving (very rough) location data.
users_locs = [[tweet.user.screen_name, tweet.user.location] for tweet in tweets]
users_locs
Note: only about 1% of tweets have a latitude/longitude.
Difficult to extract geographic trends without pulling a large number of tweets, requiring a premium API.
Use case #1: calculating word frequencies¶
An example of text mining
Load the most recent 1,000 tweets¶
Save the text of 1,000 tweets after querying our cursor object.
cursor = tw.Cursor(api.search,
q="#phillies -filter:retweets",
lang="en",
tweet_mode='extended')
tweets = [tweet for tweet in cursor.items(1000)]
# get the text of the tweets
tweets_text = [tweet.full_text for tweet in tweets]
# the first five tweets
tweets_text[:5]
Text mining and dealing with messy data¶
- Remove URLs $\rightarrow$ regular expressions
- Remove stop words
- Remove the search terms
Step 1: removing URLs¶
Don't worry about mastering regular expression syntax...
StackOverflow is your friend
def remove_url(txt):
"""
Replace URLs found in a text string with nothing
(i.e. it will remove the URL from the string).
Parameters
----------
txt : string
A text string that you want to parse and remove urls.
Returns
-------
The same txt string with url's removed.
"""
import re
return " ".join(re.sub("https://t.co/[A-Za-z\\d]+|&", "", txt).split())
Remove any URLs¶
tweets_no_urls = [remove_url(tweet) for tweet in tweets_text]
tweets_no_urls[:5]
Extract a list of lower-cased words in a tweet¶
.lower()makes all words lower cased.split()splits a string into the individual words
"This is an Example".lower()
"This is an Example".lower().split()
Apply these functions to all tweets:
words_in_tweet = [tweet.lower().split() for tweet in tweets_no_urls]
words_in_tweet[:2]
Counting word frequencies¶
We'll define a helper function to calculate word frequencies from our lists of words.
def count_word_frequencies(words_in_tweet, top=15):
"""
Given a list of all words for every tweet, count
word frequencies across all tweets.
By default, this returns the top 15 words, but you
can specify a different value for `top`.
"""
import itertools, collections
# List of all words across tweets
all_words = list(itertools.chain(*words_in_tweet))
# Create counter
counter = collections.Counter(all_words)
return pd.DataFrame(counter.most_common(top),
columns=['words', 'count'])
counts_no_urls = count_word_frequencies(words_in_tweet, top=15)
counts_no_urls.head(n=15)
Now let's plot the frequencies¶
Use seaborn to plot our DataFrame of word counts...
fig, ax = plt.subplots(figsize=(8, 8))
# Plot horizontal bar graph
sns.barplot(
y="words",
x="count",
data=counts_no_urls.sort_values(by="count", ascending=False),
ax=ax,
color="#cc3000",
saturation=1.0,
)
ax.set_title("Common Words Found in Tweets (Including All Words)", fontsize=16)
Step 2: remove stop words and punctuation¶
Common words that do not carry much significance and are often ignored in text analysis.
We can use the nltk package.
The "Natural Language Toolkit" https://www.nltk.org/
Import and download the stop words¶
import nltk
nltk.download('stopwords');
Get the list of common stop words¶
stop_words = list(set(nltk.corpus.stopwords.words('english')))
stop_words[:10]
len(stop_words)
Get the list of common punctuation¶
import string
punctuation = list(string.punctuation)
punctuation[:5]
Remove stop words from our tweets¶
ignored = stop_words + punctuation
ignored[:10]
# Remove stop words from each tweet list of words
tweets_nsw = [[word for word in tweet_words if word not in ignored]
for tweet_words in words_in_tweet]
tweets_nsw[0]
Get our DataFrame of frequencies¶
counts_nsw = count_word_frequencies(tweets_nsw)
counts_nsw.head()
And plot...
fig, ax = plt.subplots(figsize=(8, 8))
sns.barplot(
y="words",
x="count",
data=counts_nsw.sort_values(by="count", ascending=False),
ax=ax,
color="#cc3000",
saturation=1.0,
)
ax.set_title("Common Words Found in Tweets (Without Stop Words)", fontsize=16);
Step 3: remove our query terms¶
Now, we'll be left with only the meanigful words...
search_terms = ['#phillies', "phillies", "@phillies"]
tweets_final = [[w for w in word if w not in search_terms]
for word in tweets_nsw]
# frequency counts
counts_final = count_word_frequencies(tweets_final)
And now, plot the cleaned tweets...¶
fig, ax = plt.subplots(figsize=(8, 8))
sns.barplot(
y="words",
x="count",
data=counts_final.sort_values(by="count", ascending=False),
ax=ax,
color="#cc3000",
saturation=1.0,
)
ax.set_title("Common Words Found in Tweets (Cleaned)", fontsize=16)
At home exercise¶
Get 1,000 tweets using a query string of your choice and plot the word frequencies.
Be sure to:
- remove URLs
- remove stop words / punctuation
- remove your search query terms
Note: if you try to pull more than 1,000 tweets you will likely run into the rate limit and have to wait 15 minutes.
Remember: the API documentation describes how to customize a query string.
Use case #2: sentiment analysis¶
The goal of a sentiment analysis is to determine the attitude or emotional state of the person who sent a particular tweet.
Often used by brands to evaluate public opinion about a product.
The goal:¶
Determine the "sentiment" of every word in the English language
The hard way¶
Train a machine learning algorithm to classify words as positive vs. negative, given an input training sample of words.
The easy way¶
Luckily, this is a very common task in NLP and there are several packages available that have done the hard work for you.
They provide out-of-the-box sentiment analysis using pre-trained machine learning algorithms.
We'll be using textblob¶
import textblob
Let's analyze our set of 1,000 #phillies tweets¶
Create our "text blobs"¶
Simply pass the tweet text to the TextBlob() object.
Note: it's best to remove any URLs first!
blobs = [textblob.TextBlob(remove_url(t.full_text)) for t in tweets]
blobs[0]
blobs[0].sentiment
Combine the data into a DataFrame¶
Track the polarity, subjectivity, and date of each tweet.
data = {}
data['date'] = [t.created_at for t in tweets]
data['polarity'] = [b.sentiment.polarity for b in blobs]
data['subjectivity'] = [b.sentiment.subjectivity for b in blobs]
data['text'] = [remove_url(t.full_text) for t in tweets]
data = pd.DataFrame(data)
data.head()
How many are unbiased?¶
We can remove tweets with a polarity of zero to get a better sense of emotions.
zero = (data['polarity']==0).sum()
print("number of unbiased tweets = ", zero)
# remove unbiased tweets
biased = data.loc[ data['polarity'] != 0 ].copy()
What does a polarized tweet look like?¶
We can find the tweet with the maximum positive/negative scores
The most negative¶
Use the idxmin() function:
biased.loc[biased['polarity'].idxmin(), 'text']
The most positive¶
Use the idxmax() function
biased.loc[biased['polarity'].idxmax(), 'text']
Plot a histogram of polarity¶
We can use matplotlib's hist() function:
# create a figure and axes
fig, ax = plt.subplots(figsize=(10, 6))
# histogram
ax.hist(biased['polarity'], bins='auto')
ax.axvline(x=0, c='k', lw=2)
# format
ax.set_xlabel("Polarity")
ax.set_title("Polarity of #phillies Tweets", fontsize=16);
And subjectivity too...¶
The most objective¶
biased.loc[biased['subjectivity'].idxmin(), 'text']
The most subjective¶
biased.loc[biased['subjectivity'].idxmax(), 'text']
The distribution of subjectivity¶
# create a figure and axes
fig, ax = plt.subplots(figsize=(10, 6))
# histogram
ax.hist(biased['subjectivity'], bins='auto')
ax.axvline(x=0.5, c='k', lw=2)
# format
ax.set_xlabel("Subjectivity")
ax.set_title("Subjectivity of #phillies Tweets", fontsize=16);
How does polarity influence subjectivity?¶
Are positive/negative tweets more or less objective?
Seaborn's regplot() function¶
Is there a linear trend?
ax = sns.regplot(x=biased['subjectivity'], y=biased['polarity'])
Seaborn's kdeplot()¶
Shade the bivariate relationship
ax = sns.kdeplot(data=biased['subjectivity'], data2=biased['polarity'])
Insight: the most subjective tweets tend to be most polarized as well...
Let's check for hourly trends too¶
We can plot the distribution of polarity by the tweet's hour
First, we'll add a new column that gives the day and hour of the tweet.
We can use the built-in strftime() function.
# this is month/day hour AM/PM
biased['date_string'] = biased['date'].dt.strftime("%-m/%d %I %p")
biased.head()
Sort the tweets in chronological order...
biased = biased.sort_values(by='date', ascending=True)
Make a box and whiskers plot of the polarity¶
Use Seaborn's boxplot() function
ax = sns.boxplot(y='date_string', x='polarity', data=biased)
ax.axvline(x=0, c='k', lw=2) # neutral
# format
plt.setp(ax.get_yticklabels(), fontsize=10)
ax.figure.set_size_inches((8,12))
And subjectivity over time...¶
ax = sns.boxplot(y='date_string', x='subjectivity', data=biased)
ax.axvline(x=0.5, c='k', lw=2) # neutral
# format
plt.setp(ax.get_yticklabels(), fontsize=10)
ax.figure.set_size_inches((8,14))
At home exercise: sentiment analysis¶
Analyze your set of tweets from the last exercise (or get a new set), and explore the sentiments by:
- plotting histograms of the subjectivity and polarity
- finding the most/least subjective and polarized tweets
- plotting the relationship between polarity and subjectivity
- showing hourly trends in polarity/subjectivity
Or explore trends in some new way!
That's it!¶
- Next week: creating your own datasets through web scraping
- Pre-recorded lecture will be posted on Sunday/Monday
- See you next Thursday!