51 Use API-wrapping packages

This section explores how to use R packages that wrap APIs, allowing easy access to web data. We’ll cover various approaches from simple downloads to more complex API interactions. This content builds on Jenny Bryan’s stat545 materials, with significant updates and additions to reflect current best practices. For a more in-depth look at APIs, check out the API chapter in the R for Data Science book. For a huge list of free apis to play with, check out this list

51.1 The Data Acquisition Spectrum

When it comes to obtaining data from the internet, we can categorize methods into four main type * Direct Download: Grabbing readily available flat files (CSV, XLS, etc.) * Wrapped API Access: Using R packages designed for specific APIs * Raw API Interaction: Crafting custom queries for APIs * Web Scraping: Extracting data embedded in HTML structures

We’ll focus primarily on the second method, but now you know about the full spectrum of options at your disposal. For a comprehensive list of R tools for interacting with the internet, check out the rOpenSci repository. These tools include packages for APIs, scraping, and more.

51.2 Direct Download

In the simplest case, the data you need is already on the internet in a tabular format. Effectively, you just need to click and download whatever data you need. There are a couple of strategies here:

  • Use read.csv or readr::read_csv to read the data straight into R.
  • Use the command line program curl to do that work, and place it in a Makefile or shell script (see the section on make for more on this).

The second case is most useful when the data you want has been provided in a format that needs cleanup. For example, the World Value Survey makes several datasets available as Excel sheets. The safest option here is to download the .xls file, then read it into R with readxl::read_excel() or something similar. An exception to this is data provided as Google Spreadsheets, which can be read straight into R using the googlesheets package.

51.2.1 From rOpenSci web services page

From rOpenSci’s CRAN Task View: Web Technologies and Services:

  • downloader::download() for SSL.
  • curl::curl() for SSL.
  • httr::GET data read this way needs to be parsed later with read.table().
  • rio::import() can “read a number of common data formats directly from an https:// URL”. Isn’t that very similar to the previous?

What about packages that install data?

51.3 Data supplied on the web

Many times, the data that you want is not already organized into one or a few tables that you can read directly into R. More frequently, you find this data is given in the form of an API. Application Programming Interfaces (APIs) are descriptions of the kind of requests that can be made of a certain piece of software, and descriptions of the kind of answers that are returned.

Many sources of data – databases, websites, services – have made all (or part) of their data available via APIs over the internet. Computer programs (“clients”) can make requests of the server, and the server will respond by sending data (or an error message). This client can be many kinds of other programs or websites, including R running from your laptop.

51.4 Streamlined Data Retrieval with API Wrappers

Many common web services and APIs have been “wrapped”, i.e. R functions have been written around them which send your query to the server and format the response. This is a great way to get started with APIs, as you don’t need to worry about the details of the API itself. You can just focus on the data you want to get.

API-wrapping packages act as intermediaries between your R environment and web services. They handle the nitty-gritty of API calls, authentication, and data parsing, allowing you to focus on analysis rather than data acquisition logistics. These packages are especially useful for beginners, as they abstract away the complexities of web service interaction. For added bonuses, they often ensure that the data you receive is actually from the API you intended to query, and they provide a structured reproducible way to access the data.

51.4.1 Case Study: Ornithological Data with rebird

Let’s dive into a practical example using the rebird package, which interfaces with the eBird database. eBird lets birders upload sightings of birds, and allows everyone access to those data. rebird makes it easy to access this data from R (as long as you request an API key)

library(tidyverse)
library(kableExtra)
library(rebird)

First, let’s fetch recent bird sightings from a specific location:

51.4.1.1 Search birds by geography

The eBird website categorizes some popular locations as “Hotspots”. These are areas where there are both lots of birds and lots of birders. One such location is at Iona Island, near Vancouver. You can see data for this Hotspot at http://ebird.org/ebird/hotspot/L261851.

At that link, you will see a page like this:

Iona Island

Figure 51.1: Iona Island

The data already looks to be organized in a data frame! rebird allows us to read these data directly into R (the ID code for Iona Island is “L261851”). Note that this requires an API key which you have to request from ebird via this link . I have set my key as an environment variable. However you can set it as a global variable in your R session. Like this:

ebirdkey <- "SECRET API KEY"
ebirdregion(loc = "L261851", key = ebirdkey) %>%
  head() %>%
  kable()
speciesCode comName sciName locId locName obsDt howMany lat lng obsValid obsReviewed locationPrivate subId exoticCategory
norsho Northern Shoveler Spatula clypeata L261851 Iona Island (General) 2025-03-22 15:37 5 49.2 -123 TRUE FALSE FALSE S220118840 NA
gadwal Gadwall Mareca strepera L261851 Iona Island (General) 2025-03-22 15:37 10 49.2 -123 TRUE FALSE FALSE S220118840 NA
amewig American Wigeon Mareca americana L261851 Iona Island (General) 2025-03-22 15:37 2 49.2 -123 TRUE FALSE FALSE S220118840 NA
mallar3 Mallard Anas platyrhynchos L261851 Iona Island (General) 2025-03-22 15:37 20 49.2 -123 TRUE FALSE FALSE S220118840 NA
norpin Northern Pintail Anas acuta L261851 Iona Island (General) 2025-03-22 15:37 10 49.2 -123 TRUE FALSE FALSE S220118840 NA
rinduc Ring-necked Duck Aythya collaris L261851 Iona Island (General) 2025-03-22 15:37 6 49.2 -123 TRUE FALSE FALSE S220118840 NA

We can use the function ebirdgeo() to get a list for an area (note that South and West are negative):

vanbirds <- ebirdgeo(lat = 49.2500, lng = -123.1000, key = ebirdkey)
vanbirds %>%
  head() %>%
  kable()
speciesCode comName sciName locId locName obsDt howMany lat lng obsValid obsReviewed locationPrivate subId exoticCategory
killde Killdeer Charadrius vociferus L22006609 Malekaw Bird Observatory 2025-03-24 20:20 2 49.2 -123 TRUE FALSE TRUE S220612403 NA
cangoo Canada Goose Branta canadensis L374029 Vancouver–Hadden & Vanier Parks 2025-03-24 19:20 20 49.3 -123 TRUE FALSE FALSE S220610455 NA
norsho Northern Shoveler Spatula clypeata L374029 Vancouver–Hadden & Vanier Parks 2025-03-24 19:20 NA 49.3 -123 TRUE FALSE FALSE S220610455 NA
amewig American Wigeon Mareca americana L374029 Vancouver–Hadden & Vanier Parks 2025-03-24 19:20 NA 49.3 -123 TRUE FALSE FALSE S220610455 NA
mallar3 Mallard Anas platyrhynchos L374029 Vancouver–Hadden & Vanier Parks 2025-03-24 19:20 NA 49.3 -123 TRUE FALSE FALSE S220610455 NA
tufduc Tufted Duck Aythya fuligula L374029 Vancouver–Hadden & Vanier Parks 2025-03-24 19:20 1 49.3 -123 TRUE TRUE FALSE S220610455 NA

Note: Check the defaults on this function (e.g. radius of circle, time of year).

We can also search by “region”, which refers to short codes which serve as common shorthands for different political units. For example, France is represented by the letters FR.

frenchbirds <- ebirdregion("FR", key = ebirdkey)
frenchbirds %>%
  head() %>%
  kable()
speciesCode comName sciName locId locName obsDt howMany lat lng obsValid obsReviewed locationPrivate subId exoticCategory
rocpig Rock Pigeon Columba livia L41648604 26 Place du Marché Gayot, Strasbourg FR-Grand Est 48.58279, 7.75310 2025-03-25 12:53 3 48.6 7.753 TRUE FALSE TRUE S220647673 N
firecr1 Common Firecrest Regulus ignicapilla L41648604 26 Place du Marché Gayot, Strasbourg FR-Grand Est 48.58279, 7.75310 2025-03-25 12:53 1 48.6 7.753 TRUE FALSE TRUE S220647673 NA
carcro1 Carrion Crow Corvus corone L41648604 26 Place du Marché Gayot, Strasbourg FR-Grand Est 48.58279, 7.75310 2025-03-25 12:53 1 48.6 7.753 TRUE FALSE TRUE S220647673 NA
blakit1 Black Kite Milvus migrans L26921758 Le Bugue Vezere 2025-03-25 12:21 1 44.9 0.952 TRUE FALSE TRUE S220642123 NA
gretit1 Great Tit Parus major L26921758 Le Bugue Vezere 2025-03-25 12:21 2 44.9 0.952 TRUE FALSE TRUE S220642123 NA
combuz1 Common Buzzard Buteo buteo L26921758 Le Bugue Vezere 2025-03-25 12:21 1 44.9 0.952 TRUE FALSE TRUE S220642123 NA

Find out when a bird has been seen in a certain place! Choosing a name from vanbirds above (the Bald Eagle):

eagle <- ebirdgeo(species = "baleag", 
                  lat = 42, lng = -76, key = ebirdkey)
eagle %>%
  head() %>%
  kable()
speciesCode comName sciName locId locName obsDt howMany lat lng obsValid obsReviewed locationPrivate subId
baleag Bald Eagle Haliaeetus leucocephalus L1005329 Brick Pond Wetland Preserve 2025-03-24 14:39 2 42.1 -76.2 TRUE FALSE FALSE S220541184
baleag Bald Eagle Haliaeetus leucocephalus L2347159 Tri-City Airport 2025-03-24 13:56 1 42.1 -76.1 TRUE FALSE FALSE S220539254
baleag Bald Eagle Haliaeetus leucocephalus L19140159 1099 Powderhouse Rd, Vestal US-NY 42.07361, -75.95256 2025-03-23 15:55 1 42.1 -76.0 TRUE FALSE TRUE S220335851
baleag Bald Eagle Haliaeetus leucocephalus L2683280 Wolfe Park, Town of Chenango 2025-03-23 14:53 1 42.2 -75.9 TRUE FALSE FALSE S220347356
baleag Bald Eagle Haliaeetus leucocephalus L41575561 I-86 E, Binghamton US-NY 42.11985, -75.94592 2025-03-23 13:16 1 42.1 -75.9 TRUE FALSE TRUE S220294282
baleag Bald Eagle Haliaeetus leucocephalus L2347076 Lockheed Martin Pond 2025-03-23 09:40 2 42.1 -76.2 TRUE FALSE FALSE S220448002

rebird knows where you are:

ebirdgeo(species = "rolhaw", key = ebirdkey)
#> Warning: As a complete lat/long pair was not provided, your location was
#> determined using your computer's public-facing IP address. This will likely not
#> reflect your physical location if you are using a remote server or proxy.
#> # A tibble: 0 × 0

51.4.2 Searching geographic info: geonames

rOpenSci has a package called geonames for accessing the GeoNames API. First, install the geonames package from CRAN and load it.

# install.packages("geonames")
library(geonames)

The geonames package website tells us that there are a few things we need to do before we can use geonames to access the GeoNames API:

  1. Go to the GeoNames site and create a new user account.
  2. Check your email and follow the instructions to activate your account.
  3. You have to manually enable the free web services for your account (Note! You must be logged into your GeoNames account).
  4. Tell R your GeoNames username.

To do the last step, we could run this line in R…

options(geonamesUsername="my_user_name")

…but this is insecure. We don’t want to risk committing this line and pushing it to our public GitHub page!

Instead, we can add this line to our .Rprofile so it will be hidden. One way to edit your .Rprofile is with the helper function edit_r_profile() from the usethis package. Install/load the usethis package and run edit_r_profile() in the R Console:

# install.packages("usethis")
library(usethis)
edit_r_profile()

This will open up your .Rprofile file. Add options(geonamesUsername="my_user_name") on a new line (replace “my_user_name” with your GeoNames username).

Important: Make sure your .Rprofile ends with a blank line!

Save the file, close it, and restart R. Now we’re ready to start using geonames to search the GeoNames API.

(Also see the Cache credentials for HTTPS chapter of Happy Git and GitHub for the useR.)

51.4.2.1 Using GeoNames

What can we do? We can get access to lots of geographical information via the various GeoNames WebServices.

countryInfo <- GNcountryInfo()
glimpse(countryInfo)
#> Rows: 250
#> Columns: 18
#> $ continent        <chr> "EU", "AS", "AS", "NA", "NA", "EU", "AS", "AF", "AN",…
#> $ capital          <chr> "Andorra la Vella", "Abu Dhabi", "Kabul", "Saint John…
#> $ languages        <chr> "ca", "ar-AE,fa,en,hi,ur", "fa-AF,ps,uz-AF,tk", "en-A…
#> $ geonameId        <chr> "3041565", "290557", "1149361", "3576396", "3573511",…
#> $ south            <chr> "42.4287475", "22.6315119400001", "29.3770645357176",…
#> $ isoAlpha3        <chr> "AND", "ARE", "AFG", "ATG", "AIA", "ALB", "ARM", "AGO…
#> $ north            <chr> "42.6558875", "26.0693916590001", "38.4907920755748",…
#> $ fipsCode         <chr> "AN", "AE", "AF", "AC", "AV", "AL", "AM", "AO", "AY",…
#> $ population       <chr> "77006", "9630959", "37172386", "96286", "13254", "28…
#> $ east             <chr> "1.7866939", "56.381222289", "74.8894511481168", "-61…
#> $ isoNumeric       <chr> "020", "784", "004", "028", "660", "008", "051", "024…
#> $ areaInSqKm       <chr> "468.0", "82880.0", "647500.0", "443.0", "102.0", "28…
#> $ countryCode      <chr> "AD", "AE", "AF", "AG", "AI", "AL", "AM", "AO", "AQ",…
#> $ west             <chr> "1.4135734", "51.5904085340001", "60.4720833972263", …
#> $ countryName      <chr> "Principality of Andorra", "United Arab Emirates", "I…
#> $ postalCodeFormat <chr> "AD###", "##### #####", "", "", "AI-####", "####", "#…
#> $ continentName    <chr> "Europe", "Asia", "Asia", "North America", "North Ame…
#> $ currencyCode     <chr> "EUR", "AED", "AFN", "XCD", "XCD", "ALL", "AMD", "AOA…

This countryInfo dataset is very helpful for accessing the rest of the data because it gives us the standardized codes for country and language.

51.4.2.2 Remixing geonames

What are the cities of France?

francedata <- countryInfo %>%
  filter(countryName == "France")
frenchcities <- with(francedata, GNcities(
  north = north, east = east,
  south = south, west = west,
  maxRows = 500
))
glimpse(frenchcities)
#> Rows: 133
#> Columns: 12
#> $ lng         <chr> "2.3488", "4.34878349304199", "7.44744300842285", "6.13", …
#> $ geonameId   <chr> "2988507", "2800866", "2661552", "2960316", "2993458", "30…
#> $ countrycode <chr> "FR", "BE", "CH", "LU", "MC", "JE", "AD", "GG", "ES", "IT"…
#> $ name        <chr> "Paris", "Brussels", "Bern", "Luxembourg", "Monaco", "Sain…
#> $ fclName     <chr> "city, village,...", "city, village,...", "city, village,.…
#> $ toponymName <chr> "Paris", "Brussels", "Bern", "Luxembourg", "Monaco", "Sain…
#> $ fcodeName   <chr> "capital of a political entity", "capital of a political e…
#> $ wikipedia   <chr> "en.wikipedia.org/wiki/Paris", "en.wikipedia.org/wiki/City…
#> $ lat         <chr> "48.85341", "50.8504450552593", "46.9480943365053", "49.61…
#> $ fcl         <chr> "P", "P", "P", "P", "P", "P", "P", "P", "P", "P", "P", "P"…
#> $ population  <chr> "2138551", "1019022", "121631", "76684", "32965", "28000",…
#> $ fcode       <chr> "PPLC", "PPLC", "PPLC", "PPLC", "PPLC", "PPLC", "PPLC", "P…

51.4.3 Wikipedia searching

We can use geonames to search for georeferenced Wikipedia articles. Here are those within 20 km of Rio de Janerio, comparing results for English-language Wikipedia (lang = "en") and Portuguese-language Wikipedia (lang = "pt"):

rio_english <- GNfindNearbyWikipedia(
  lat = -22.9083, lng = -43.1964,
  radius = 20, lang = "en", maxRows = 500
)
rio_portuguese <- GNfindNearbyWikipedia(
  lat = -22.9083, lng = -43.1964,
  radius = 20, lang = "pt", maxRows = 500
)
nrow(rio_english)
#> [1] 457
nrow(rio_portuguese)
#> [1] 500

51.4.4 Is it a boy or a girl? gender-associated names throughout US history

The gender package allows you access to data on the gender of names in the US. Because names change gender over the years, the probability of a name belonging to a man or a woman also depends on the year.

First, install/load the gender package from CRAN. You may be prompted to also install the companion package, genderdata. Go ahead and say yes. If you don’t see this message no need to worry, it is a one-time install.

# install.packages("gender")
#remotes::install_github("lmullen/genderdata")
library(gender)

Let’s do some searches for the name Kelsey.

gender("Kelsey")
#> # A tibble: 1 × 6
#>   name   proportion_male proportion_female gender year_min year_max
#>   <chr>            <dbl>             <dbl> <chr>     <dbl>    <dbl>
#> 1 Kelsey          0.0314             0.969 female     1932     2012
gender("Kelsey", years = 1940)
#> # A tibble: 1 × 6
#>   name   proportion_male proportion_female gender year_min year_max
#>   <chr>            <dbl>             <dbl> <chr>     <dbl>    <dbl>
#> 1 Kelsey               1                 0 male       1940     1940

As you can see, the probability of a name belonging to a specific gender can change over time.

df <- gender("Kelsey")

years <- c(df$year_min:df$year_max)

for(i in 1:length(years)){
  
df <-rbind(df,gender("Kelsey", years = years[i]))

}
df %>% filter(year_min==year_max) %>% 
ggplot( aes(year_min, proportion_male)) +
  geom_smooth(span = 0.1) +
  labs(title = "Proportion of men named Kelsey over time",
       x = "Year",
       y = "Proporiton Male") + ggthemes::theme_excel()
#> `geom_smooth()` using method = 'loess' and formula = 'y ~ x'

In contrast, the name Mason has been more consistently male.

gender("Mason")
#> # A tibble: 1 × 6
#>   name  proportion_male proportion_female gender year_min year_max
#>   <chr>           <dbl>             <dbl> <chr>     <dbl>    <dbl>
#> 1 Mason           0.987            0.0134 male       1932     2012
gender("Mason", years = 1940)
#> # A tibble: 1 × 6
#>   name  proportion_male proportion_female gender year_min year_max
#>   <chr>           <dbl>             <dbl> <chr>     <dbl>    <dbl>
#> 1 Mason               1                 0 male       1940     1940
mason <- gender("Mason")
years <- c(mason$year_min:mason$year_max)

df <- rbind(df,gender("Mason"))

for(i in 1:length(years)){
  
df <-rbind(df,gender("Mason", years = years[i]))

}

df %>% filter(year_min==year_max&name=="Mason") %>% 
ggplot( aes(year_min, proportion_male)) +
  geom_point() +
  geom_smooth(span = 0.1) +
  labs(title = "Proportion of men named Mason over time",
       x = "Year",
       y = "Proporiton Male") + ggthemes::theme_excel()
#> `geom_smooth()` using method = 'loess' and formula = 'y ~ x'

And when we compare the two names, we see that Kelsey has changed a lot more over time than Mason.

df %>% filter(year_min==year_max) %>% 
ggplot( aes(year_min, proportion_male)) +
  geom_point() +
  geom_smooth(span = 0.1) +
  labs(title = "Proportion of men over time",
       x = "Year",
       y = "Proporiton Male")  + ggthemes::theme_excel() + facet_wrap(~name)
#> `geom_smooth()` using method = 'loess' and formula = 'y ~ x'

51.5 Conclusion

API-wrapping packages in R provide powerful tools for accessing diverse data sources. As you progress in your data science journey, mastering these tools will greatly expand the range of data available for your analyses. Remember to always check the terms of service for any API you use, and be mindful of rate limits and data usage policies.