---
title: "scholar: Analyse Citation Data from Google Scholar"
author: "Guangchuang Yu, James Keirstead and Gregory Jefferis"
date: "`r Sys.Date()`"
format:
  html:
    toc: true
    theme: cosmo
    code-fold: false
execute:
  message: false
  warning: false
  freeze: auto
vignette: >
  %\VignetteIndexEntry{scholar introduction}
  %\VignetteEngine{quarto::html}
  %\usepackage[utf8]{inputenc}
---

```{r}
#| label: setup
#| echo: false
#| results: hide
knitr::opts_chunk$set(
  tidy = FALSE,
  message = FALSE,
  warning = FALSE,
  fig.width = 7,
  fig.height = 4
)

has_scholar <- yulab.utils::has_internet("https://scholar.google.com")
run_examples <- has_scholar &&
  identical(tolower(Sys.getenv("SCHOLAR_RUN_VIGNETTE_EXAMPLES")), "true")
```

```{r}
#| label: libraries
#| echo: false
#| results: hide
library("scholar")
library("ggplot2")
theme_set(theme_minimal())
```

# Overview

The **scholar** package extracts citation and publication data from Google
Scholar profiles. It can be used to:

* find Google Scholar profile IDs;
* retrieve profile metadata, publication records, and citation histories;
* calculate citation metrics such as h-index, g-index, i10-index, i50-index,
  and i100-index;
* inspect article-level metadata and article citation histories;
* complete truncated author lists;
* compare multiple scholars;
* explore coauthor networks;
* format publication lists for CVs;
* match publication venues against SCImago journal rankings; and
* predict future h-index values.

Google Scholar does not provide an official public API. Results can change when
Google changes page markup, and repeated requests may be rate limited. The
examples below use small requests and are skipped automatically when Google
Scholar is not reachable.

# Scholar IDs

Most functions need a Google Scholar profile ID. In a profile URL such as
`https://scholar.google.com/citations?user=B7vSqZsAAAAJ`, the ID is
`B7vSqZsAAAAJ`.

```{r ids-basic}
id <- "B7vSqZsAAAAJ"
```

If you have a URL or an ID with trailing query parameters, `tidy_id()` keeps only
the Scholar ID.

```{r tidy-id}
tidy_id("https://scholar.google.com/citations?user=B7vSqZsAAAAJ&hl=en")
tidy_id("B7vSqZsAAAAJ&hl=en")
```

Author search is the endpoint Google has put behind a login, so
`get_scholar_id()` and `search_scholar_ids()` can come back empty on a machine
where a browser shows the profile plainly; the package warns when the result is
a refusal rather than an absence of matches. If you have the profile URL, take
the ID from it. If you are matching authors programmatically, OpenAlex is the
sanctioned source.

```{r get-scholar-id, eval=run_examples}
get_scholar_id(first_name = "Richard", last_name = "Feynman")
```

For ambiguous names, `search_scholar_ids()` returns all author-search matches it
can find, including IDs, affiliations, verified email text, research interests,
and profile URLs. `max_pages` limits pagination so that examples and scripts do
not make excessive requests.

```{r search-scholar-ids, eval=run_examples}
search_scholar_ids("hao xu", max_pages = 1)
```

# Profile Data

`get_profile()` returns a list with name, affiliation, citation metrics, fields,
homepage, coauthors, and availability counts.

```{r profile, eval=run_examples}
profile <- get_profile(id)
profile$name
profile[c("total_cites", "h_index", "i10_index")]
profile$fields
```

# Publications

`get_publications()` returns a data frame of publications from a Scholar
profile. It includes the publication title, displayed author list, venue text,
volume/page details, citation count, year, the Google Scholar citation ID
(`cid`), and the publication ID (`pubid`).

```{r publications, eval=run_examples}
pubs <- get_publications(id, pagesize = 20)
head(pubs, 3)
```

You can sort the profile page by citation count or by publication year.

```{r publications-sort, eval=run_examples}
head(get_publications(id, pagesize = 10, sortby = "year"), 3)
```

Google Scholar sometimes truncates long author lists with `...`. Use
`get_publications_all_authors()` when you need the complete author list for
those rows. It only calls `get_complete_authors()` for rows that appear to be
truncated.

```{r all-authors, eval=FALSE}
pubs_full <- get_publications_all_authors(id, pagesize = 20, delay = 1)
head(pubs_full, 3)
```

For a single publication ID, call `get_complete_authors()` directly.

```{r complete-authors, eval=run_examples}
get_complete_authors(id, pubs$pubid[1], delay = 0)
```

`author_position()` helps analyse where a named author appears in publication
author lists. `Position_Normalized` is 0 for first author, 1 for last author,
and values between 0 and 1 for middle positions.

```{r author-position, eval=run_examples}
author_position(pubs$author[1:5], profile$name)
```

# Citation Histories

`get_citation_history()` retrieves the yearly citation bar chart for a scholar.

```{r citation-history, eval=run_examples}
ct <- get_citation_history(id)
ct
```

```{r citation-history-plot, eval=run_examples}
ggplot(ct, aes(year, cites)) +
  geom_line() +
  geom_point() +
  labs(x = "Year", y = "Citations")
```

`get_article_cite_history()` retrieves the yearly citation history for one
publication. Missing years are filled with zero citations.

```{r article-citation-history, eval=run_examples}
article_history <- get_article_cite_history(id, pubs$pubid[1])
article_history
```

```{r article-citation-history-plot, eval=run_examples}
ggplot(article_history, aes(year, cites)) +
  geom_segment(aes(xend = year, yend = 0), linewidth = 1, colour = "grey70") +
  geom_point(size = 3, colour = "firebrick") +
  labs(x = "Year", y = "Citations")
```

`get_citing_articles()` goes one step further than a citation count and lists
the works citing a publication, with their venue and year, which is what you
need to split a citation curve by community or to measure an autocitation rate.
Google caps the list the same way the "cited by" link does, and guards it more
closely than a profile page, so expect it to be refused from a server.

```{r citing-articles, eval=FALSE}
pubs <- get_publications(id)
top <- pubs[which.max(pubs$cites), ]
get_citing_articles(id, top$pubid, publications = pubs, max_pages = 2)
```

# How the h-index Grew

`get_profile()` reports one h-index at one moment. `get_h_index_history()`
rebuilds it year by year from the citation history of each publication, which is
the difference between knowing someone is at h-index 39 and knowing whether they
got there in five years or thirty.

```{r h-index-history, eval=FALSE}
history <- get_h_index_history(id, delay = 1)
head(history)
```

```{r h-index-history-plot, eval=FALSE}
ggplot(history, aes(year, h_index)) +
  geom_step(direction = "hv") +
  labs(x = NULL, y = "h-index",
       title = "How the h-index accumulated",
       subtitle = "each step is a year in which another paper reached the threshold")
```

The cost is one request per cited publication, so the histories are cached with
`R.cache` and a second call for the same profile makes no requests. Pass
`flush = TRUE` to refresh. Publications with no citations are skipped, since they
cannot enter an h-index in any year, and the curve is only as long as the years
Google reports per publication.

# Citation Metrics

`get_publication_metrics()` calculates metrics from a publication data frame.
`get_scholar_metrics()` fetches publications and calculates the same summary in
one call.

```{r publication-metrics, eval=run_examples}
get_publication_metrics(pubs)
get_scholar_metrics(id, pagesize = 20)
```

The package also includes focused helpers used by the h-index prediction model.

```{r publication-helpers, eval=run_examples}
get_num_articles(id)
get_oldest_article(id)
get_num_distinct_journals(id)
get_num_top_journals(id)
```

# Publication Details

Google Scholar publication detail pages can expose additional fields. These
helpers accept a Scholar ID and a publication ID.

```{r publication-detail, eval=run_examples}
pubid <- pubs$pubid[1]

get_article_scholar_url(id, pubid)
get_publication_url(id, pubid)
get_publication_date(id, pubid)
get_publication_abstract(id, pubid)
get_publication_data_extended(id, pubid)
```

Some publications have no linked PDF, abstract, or publication date on Google
Scholar. In those cases the corresponding helper may return an empty value or
`NA`.

# Comparing Scholars

`compare_scholars()` compares citation totals from publication records by
publication year.

```{r compare-scholars, eval=run_examples}
ids <- c("B7vSqZsAAAAJ", "DO5oG40AAAAJ")
cs <- compare_scholars(ids)
head(cs, 3)
```

```{r compare-scholars-plot, eval=run_examples}
cs <- dplyr::filter(cs, !is.na(year) & year > 1900)

ggplot(cs, aes(year, cites, colour = name, group = name)) +
  geom_line() +
  labs(x = "Publication year", y = "Citations", colour = NULL) +
  theme(legend.position = "bottom")
```

`compare_scholar_careers()` compares citation histories after aligning each
scholar by career year.

```{r compare-careers, eval=run_examples}
csc <- compare_scholar_careers(ids)
head(csc, 3)
```

```{r compare-careers-plot, eval=run_examples}
ggplot(csc, aes(career_year, cites, colour = name, group = name)) +
  geom_line() +
  geom_point() +
  labs(x = "Career year", y = "Citations", colour = NULL) +
  theme(legend.position = "bottom")
```

# Coauthor Networks

`get_coauthors()` retrieves the coauthors listed by Google Scholar for a profile
and can optionally expand the network by following coauthors' profiles. Keep
`n_coauthors` and `n_deep` small for exploratory work; network plots get messy
quickly and repeated requests can be rate limited.

```{r coauthors, eval=run_examples}
coauthor_network <- get_coauthors("DO5oG40AAAAJ", n_coauthors = 4, n_deep = 0)
coauthor_network
```

```{r coauthors-plot, eval=run_examples}
if (nrow(coauthor_network) > 1) {
  plot_coauthors(coauthor_network)
}
```

# Journal Rankings

`get_journalrank()` is deprecated and fetches nothing. SCImago has served its
ranking page behind a JavaScript challenge since late 2025, so a script is
answered with HTTP 403 and cannot read the table, and the table carries no
published licence. The function remains so that old code fails with an
explanation rather than a stack trace:

```{r journalrank-deprecated, error=TRUE}
get_journalrank(c("Nature", "Science"))
```

For journal metrics, use OpenAlex, which is openly licensed, built for
programmatic use, and needs no session or scraping. It publishes the h-index,
i10-index, work and citation counts, and the ISSNs of a journal:

```{r openalex-sources, eval=FALSE}
openalex <- jsonlite::fromJSON(
  "https://api.openalex.org/sources?search=Nature&per-page=1")
openalex$results[, c("display_name", "works_count", "cited_by_count")]
openalex$results$summary_stats
```

If you need SCImago's SJR quartiles specifically, save the table by hand in a
browser from `https://www.scimagojr.com/journalrank.php` and read it with
`utils::read.csv2()` — the separators are semicolons.

# Predicting Future h-index

`predict_h_index()` implements the model from Acuna et al. for predicting future
h-index values. The original model was calibrated on neuroscientists, so results
for other fields should be treated as illustrative rather than authoritative.

```{r predict-h-index, eval=run_examples}
predict_h_index("GAi23ssAAAAJ")
```

You can provide a custom list of top journals for the model's top-journal
predictor.

```{r predict-h-index-custom, eval=FALSE}
predict_h_index(
  id,
  journals = c("Nature", "Science", "Proceedings of the National Academy of Sciences")
)
```

# Formatting Publications for CVs

`format_publications()` returns publication strings that can be printed in
R Markdown, Quarto, or a CV workflow. The `author.name` argument highlights a
chosen author in bold after converting the supplied name to the abbreviated
format used by the output.

```{r format-publications-asis, results="asis", eval=run_examples}
format_publications("DO5oG40AAAAJ", "Guangchuang Yu") |>
  head() |>
  cat(sep = "\n\n")
```

For numbered output, print the returned vector directly.

```{r format-publications-print, eval=run_examples}
format_publications("DO5oG40AAAAJ", "Guangchuang Yu") |>
  head() |>
  print(quote = FALSE)
```

# Mirrors, Retries, and Caching

Use `set_scholar_mirror()` if you need to point the package at a compatible
Scholar mirror.

```{r mirror, eval=FALSE}
set_scholar_mirror("https://scholar.google.com")
```

All network requests go through `get_scholar_resp()`, which stores Google
Scholar session cookies and retries transient failures. `get_publications()`
also caches profile publication data with `R.cache` in a temporary cache
directory. Use `flush = TRUE` in `get_publications()` or functions that forward
to it when you need fresh data.

```{r flush-cache, eval=FALSE}
get_publications(id, flush = TRUE)
get_scholar_metrics(id, flush = TRUE)
```

## When Scholar refuses: servers, CI, and hosted apps

Google guards Scholar against automated clients, and how it responds depends on
where the request comes from. On your machine a profile page opens; on
shinyapps.io, a GitLab runner or a GitHub Actions job the same request can come
back as a sign-in redirect, a captcha, or HTTP 403. The package no longer
describes that as a bad ID: `get_scholar_resp()` inspects the address the request
finally landed on and the body it carries, then warns with the reason.

```{r refused, eval=FALSE}
get_profile("B7vSqZsAAAAJ")
#> Warning: Google Scholar returned a captcha page that needs JavaScript, so the
#>   response contains no data. ... Carry the session of a browser that opens the
#>   page with set_scholar_cookie(), ... or run it from a workstation.
```

Four ways out, in the order they are worth trying:

1. **Fetch outside the app.** Pull the data where a person browses, save it, and
   have the deployed app read the saved table. Scholar data changes slowly and
   an app rarely needs a live scrape per visit.
2. **Pace the requests.** `options(scholar_request_delay = 1)` (or
   `SCHOLAR_REQUEST_DELAY=1` in CI) puts a pause in front of every request, and
   rate limits already back off geometrically within
   `scholar_retry_max_delay`. Loops over many publications are what get an IP
   flagged.
3. **Present a session.** `set_scholar_cookie(file = "cookies.txt")`, or
   `SCHOLAR_COOKIE_FILE` on a server, reuses the session of a browser that opens
   the page. This is what author search now requires. A session is a credential:
   keep it out of the command history and out of version control, and note that
   the package sends it only to Google hosts, never to a mirror.
4. **Use another source.** For anything at scale, OpenAlex
   (`https://api.openalex.org`) is sanctioned, needs no session, and returns
   h-index and citation counts alongside the works.

The settings the package reads, each with an environment variable of the same
name in upper case: `scholar_site`, `scholar_user_agent`,
`scholar_accept_language`, `scholar_request_delay`, `scholar_retry_delay`,
`scholar_retry_max_delay`, `scholar_timeout`, `scholar_cookie`,
and `scholar_cookie_file`.

For large workflows, prefer fetching each scholar's publications once and then
reusing the returned data frame for multiple summaries.

```{r reusable-publications, eval=FALSE}
pubs <- get_publications(id)

recent <- subset(pubs, !is.na(year) & year >= 2020)
nrow(recent)
sum(recent$cites, na.rm = TRUE)
get_publication_metrics(recent)
```