Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Twitter API v2 Full-Archive Search #18

Merged
merged 9 commits into from
Dec 1, 2022

Conversation

dslucas
Copy link
Contributor

@dslucas dslucas commented Nov 29, 2022

As-is, Harassment Manager leverages the Enterprise Full-Archive Search API to fetch tweets directed at the logged-in user. This change enables developers to use either Enterprise or Twitter API v2 Full-Archive Search, which is the latest version of the Twitter API.

To implement this, we add the relevant request logic to twitter.middleware.ts. We "pack" the v2 response format into the Enterprise response format to enable developers to switch between Enterprise and v2 forwards and backwards, without breaking usage for existing users of the application.

Additional changes include:

  • Removing unused types and defining new ones for the v2 format
  • Changing fromDate and toDate to startDateTimeMs and endDateTimeMs and using formatting functions to accommodate differences in the timestamps expected by Enterprise vs. v2
  • Modifying server_config_template.json with a bearerToken field, which is necessary for using v2 Full-Archive Search
  • Updating the AppEngine runtime to nodejs12 to support some new operators, like flatMap
  • Updating documentation

We deployed and tested an instance of Harassment Manager with each API and the application's behavior is largely identical between both versions, with a couple minor differences:

  1. The APIs return slightly different sets of tweets because of differences in the granularity of the timestamp format expected by each API. This difference is usually no more than an additional 1-3 tweets in the v2 instance.
  2. The tweets are displayed in slightly different orders when sorted by "Priority". This is due to small differences in how we parse the tweet text, which causes some variation in the Perspective API scores for the text. We opened issue Stripped-out text is different when using v2 vs. Enterprise #19 to investigate.

We also noticed the Enterprise instance does not render some images that the v2 instance does. This seems more like an implementation issue on our end, rather than an API difference. We opened issue #17 to investigate.

sorensenjs
sorensenjs previously approved these changes Nov 29, 2022
docs/1_setup.md Outdated Show resolved Hide resolved
docs/2_development.md Show resolved Hide resolved
docs/2_development.md Outdated Show resolved Hide resolved
src/common-types.ts Show resolved Hide resolved
src/server/middleware/twitter.middleware.ts Show resolved Hide resolved
src/server/middleware/twitter.middleware.ts Outdated Show resolved Hide resolved
src/server/middleware/twitter.middleware.ts Outdated Show resolved Hide resolved
src/server/middleware/twitter.middleware.ts Show resolved Hide resolved
src/server/middleware/twitter.middleware.ts Outdated Show resolved Hide resolved
src/server/middleware/twitter.middleware.ts Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants