Add support for Twitter API v2 Full-Archive Search

.gcloudignore

@@ -14,4 +14,7 @@
 .gitignore
 
 # Node.js dependencies:
-node_modules/
+node_modules/
+
+# Miscellaneous
+.angular/cache

app.yaml

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-runtime: nodejs10
+runtime: nodejs12
 service: default
 instance_class: F4_1G
 
@@ -25,7 +25,7 @@ automatic_scaling:
 
 # Required for min_idle_instances!
 inbound_services:
-- warmup
+  - warmup
 
 env_variables:
   NODE_ENV: production

docs/1_setup.md

@@ -18,40 +18,60 @@ of developers.
 
 ## 1. Get access to Twitter APIs
 
-**NOTE: The full suite of Twitter APIs the app uses require additional access
-beyond the default Twitter API [Essential access
-level](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api).
-The [Enterprise
-search](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview)
-additionally requires an [enterprise
-account](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview).**
-
-**NOTE: We plan to migrate the Enterprise Full-Archive Search API to the v2 Search Tweets
-in the future. We will update this documentation accordingly.**
-
 The app makes use of several Twitter APIs, including:
 
-- [The Enterprise Full-Archive Search
-  API](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview) to fetch
-  tweets directed at the logged in user
-- The v2 [blocks](https://developer.twitter.com/en/docs/twitter-api/users/blocks/introduction)
+- The [Enterprise Full-Archive Search
+  API](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview)
+  **or** the [v2 Full-Archive Search
+  endpoint](https://developer.twitter.com/en/docs/twitter-api/tweets/search/quick-start/full-archive-search)
+  to fetch tweets directed at the logged in user
+- The v2
+  [blocks](https://developer.twitter.com/en/docs/twitter-api/users/blocks/introduction)
   endpoint to block users on behalf of the authenticated user
-- The v2 [mutes](https://developer.twitter.com/en/docs/twitter-api/users/mutes/introduction)
+- The v2
+  [mutes](https://developer.twitter.com/en/docs/twitter-api/users/mutes/introduction)
   endpoint to mute users on behalf of the authenticated user
 - The v2 [hide
   replies](https://developer.twitter.com/en/docs/twitter-api/tweets/hide-replies/introduction)
   endpoint to hide replies on behalf of the authenticated user
 
 To support all this functionality, you'll need to [get access to the Twitter
-API](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api)
-and the Enterprise Full-Archive Search API.
+API](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api).
 
-Once granted, take note of the:
+**NOTE: The full suite of Twitter APIs the app uses require additional access
+beyond the default Twitter API [Essential access
+level](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api).
+The [Enterprise
+search](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview)
+API requires an [enterprise
+account](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview),
+while v2 Full-Archive Search requires [academic
+access](https://developer.twitter.com/en/products/twitter-api/academic-research).**
 
-- Account name, app key, and app secret for your Twitter API developer account
-- Username and password for your Enterpise Full-Archive Search API account
+Once granted access, take note of the:
 
-You'll need both sets of credentials later on.
+- Account name, app key, and app secret for your Twitter API developer account
+- If using the Enterprise Full-Archive Search API, the Username and password for
+  your enterprise account
+- If using the v2 Full-Archive Search endpoint, the Twitter bearer token for
+  your app
+
+You'll need these credentials later on.
+
+### Enterprise Full-Archive Search vs. v2 Full-Archive Search
+
+The tool is implemented in a way that either API can be used. While both APIs
+offer similar functionality, there are key differences in rate limits. We refer
+users to [Twitter's
+comparison](https://developer.twitter.com/en/docs/twitter-api/tweets/search/migrate)
+for more details. You may also see minute differences in:
+
+- Which tweets are fetched. This is due to differences in granularity for
+  timestamp format each API supports (YYYYMMDD for Enterprise and
+  YYYY-MM-DDTHH:mm:ssZ for v2).
+- The order the tweets are displayed when sorted by "Priority". This is due to
+  small differences in how the APIs return the tweet text, which causes some
+  variation in the Perspective API scores for the text.
 
 ## 2. Create a Google Cloud Platform (GCP) project
 

docs/2_development.md

@@ -145,16 +145,12 @@ The required fields are:
   be the server-side key created in [setup](1_setup.md) in GCP
   [Credentials](https://console.cloud.google.com/apis/credentials)
 - `cloudProjectId`: Your Google Cloud project ID
+- `twitterApiCredentials`: Your credentials for the Twitter APIs.
 
-The optional fields are:
+All together, your config should look something like one of the two configs
+below, with the relevant credentials and key values replaced.
 
-- `twitterApiCredentials`: Your credentials for the Twitter APIs. The server
-  expect this field to be an object with `accountName`, `username`, and
-  `password` fields for the Enterprise Search API and `appKey` and `appToken`
-  for the Standard API.
-
-All together, your config should look something like the config below, with the
-relevant credentials and key values replaced.
+### If using the Enteprise Full-Archive Search API:
 
 ```json
 {
@@ -166,12 +162,27 @@ relevant credentials and key values replaced.
     "accountName": "{TWITTER_API_ACCOUNT_NAME}",
     "username": "{TWITTER_API_USERNAME}",
     "password": "{TWITTER_API_PASSWORD}",
-    "appKey": "{APP_KEY}",
-    "appToken": "{APP_TOKEN}"
+    "appKey": "{TWITTER_APP_KEY}",
+    "appToken": "{TWITTER_APP_TOKEN}"
   }
 }
 ```
 
+### If using the v2 Full-Archive Search endpoint:
+
+````json
+{
+  "port": "3000",
+  "staticPath": "dist/harassment-manager",
+  "googleCloudApiKey": "{YOUR_GOOGLE_CLOUD_API_KEY}",
+  "cloudProjectId": "{YOUR_GOOGLE_CLOUD_PROJECTID}",
+  "twitterApiCredentials": {
+    "appKey": "{TWITTER_APP_KEY}",
+    "appToken": "{TWITTER_APP_TOKEN}",
+    "bearerToken": "{TWITTER_APP_BEARER_TOKEN}"
+  }
+}
+
 ## 7. (Optional) Enable Google Analytics
 
 If you created a Google Analytics property in [setup](1_setup.md) and want to
@@ -192,7 +203,7 @@ To build and run the app and a local development server, run
 
 ```shell
 npm run build:all:dev && npm run start:dev-server
-```
+````
 
 To build and run the app and a local production server, run
 

package.json

@@ -105,7 +105,7 @@
     "@types/jasmine": "^3.10.3",
     "@types/jasminewd2": "~2.0.10",
     "@types/jspdf": "^1.3.3",
-    "@types/node": "^17.0.21",
+    "@types/node": "17.0.10",
     "@types/supertest": "^2.0.11",
     "@typescript-eslint/eslint-plugin": "5.14.0",
     "@typescript-eslint/parser": "5.14.0",

src/app/social-media-item.service.ts

@@ -126,11 +126,11 @@ export class SocialMediaItemService {
   ): Observable<Tweet[]> {
     return from(
       this.twitterApiService.getTweets({
-        fromDate: formatTimestamp(startDateTimeMs),
-        // Subtract 1 minute from the end time because the Twitter API
-        // sometimes returns an error if we request data for the most recent
-        // minute of time.
-        toDate: formatTimestamp(endDateTimeMs - 60000),
+        startDateTimeMs,
+        // Subtract 1 minute from the end time because the Twitter API sometimes
+        // returns an error if we request data for the most recent minute of
+        // time.
+        endDateTimeMs: endDateTimeMs - 60000,
       })
     ).pipe(map((response: GetTweetsResponse) => response.tweets));
   }
@@ -166,22 +166,3 @@ export class SocialMediaItemService {
       .pipe(map(scores => ({ item, scores })));
   }
 }
-
-// Format a millisecond-based timestamp into a date format suitable for the
-// Twitter API, as defined in:
-// https://developer.twitter.com/en/docs/tweets/search/api-reference/enterprise-search
-function formatTimestamp(ms: number): string {
-  const date = new Date(ms);
-  const MM = date.getUTCMonth() + 1; // getMonth() is zero-based
-  const dd = date.getUTCDate();
-  const hh = date.getUTCHours();
-  const mm = date.getUTCMinutes();
-
-  return (
-    `${date.getFullYear()}` +
-    `${(MM > 9 ? '' : '0') + MM}` +
-    `${(dd > 9 ? '' : '0') + dd}` +
-    `${(hh > 9 ? '' : '0') + hh}` +
-    `${(mm > 9 ? '' : '0') + mm}`
-  );
-}

src/app/tweet-image/tweet-image.component.html

@@ -16,17 +16,17 @@
 
 <ng-container *ngIf="tweet && thumbnail">
   <span class="image-thumbnail"
-        *ngIf="tweet.extended_entities && tweet.extended_entities.media">
+        *ngIf="tweet.extended_entities?.media?.length">
     <img [class.blur]="shouldBlur"
          [src]="tweet.extended_entities.media[0].media_url" [alt]="shouldBlur ? 'Blurred image from tweet.' : 'Image from tweet.'">
   </span>
 </ng-container>
 <ng-container *ngIf="tweet && !thumbnail">
   <div class="full-image"
        [class.collapsed]="collapsed"
-       *ngIf="tweet.extended_entities && tweet.extended_entities.media">
+       *ngIf="tweet.extended_entities?.media?.length">
     <img *ngFor="let media of tweet.extended_entities.media"
-        [class.blur]="shouldBlur" 
+        [class.blur]="shouldBlur"
         [src]="media.media_url" [alt]="shouldBlur ? 'Blurred image from tweet.' : 'Image from tweet.'">
   </div>
 </ng-container>