Stripped-out text is different when using v2 vs. Enterprise #19
Comments
dslucas
added a commit
that referenced
this issue
Dec 1, 2022
As-is, Harassment Manager leverages the Enterprise Full-Archive Search API to fetch tweets directed at the logged-in user. This change enables developers to use either Enterprise or Twitter API v2 Full-Archive Search, which is the latest version of the Twitter API. To implement this, we add the relevant request logic to `twitter.middleware.ts`. We "pack" the v2 response format into the Enterprise response format to enable developers to switch between Enterprise and v2 forwards and backwards, without breaking usage for existing users of the application. Additional changes include: - Removing unused types and defining new ones for the v2 format - Changing `fromDate` and `toDate` to `startDateTimeMs` and `endDateTimeMs` and using formatting functions to accommodate differences in the timestamps expected by Enterprise vs. v2 - Modifying `server_config_template.json` with a `bearerToken` field, which is necessary for using v2 Full-Archive Search - Updating the AppEngine runtime to `nodejs12` to support some new operators, like `flatMap` - Updating documentation We deployed and tested an instance of Harassment Manager with each API and the application's behavior is largely identical between both versions, with a couple minor differences: 1. The APIs return slightly different sets of tweets because of differences in the granularity of the timestamp format expected by each API. This difference is usually no more than an additional 1-3 tweets in the v2 instance. 2. The tweets are displayed in slightly different orders when sorted by "Priority". This is due to small differences in how we parse the tweet text, which causes some variation in the Perspective API scores for the text. We opened issue #19 to investigate. We also noticed the Enterprise instance does not render some images that the v2 instance does. This seems more like an implementation issue on our end, rather than an API difference. We opened issue #17 to investigate.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In PR #18, we added support for v2 Full-Archive Search and noticed small differences in the sort order of tweets in the Create Report page. This seems like it's due to small bugs in how we strip out entities like URLs, hashtags, etc. in Enterprise. We should investigate and make the text as consistent as possible.
The text was updated successfully, but these errors were encountered: