[ Windows ] empty response ; something was probably wrong with the request

[dmanzano2193]

trends <- tk_posts(scope = "trends", n = 200)

! [2020-11-23 20:38:19] t- (not found)
Warning message:
In get_data(url, parse = T, ...) :
empty response ; something was probably wrong with the request

[benjaminguinaudeau]

Hi Diego,

I'm sorry that you are experiencing this problem and thanks for sharing it.

Here are a couple of questions to help me identify the problem.

Can you please confirm that you run the following lines?

tk_init()
tk_auth(cookie= "the cookie you get from tiktok.com")

Can you try with other endpoints and tell me if you experience the same errors?

user <- tk_info(scope = "user", query = "willsmith")
hashtag <- tk_info(scope = "hashtag", query = "maincharacter")
user_post <- tk_posts(scope = "user", query = "willsmith", n = 40)
hashtag <- tk_posts(scope = "hashtag", query = "maincharacter", n = 25)

Finally, it would help if you could print the output of sessionInfo()

Thank you!

[dmanzano2193]

I was testing with the code you sent me, but I keep having the same problem. I had previously included the tk_init and the tk_auth, just as you said. However, when I test with Will Smith, only one post comes out. I attach the session info.

- Session info ------------------------------------------------------------------------------- setting value version R version 4.0.2 (2020-06-22) os Windows 10 x64 system x86_64, mingw32 ui RStudio language (EN) collate Spanish_El Salvador.1252 ctype Spanish_El Salvador.1252 tz America/Guatemala date 2020-11-23

` Packages -----------------------------------------------------------------------------------
package * version date lib source
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.2)
cli 2.0.2 2020-02-28 [1] CRAN (R 4.0.2)
colorspace 1.4-1 2019-03-18 [1] CRAN (R 4.0.2)
crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.2)
curl 4.3 2019-12-02 [1] CRAN (R 4.0.2)
data.table 1.13.0 2020-07-24 [1] CRAN (R 4.0.2)
digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.3)
dplyr 1.0.2 2020-08-18 [1] CRAN (R 4.0.2)
ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.2)
evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.2)
fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.2)
generics 0.0.2 2018-11-29 [1] CRAN (R 4.0.2)
ggplot2 3.3.2 2020-06-19 [1] CRAN (R 4.0.2)
glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.2)
htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.2)
htmlwidgets 1.5.1 2019-10-08 [1] CRAN (R 4.0.2)
httr 1.4.2 2020-07-20 [1] CRAN (R 4.0.2)
jsonlite 1.7.1 2020-09-07 [1] CRAN (R 4.0.3)
knitr 1.29 2020-06-23 [1] CRAN (R 4.0.2)
lattice 0.20-41 2020-04-02 [2] CRAN (R 4.0.2)
lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.2)
magrittr 1.5 2014-11-22 [1] CRAN (R 4.0.2)
Matrix 1.2-18 2019-11-27 [2] CRAN (R 4.0.2)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.2)
NLP 0.2-0 2018-10-18 [1] CRAN (R 4.0.0)
openxlsx 4.1.5 2020-05-06 [1] CRAN (R 4.0.2)
pillar 1.4.6 2020-07-10 [1] CRAN (R 4.0.2)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.2)
purrr 0.3.4 2020-04-17 [1] CRAN (R 4.0.2)
R6 2.4.1 2019-11-12 [1] CRAN (R 4.0.2)
rappdirs 0.3.1 2016-03-28 [1] CRAN (R 4.0.2)
Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.2)
reticulate * 1.16 2020-05-27 [1] CRAN (R 4.0.2)
rlang 0.4.7 2020-07-09 [1] CRAN (R 4.0.2)
rlist 0.4.6.1 2016-04-04 [1] CRAN (R 4.0.3)
rmarkdown 2.3 2020-06-18 [1] CRAN (R 4.0.2)
rstudioapi 0.11 2020-02-07 [1] CRAN (R 4.0.2)
scales 1.1.1 2020-05-11 [1] CRAN (R 4.0.2)
sessioninfo * 1.1.1 2018-11-05 [1] CRAN (R 4.0.3)
slam 0.1-47 2019-12-21 [1] CRAN (R 4.0.0)
stringi 1.4.6 2020-02-17 [1] CRAN (R 4.0.0)
stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.2)
tibble 3.0.3 2020-07-10 [1] CRAN (R 4.0.2)
tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.2)
tiktokr * 0.0.0.9000 2020-11-24 [1] Github (d47e692)
tm 0.7-7 2019-12-12 [1] CRAN (R 4.0.2)
vctrs 0.3.4 2020-08-29 [1] CRAN (R 4.0.2)
withr 2.2.0 2020-04-20 [1] CRAN (R 4.0.2)
wordcloud2 0.2.1 2018-01-03 [1] CRAN (R 4.0.2)
xfun 0.16 2020-07-24 [1] CRAN (R 4.0.2)
xml2 1.3.2 2020-04-23 [1] CRAN (R 4.0.2)
yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.2)
zip 2.1.1 2020-08-27 [1] CRAN (R 4.0.2)

[1] C:/Users/Diego Alejandro/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.2/library`
And this is the error:

> user_post <- tk_posts(scope = "user", query = "willsmith", n = 40) i [2020-11-23 21:12:31] u-willsmith (no videos) Warning message: In get_data(url, parse = T, ...) : empty response ; something was probably wrong with the request

[benjaminguinaudeau]

Alright. Here are a few more diagnostics:

get_signature("test") # you should get a string as result starting with "test&_signature=_" and followed by a very long token 

verify <- get_current_verify() # you should get a string starting with "verify_" followed by a long token

url <- glue::glue("https://m.tiktok.com/api/item_list/?aid=1988&id=6727327145951183878&secUid=MS4wLjABAAAA8ezUaW4ecJX222ObGXxt07F9BIh4QH3-g1P1DHyChT2LLi2cn-vAE2R53-H672ZO&count=50&maxCursor=0&minCursor=0&sourceType=8&verifyFp={get_current_verify()}&user_agent=Mozilla%2F5.0+(Macintosh%3B+Intel+Mac+OS+X+10_14_7)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F86.0.4240.198+Safari%2F537.36")

out <- get_data(url, parse = T)

[dmanzano2193]

I already tried that code, and it returns only a post from the user that I am trying to collect data. And this is what appears to me.

`get_signature("test")
[1] "test&_signature="

verify <- get_current_verify()
url <- glue::glue("https://m.tiktok.com/api/item_list/?aid=1988&id=6727327145951183878&secUid=MS4wLjABAAAA8ezUaW4ecJX222ObGXxt07F9BIh4QH3-g1P1DHyChT2LLi2cn-vAE2R53-H672ZO&count=50&maxCursor=0&minCursor=0&sourceType=8&verifyFp={get_current_verify()}&user_agent=Mozilla%2F5.0+(Macintosh%3B+Intel+Mac+OS+X+10_14_7)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F86.0.4240.198+Safari%2F537.36")

out <- get_data(url, parse = T)
Warning message:
In get_data(url, parse = T) :
empty response ; something was probably wrong with the request
View(user_post)`

[benjaminguinaudeau]

The issue seems to come from running puppeteer on windows.

A friend could reproduce it on a windows computer. I will look into it tomorrow and will hopefully find a quick fix. Sorry for this...

[dmanzano2193]

No do not worry. I really appreciate your help and time, it is important to me. I'll be waiting for your responses tomorrow, and thanks for creating the package.

[benjaminguinaudeau]

Hier is some update.

I know where the issue is, but I don't know how to fix this. It's happening in a python script that runs in the background and which is shared with the python tiktok-api. When trying to run the python API, I observe the exact same problem.

I'm working on an alternative way based on docker, but this will take a couple of days before being operational.

Stay tuned; I'll update you as soon as I have something working on windows.

[dmanzano2193]

Good morning, Benjamin. I am very sorry I did not answer you before, but I had several pending activities. First of all, I want to thank you very much for the time you have taken to help me solve the problem. Second, I'll be on the lookout for any progress you've made. Thanks for everything.

[mancebral]

Hello, I have the same problem exactly, but also on MacOs, I mean, I tried both OS and MacOs also returned me that problem Many thanks!!!

[benjaminguinaudeau]

Does get_signature("test") also return "test&_signature=" ?

[mancebral]

sorry, I check again on MacOs and now is running well. On windows the same problem with the answer "test&_signature=" to get_signature("test"). Thanks again!!!!

[benjaminguinaudeau]

Can you please start another issue, because the problem of Diego is specific to windows? I'll help you there

[benjaminguinaudeau]

Dieog, I'm still working on a solution for windows. I've made it worked on a friends' computer ; I'm now adapting tiktokr. Thank you for your patience again.

[dmanzano2193]

Thank you so much, Benjamin. Dont worry, I wait.

[benjaminguinaudeau]

Hi Diego,

We've pushed an update which works on windows computer now. You'll need to install docker though. More info here: https://github.com/benjaminguinaudeau/tiktokr#using-tiktokr-with-docker

This is kind of experimental and hard to test, because I don't have any windows computer ; so feel free to shared any issue here, which could help us to improve the code

[dmanzano2193]

Hi, Benjamin. Thanks a lot for all your help, I am very grateful. Note that initially the code served me and I was able to extract data from Will Smith, but then, when I tried with another account, it gives me this error from Docker.
I already tried to fix it, but couldn't.

user_posts <- tk_posts(scope = "user", query = "willsmith", n = 50) Container was stopped. Starting container Error in curl::curl_fetch_memory(url, handle = handle) : Failed to connect to localhost port 32768: Connection refused Además: Warning message: In system("docker start tiktoksignature", intern = T) : comando ejecutado 'docker start tiktoksignature' tiene estatus 1 Error in curl::curl_fetch_memory(url, handle = handle) : Could not resolve host: localshost Además: Warning messages: 1: In system("docker stop tiktoksignature", intern = T) : comando ejecutado 'docker stop tiktoksignature' tiene estatus 1 2: In system("docker start tiktoksignature", intern = T) : comando ejecutado 'docker start tiktoksignature' tiene estatus 1 Error: $ operator is invalid for atomic vectors

[dmanzano2193]

And when I give tk_auth (docker = TRUE) I get the following message:

TIKTOK_DOCKER found. Tiktokr will use a docker container to sign the urls Tiktok cookie already specified in .RenvironUser agent already specified in.Renviron``

But then happen this:

> tk_init() v Docker version 19.03.13, build 4484c46d9d ! Unable to found image tiktoksignature; pull it fromgithub.com/benjaminguinaudeau/tiktok_signature`
This will take some time (at least 5 minutes)cloning into 'C:\Users\Diego Alejandro\AppData\Local\Temp\RtmpqcHEr3\file329057831146'...
Receiving objects: 4% (1/23), 5 kb
Receiving objects: 13% (3/23), 5 kb
Receiving objects: 21% (5/23), 5 kb
Receiving objects: 34% (8/23), 5 kb
Receiving objects: 43% (10/23), 5 kb
Receiving objects: 52% (12/23), 5 kb
Receiving objects: 65% (15/23), 5 kb
Receiving objects: 73% (17/23), 5 kb
Receiving objects: 82% (19/23), 5 kb
Receiving objects: 91% (21/23), 5 kb
Receiving objects: 100% (23/23), 5 kb, done.
"docker build" requires exactly 1 argument.
See 'docker build --help'.

Usage: docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile
x Unable to install image tiktoksignature Creating container tiktoksignatureand hosting plumber-API on port localhost:32768 x Unable to create containertiktoksignature`
Container was stopped. Starting container
Error in curl::curl_fetch_memory(url, handle = handle) :
Failed to connect to localhost port 32768: Connection refused
Además: Warning messages:
1: In system("docker run -dt -p 32768:6543 --name tiktoksignature tiktoksignature:latest", :
comando ejecutado 'docker run -dt -p 32768:6543 --name tiktoksignature tiktoksignature:latest' tiene estatus 125
2: In system("docker start tiktoksignature", intern = T) :
comando ejecutado 'docker start tiktoksignature' tiene estatus 1
Error in curl::curl_fetch_memory(url, handle = handle) :
Could not resolve host: localshost
Además: Warning messages:
1: In system("docker stop tiktoksignature", intern = T) :
comando ejecutado 'docker stop tiktoksignature' tiene estatus 1
2: In system("docker start tiktoksignature", intern = T) :
comando ejecutado 'docker start tiktoksignature' tiene estatus 1
Error: $ operator is invalid for atomic vectors

user_posts <- tk_posts(scope = "user", query = "willsmith", n=50)
Container was stopped. Starting container
Error in curl::curl_fetch_memory(url, handle = handle) :
Failed to connect to localhost port 32768: Connection refused
Además: Warning message:
In system("docker start tiktoksignature", intern = T) :
comando ejecutado 'docker start tiktoksignature' tiene estatus 1
Error in curl::curl_fetch_memory(url, handle = handle) :
Could not resolve host: localshost
Además: Warning messages:
1: In system("docker stop tiktoksignature", intern = T) :
comando ejecutado 'docker stop tiktoksignature' tiene estatus 1
2: In system("docker start tiktoksignature", intern = T) :
comando ejecutado 'docker start tiktoksignature' tiene estatus 1
Error: $ operator is invalid for atomic `vectors``

[benjaminguinaudeau]

Thank you for your patience! This is good feedback and very helpful to observe these errors.

The docker container cannot build for some reason. All other errors come from that point ; I'll build a trigger to make sure the code breaks if the image is not built.

Can you run docker ps -a (in R system("docker ps -a")) and docker images(in R system("docker images") ?

[dmanzano2193]

I already tried, Benjamin, but I get the following message. I ran the script in the R Studio console.

docker ps -a (in R system("docker ps -a")) and docker images(in R system("docker images") Error: unexpected symbol in "docker ps"

[dmanzano2193]

I dont know if the problem will be Docker because I didnt have any container created.

[benjaminguinaudeau]

that's weird. Can you try to run in Rsystem("docker -v"); if this does not produce any output or send an error, this means that docker is not installed/not installed where it should be installed

[dmanzano2193]

[dmanzano2193]

I send you my screenshot.

[benjaminguinaudeau]

This looks good.
But you should run system('docker ps -a'), so that docker ps -a is run a bash/powershell compiler .
Can you try it?

[dmanzano2193]

Yes, I send you the screenshot with the new code:

[benjaminguinaudeau]

Ok. no container were created. I suspect something went run while building the image of the container.

Can you run the following chunk:

folder <- tempfile()
git2r::clone("https://github.com/benjaminguinaudeau/tiktok_signature.git", local_path = folder)
system(glue::glue("docker build {folder} -t tiktoksignature"))

[dmanzano2193]

Ok, this is the image:

[benjaminguinaudeau]

Ok: Can you please try:

system(glue::glue("docker build -t tiktoksignature {folder}"))

[dmanzano2193]

I send you:

[benjaminguinaudeau]

What does your folder variable look like? Is there any space in the path that should be escaped?

[dmanzano2193]

This is looks like my folder:

[benjaminguinaudeau]

Yes, the spece between 'Diego' and 'Alejandro' must be escaped. You should do:

system(glue::glue("docker build -t tiktoksignature '{folder}'"))

[dmanzano2193]

Thank you, this is the message:

[dmanzano2193]

Yeah, i've already put the new chunk:

[benjaminguinaudeau]

reupdated (sorry, this an issue with the path and I'm not familiar with how to escape path in windows)

[dmanzano2193]

Ok, I still appreciate all the help and support you gave me. Too bad it couldn't be fixed.

[benjaminguinaudeau]

I'll look into it tomorrow and try this specific case with spaces in paths. If I get a fix, I'll let you know.

[benjaminguinaudeau]

Hi Diego,
We've tested on windows with a path including a space and this seem to be the way to fix it:

system(glue::glue('docker build "{normalizePath(folder)}" -t tiktoksignature'))

Let me know about the result if you give it a try.

[dmanzano2193]

Good night, Benjamin. I'm sorry it took me so long to answer, but I have been to several activities. Notice that I tried to use the command, but it didn't work for me. I don't know if Docker is the problem, but no problem. I want to ask you if you know how to extract data from Facebook as well.