Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sending push notifications with 40k installations fails #4242

Closed
adammlevy opened this issue Oct 8, 2017 · 10 comments
Closed

Sending push notifications with 40k installations fails #4242

adammlevy opened this issue Oct 8, 2017 · 10 comments

Comments

@adammlevy
Copy link
Contributor

adammlevy commented Oct 8, 2017

Issue Description

I have a background job on my parse instance which fetches an XML feed every 1 minute and sends a push notification to a specific channel every time it detects new data from the feed. I have noticed that sometimes the push is sent and sometimes it is not sent. On all cases I see that the success block is called but when I look in the parse-dashboard I see the push in the sending state.

// push to channels which usually fails.  
Parse.Push.send({
		channels: [channel],
			data: { ...

Since I have no problem receiving push notifications on my dev instance (which only has 6 installations), on my prod instance I suspected this had to do with the fact that I am querying 40k installations where each channels record may contain 20 channels.
To prove the number of installations was the issue, I passed a query in the push request instead of using channels which would force the push to only be sent to my device

// Query pushes to only my device
var queryInstall = new Parse.Query(Parse.Installation);
queryInstall.equalTo('deviceType', 'ios');
queryInstall.equalTo('installationId', [MY_INSTALLATION_ID]);

Parse.Push.send({
	where: queryInstall,
	data: { ...

This query made every push a success to my device but if I remove the query and use channels instead no pushes are received. Note that although there are 40k installations, not all of them subscribe to notifications.

Steps to reproduce

Send a push to a channel observed by multiple Installation objects.

Expected Results

The push gets sent to all users.

Actual Outcome

The push notification does not get sent.

Environment Setup

  • Server

    • parse-server version : 2.6.3
    • Operating System: N/A
    • Hardware: N/A
    • Localhost or remote server? : Heroku (1 Web Dyno: 2x, 1 Worker Dyno: 2x)
  • Database

    • MongoDB version: 3.2.10
    • Storage engine: - Hardware:
    • Localhost or remote server?: mLab, Shared cluster 1GB

Logs/Trace

 verbose: REQUEST for [POST] /parse/push: {
   "where": {
     "channels": "CHPIncident-WestLA"
   },
  "data": {
     "alert": {
       "title": "CHP Incident West LA",
       "body": "West LA: traffic Collision-unknown injury  at I405 N   National Blvd"
     },
     "chpIncident": {
       "centerId": "LAHB",
       "dispatchId": "LACC",
       "logId": "171008LA01569",
       "logTime": "Oct  8 2017 10:58AM",
       "incidentId": "171008LA01569Oct  8 2017 10:58AM",
       "logType": "1183-Trfc Collision-Unkn Inj",
       "location": "I405 N / National Blvd",
       "drivingText": " traffic Collision-unknown injury at i 4 o 5 north   National boulevard ",
       "drivingTextSoCal": " traffic Collision-unknown injury at the 4 o 5 north   National boulevard ",
       "locationDesc": "NB 405 JSO NATIONAL",
       "area": "West LA",
       "broadcastMessage": "West LA: traffic Collision-unknown injury  at I405 N   National Blvd",
       "lat": 34.02685,
       "lon": -118.429908,
       "logDetails": [
         {
           "time": "Oct  8 2017 11:00AM",
           "details": "[2] POSS DUPE LG/1556 -- RP STATED DESC OF VEHS ON LG/1556 WAS DIFFERENT FRM HER TC"
         },
         {
           "time": "Oct  8 2017 10:59AM",
           "details": "[1] BLK NISS ALT, BLK Q5, UNK DESC 3RD VEH"
         }
       ],
       "logUnits": []
     },
     "expiration_time": "2017-10-08T22:02:15.039Z",
     "sound": "chp_incident.caf",
     "PushType": "CHPIncident"
   }

2017-10-08T18:02:17.681160+00:00 app[web.1]: } method=POST, url=/parse/push, host=carr.herokuapp.com, connection=close, user-agent=node-XMLHttpRequest, Parse/js1.9.2 (NodeJS 6.11.4), accept=*/*, content-type=text/plain, x-request-id=3cd4fcd9-ef31-4039-8507-22e00ff14637, x-forwarded-for=54.205.54.184, x-forwarded-proto=https, x-forwarded-port=443, via=1.1 vegur, connect-time=1, x-request-start=1507485737673, total-route-time=0, content-length=1307, channels=CHPIncident-WestLA, title=CHP Incident West LA, body=West LA: traffic Collision-unknown injury  at I405 N   National Blvd, centerId=LAHB, dispatchId=LACC, logId=171008LA01569, logTime=Oct  8 2017 10:58AM, incidentId=171008LA01569Oct  8 2017 10:58AM, logType=1183-Trfc Collision-Unkn Inj, location=I405 N / National Blvd, drivingText= traffic Collision-unknown injury at i 4 o 5 north   National boulevard , drivingTextSoCal= traffic Collision-unknown injury at the 4 o 5 north   National boulevard , locationDesc=NB 405 JSO NATIONAL, area=West LA, broadcastMessage=West LA: traffic Collision-unknown injury  at I405 N   National Blvd, lat=34.02685, lon=-118.429908, logDetails=[time=Oct  8 2017 11:00AM, details=[2] POSS DUPE LG/1556 -- RP STATED DESC OF VEHS ON LG/1556 WAS DIFFERENT FRM HER TC, time=Oct  8 2017 10:59AM, details=[1] BLK NISS ALT, BLK Q5, UNK DESC 3RD VEH], logUnits=[], expiration_time=2017-10-08T22:02:15.039Z, sound=chp_incident.caf, PushType=CHPIncident


RESPONSE from [POST] /parse/push: {
   "headers": {
     "X-Parse-Push-Status-Id": "wtqQ4H2N5p"
   },
   "response": {
     "result": true
   }
 } X-Parse-Push-Status-Id=wtqQ4H2N5p, result=true
 verbose: _PushStatus wtqQ4H2N5p: sending push to 267 installations

@flovilmart
Copy link
Contributor

Can’t you provide the logs when running with VERBOSE=1? Do you have proper indexes in place?

@adammlevy
Copy link
Contributor Author

adammlevy commented Oct 8, 2017

@flovilmart included log with push request and response. The only index I have on the Installation record is {"channels" : 1}

@flovilmart
Copy link
Contributor

There should be more logs than that, notably from the push adapter

@adammlevy
Copy link
Contributor Author

Sorry forgot to include that. For all the successful pushes I am seeing node-pre-gyp verb parse-server-push-adapter APNS APNS transmitted to [DEVICE_TOKEN] and for pushes that failed node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device [DEVICE_TOKEN]

@flovilmart
Copy link
Contributor

There should be the detailed error message as well, wether it’s a bad payload or a bad device token.

@adammlevy
Copy link
Contributor Author

After simplifying the payload by removing the chpIncident param (which should not be there in the first place) I am receiving more notifications on my device. I didn't have any problems sending that large of a payload when the push query only targeted my device.

@adammlevy
Copy link
Contributor Author

Ok, I just noticed this error in the log which I have never seen before:

APNS error transmitting to device %s with error %s 60c0a4a22e0c3a4f5a472ccc90df8d1da6bc115da64e60f7276f35b200348e18 { VError: apn write failed: Not receiving Ping response after 150000 ms
2017-10-08T20:57:11.958304+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.stream.on.err (/app/node_modules/apn/lib/client.js:105:21)
2017-10-08T20:57:11.958339+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:96:13)
2017-10-08T20:57:11.958499+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.emit (events.js:188:7)
2017-10-08T20:57:11.958602+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at _connection._streamIds.forEach (/app/node_modules/apn/lib/protocol/endpoint.js:155:16)
2017-10-08T20:57:11.958720+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at Array.forEach (native)
2017-10-08T20:57:11.958890+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at EventEmitter.close [as _close] (/app/node_modules/apn/lib/protocol/endpoint.js:139:33)
2017-10-08T20:57:11.959063+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:101:20)
2017-10-08T20:57:11.959251+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at TLSSocket.emit (events.js:188:7)
2017-10-08T20:57:11.959476+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at _handle.close (net.js:497:12)
2017-10-08T20:57:11.959597+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at TCP.done [as _onclose] (_tls_wrap.js:332:7)
2017-10-08T20:57:11.959784+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   jse_shortmsg: 'apn write failed: Not receiving Ping response after 150000 ms',
2017-10-08T20:57:11.959975+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   jse_info: {},
2017-10-08T20:57:11.960136+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   message: 'apn write failed: Not receiving Ping response after 150000 ms',
2017-10-08T20:57:11.960327+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   stack: 'VError: apn write failed: Not receiving Ping response after 150000 ms\n    at Stream.stream.on.err (/app/node_modules/apn/lib/client.js:105:21)\n    at emitOne (events.js:96:13)\n    at Stream.emit (events.js:188:7)\n    at _connection._streamIds.forEach (/app/node_modules/apn/lib/protocol/endpoint.js:155:16)\n    at Array.forEach (native)\n    at EventEmitter.close [as _close] (/app/node_modules/apn/lib/protocol/endpoint.js:139:33)\n    at emitOne (events.js:101:20)\n    at TLSSocket.emit (events.js:188:7)\n    at _handle.close (net.js:497:12)\n    at TCP.done [as _onclose] (_tls_wrap.js:332:7)' }

@flovilmart
Copy link
Contributor

Just to rewind a bit, sending a simple payload to a channels target works correctly?

@adammlevy
Copy link
Contributor Author

Yes. Now that I have simplified the payload I can see more pushes coming in and the push adapter always logs whether it was a successful push or a failed push. While I was sending the large payload I would never see the push adapter log so I wonder if it was just silently failing?

Regarding the issue above, I think I found the solution in issue #90

@flovilmart
Copy link
Contributor

Perhaps APNS was not happy with those large payloads, or the node apn module don’t support those properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants