[Bug] Missing custom metadata

[SteadyGiant]

tap-pendo doesn't ingest our custom Account and Visitor metadata fields, also called data mappings. I've tried Stitch and running tap-pendo myself; neither gets our custom metadata. I can get those data mappings with Fivetran or by calling the Pendo API directly with Python.

This is possibly related to #18.

Calling the API directly

Suppose I use this main.py script to request all of our Accounts data from Pendo's Accounts endpoint:

import requests

headers = {
    'x-pendo-integration-key': "redacted",
    'content-type': "application/json"
}
url = "https://app.pendo.io/api/v1/aggregation"
data = {
    "response": {
        "mimeType": "application/json"
    },
    "request": {
        "pipeline": [
            {
                "source": {
                    "accounts": None
                }
            }
        ]
    }
}

response = requests.post(url, json=data, headers=headers)
results = response.json()["results"]
for r in results:
    print(r)

I run it like this on the terminal (using Python 3.9):

python main.py

And this is a sample object in the printed response (I cut a few things out):

{
    'accountId': 'account1',
    'metadata': {
        'auto': {
            'firstvisit': 123,
	    'id': 'account1',
	    'idhash': 999,
	    'lastupdated': 456,
	    'lastvisit': 789
	}, 
        'agent': {
            'application_name': 'app1',
	    'customer_id': 'customer1'
	    'field1': 'value1',
	    'field2': 'value2',
        },
        'auto__323232': {
            'firstvisit': 123,
            'lastupdated': 456,
            'lastvisit': 789
        }
    }
}

The agent group contains the custom account metadata that we want tap-pendo to ingest.

Running tap-pendo locally

I install tap-pendo into a local virtualenv with Python 3.8.13, then run this:

tap-pendo/bin/tap-pendo --config config.json --catalog catalog.json

Note: I won't include its contents here for the sake of brevity, but in catalog.json I set tap-pendo to only ingest Accounts data.

Here's config.json:

{
    "include_anonymous_visitors": "false",
    "period": "dayRange",
    "start_date": "2022-08-01T00:00:00Z",
    "x_pendo_integration_key": "redacted"
}

Here's what tap-pendo outputs for the same account shown in the above "Calling the API directly" example:

{
    "type": "RECORD",
    "stream": "accounts",
    "record": {
        "account_id": "account1",
        "lastupdated": "date1",
        "metadata_auto": {
            "firstvisit": "date2",
            "id": "account1",
            "idhash": 999,
            "lastupdated": "date1",
            "lastvisit": "date3"
        }
    }
}

There is no agent group! tap-pendo skips our custom metadata. (So does Stitch, FWIW.)