feat: CDP Internal events processor #27007

benjackwhite · 2024-12-18T09:43:58Z

Problem

Needs https://github.com/PostHog/posthog-cloud-infra/pull/3376

We want to trigger CDP destinations via "internal events".
This sets up the internal events stream and service to process it.

Changes

New consumer to consume from new topic
Proper event validation with Zod
Uses new hog function types for loading from the DB
Hooked up activity log (simple version)
Disables a bunch of messaging stuff for now (I will fix in follow up but given that is all pre-alpha code I don't want to waste time on it right now)
Infra PR for new topics
Removes sub-templates from the config area - this is now only doable via a direct load to the template
Internal destinations should have different filter validation

Follow up

Expose activity log notifications in the UI with inline editor
Allow properties to be marked as "advanced" so they don't show up by default
Add url and "human description" to server side activity log to make message delivery easier
Some sort of UI for seeing all configured internal notifcations
Make it so we can keep functions closed-source (only allow modifying inputs + filters) with the execution code pulled from the template

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

daibhin · 2024-12-18T17:00:40Z

plugin-server/src/cdp/types.ts

+export type HogFunctionTypeType =
+    | 'destination'
+    | 'transformation'
+    | 'internal-destination'


Did we say that the type would be more specific (e.g. error_tracking_alert) and just make the Kafka topic generic. I'm easy either way

Yeah I changed my mind :D There isn't any plus side that I could find tbh. But also I'm still playing with it to find the right mix of namings

plugin-server/src/cdp/schema.ts

# Conflicts: # posthog/migrations/max_migration.txt

posthog-bot · 2024-12-19T16:57:46Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-12-19T17:18:22Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-12-19T17:39:14Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-12-19T18:00:37Z

📸 UI snapshots have been updated

3 snapshot changes in total. 0 added, 3 modified, 0 deleted:

chromium: 0 added, 3 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-12-19T18:21:24Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-12-19T18:42:33Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

# Conflicts: # frontend/src/lib/constants.tsx # posthog/migrations/max_migration.txt

# Conflicts: # posthog/migrations/max_migration.txt

# Conflicts: # posthog/migrations/0535_alter_hogfunction_type.py

posthog-bot · 2024-12-23T07:51:24Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-12-23T08:11:31Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

sentry-io · 2024-12-27T10:09:06Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ TypeError: Object of type datetime is not JSON serializable /api/projects/{parent_lookup_project_id}/survey... View Issue
‼️ NoBrokersAvailable: NoBrokersAvailable /api/projects/{parent_lookup_team_id}/insights/ View Issue

_{Did you find this useful? React with a 👍 or 👎}

benjackwhite added 8 commits December 18, 2024 10:32

Started adding internal events topic

56c1588

Fixes

1e7c936

Fixes

66374cc

Fix up manager

38d25b9

Fixes

37a99ee

Fix up tests

68c2605

Fixes

b3606e3

Fix added internal events

28a8f54

benjackwhite changed the title ~~feat: Started adding internal events topic~~ feat: CDP Internal events processor Dec 18, 2024

Fix

e302623

benjackwhite marked this pull request as ready for review December 18, 2024 15:19

benjackwhite mentioned this pull request Dec 18, 2024

feat: error tracking alerting #26987

Closed

fix

3360444

daibhin reviewed Dec 18, 2024

View reviewed changes

benjackwhite added 11 commits December 19, 2024 11:30

Fixes

53d8405

Merge branch 'master' into feat/cdp-internal-events

d80ff23

Fix up types and add activity log

b635dbd

fixes

f0c858d

Fixes

de363f5

Fixes

4fc3919

Fixes

8afbcac

Fixes

b84b23d

Fixes

cdfa344

Fix up

90b8df0

Fixes

3638940

benjackwhite requested a review from a team December 19, 2024 14:07

benjackwhite added 4 commits December 19, 2024 15:23

Fixes

31a2e49

Fixes

0b7fbff

Merge branch 'master' into feat/cdp-internal-events

9716c5e

# Conflicts: # posthog/migrations/max_migration.txt

Fix - don't include choice in the DB side of things

776fe93

benjackwhite and others added 2 commits December 19, 2024 17:48

fixes

3e9b0b0

Update UI snapshots for chromium (2)

c0addcd

Update UI snapshots for chromium (2)

6499831

Update UI snapshots for chromium (2)

e83a9b2

Update UI snapshots for chromium (2)

0502553

Update UI snapshots for chromium (2)

8edb318

Update UI snapshots for chromium (2)

15a7139

benjackwhite added 3 commits December 20, 2024 13:25

Merge branch 'master' into feat/cdp-internal-events

0ac0f76

# Conflicts: # frontend/src/lib/constants.tsx # posthog/migrations/max_migration.txt

Merge branch 'master' into feat/drop-hog-function-type

9c805fc

# Conflicts: # posthog/migrations/max_migration.txt

Merge branch 'feat/drop-hog-function-type' into feat/cdp-internal-events

087a7a7

# Conflicts: # posthog/migrations/0535_alter_hogfunction_type.py

benjackwhite changed the base branch from master to feat/drop-hog-function-type December 20, 2024 12:31

benjackwhite added 2 commits December 20, 2024 14:10

Fix

a68a75f

Fix

a66b8b7

Base automatically changed from feat/drop-hog-function-type to master December 20, 2024 13:15

mariusandra approved these changes Dec 20, 2024

View reviewed changes

benjackwhite and others added 2 commits December 23, 2024 08:14

Merge branch 'master' into feat/cdp-internal-events

29070fa

Update UI snapshots for chromium (2)

b2e515d

Update UI snapshots for chromium (2)

b7266a1

benjackwhite mentioned this pull request Dec 27, 2024

CDP destinations for alerts #27164

Open

8 tasks

benjackwhite merged commit dd74b2c into master Dec 27, 2024
99 checks passed

benjackwhite deleted the feat/cdp-internal-events branch December 27, 2024 09:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: CDP Internal events processor #27007

feat: CDP Internal events processor #27007

benjackwhite commented Dec 18, 2024 •

edited

Loading

daibhin Dec 18, 2024

benjackwhite Dec 19, 2024

posthog-bot commented Dec 19, 2024

posthog-bot commented Dec 19, 2024

posthog-bot commented Dec 19, 2024

posthog-bot commented Dec 19, 2024

posthog-bot commented Dec 19, 2024

posthog-bot commented Dec 19, 2024

posthog-bot commented Dec 23, 2024

posthog-bot commented Dec 23, 2024

sentry-io bot commented Dec 27, 2024 •

edited

Loading

feat: CDP Internal events processor #27007

feat: CDP Internal events processor #27007

Conversation

benjackwhite commented Dec 18, 2024 • edited Loading

Problem

Changes

Follow up

Does this work well for both Cloud and self-hosted?

How did you test this code?

daibhin Dec 18, 2024

Choose a reason for hiding this comment

benjackwhite Dec 19, 2024

Choose a reason for hiding this comment

posthog-bot commented Dec 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 23, 2024

📸 UI snapshots have been updated

posthog-bot commented Dec 23, 2024

📸 UI snapshots have been updated

sentry-io bot commented Dec 27, 2024 • edited Loading

Suspect Issues

benjackwhite commented Dec 18, 2024 •

edited

Loading

sentry-io bot commented Dec 27, 2024 •

edited

Loading