feat(experiments): add mgmt cmd to generate experiment data

[andehen]

Problem

make it easy to generate data with known statistics.

It can be further improved, but it's already very useful so I would like to merge this to master as I use it a lot already.
Then we can iterate from there.

Changes

added mgmt cmd that generates events, so now you can do

 ./manage.py generate_experiment_data --type funnel --experiment-id funnel-test-1
 ./manage.py generate_experiment_data --type trends --experiment-id trends-test-1

to generate with the default configs.

Or you can do

./manage.py generate_experiment_data --type funnel --init-config funnel-conf.json
./manage.py generate_experiment_data --config exp-conf.json --experiment-id funnel-test

the json conf file can easily be edited to suite your test purpose.

the default currently looks like this:

{
  "number_of_users": 1000,
  "start_timestamp": "2024-12-03T07:04:51.286943",
  "end_timestamp": "2024-12-10T07:04:51.286949",
  "variants": {
    "control": {
      "weight": 0.5,
      "actions": [
        {
          "event": "$pageview",
          "count": 1,
          "probability": 0.75
        }
      ]
    },
    "test": {
      "weight": 0.5,
      "actions": [
        {
          "event": "$pageview",
          "count": 1,
          "probability": 1.0
        }
      ]
    }
  }
}

How did you test this code?

• ran the cmd and tested with this query in clickhouse

SELECT 
	properties_group_feature_flags['$feature/test-1'] as variant,
	event,
	count(*)
FROM events
WHERE mapContains(properties_group_feature_flags, '$feature/test-1)
GROUP BY all;

• used product analytics to visualize the funnel and verified that the numbers added up
• verified by looking at experiment results

[danielbachhuber]

Nice!

What do you think about including a few sample configurations too, so you don't necessarily have to futz with a JSON file each time?

[andehen]

Nice!

What do you think about including a few sample configurations too, so you don't necessarily have to futz with a JSON file each time?

Good idea! We can add the typical cases that we need during development.

[jurajmajerik]

Neither of the default commands work for me 🤔 I'm getting the following:

(env) ➜  posthog git:(mock-experiment-data) DEBUG=1 ./manage.py generate_experiment_data --type funnels --experiment-id funnel-test-1 
...
manage.py generate_experiment_data: error: unrecognized arguments: --type funnels
(env) ➜  posthog git:(mock-experiment-data) DEBUG=1 ./manage.py generate_experiment_data --type trends --experiment-id trends-test-1
...
manage.py generate_experiment_data: error: unrecognized arguments: --type trends

[andehen]

Neither of the default commands work for me 🤔 I'm getting the following:

(env) ➜  posthog git:(mock-experiment-data) DEBUG=1 ./manage.py generate_experiment_data --type funnels --experiment-id funnel-test-1 
...
manage.py generate_experiment_data: error: unrecognized arguments: --type funnels
(env) ➜  posthog git:(mock-experiment-data) DEBUG=1 ./manage.py generate_experiment_data --type trends --experiment-id trends-test-1
...
manage.py generate_experiment_data: error: unrecognized arguments: --type trends

yeah, I had forgotten to push the latest commits, sorry about that

[danielbachhuber]

💅 Fancy!

[jurajmajerik]

Works great!

Where would be a good place to put an example config file so it's easy to reference?

[andehen]

Works great!

Where would be a good place to put an example config file so it's easy to reference?

Locally you mean, or to share between us?

[jurajmajerik]

@andehen It's an open question - if I'm going to run this command, how do I know what the input config file should look like?

For example, we could add a help flag to the command, which will tell you where to find an example config (could be a github gist).

Or maybe we could store a couple of example config files somewhere in the posthog repo, so it's easy to reference them them locally.

[andehen]

@andehen It's an open question - if I'm going to run this command, how do I know what the input config file should look like?

For example, we could add a help flag to the command, which will tell you where to find an example config (could be a github gist).

Or maybe we could store a couple of example config files somewhere in the posthog repo, so it's easy to reference them them locally.

Ah, it's baked into the code so you generate one with the default values like this

./manage.py generate_experiment_data --type funnel --init-config funnel-conf.json

and then you can modify that as you like. There is also a -h option that explains this

DEBUG=1 ./manage.py generate_experiment_data -h