Interceptors OpenTelemetry: Add Docker Compose and the sample more comprehensive

interceptors-opentelemetry/README.md

@@ -1,28 +1,122 @@
 # OpenTelemetry Interceptors
 
-This sample features `@temporalio/interceptors-opentelemetry`, which uses [Interceptors](https://docs.temporal.io/typescript/interceptors) to add tracing of Workflows and Activities with [opentelemetry](https://opentelemetry.io/).
+This sample demonstrates how to configure the Temporal SDK to emit metrics and
+tracing spans through an OpenTelemetry collector.
 
-### Running this sample
+It features the `@temporalio/interceptors-opentelemetry` package, which uses
+[Interceptors](https://docs.temporal.io/develop/typescript/interceptors) to
+propagate the [OpenTelemetry](https://opentelemetry.io/) tracing context from a
+Client to Workflows, to Child Workflows, and up to Activities.
+
+This sample also contains a Docker Compose file that can be used to demonstrate
+an OpenTelemetry setup that integrates both Worker metrics and Workflow traces
+collection.
+
+## Running the sample
+
+### Preparation
+
+1. Make sure you have a local [Temporal Server](https://github.com/temporalio/cli/#installation) running:
+
+   ```sh
+   temporal server start-dev
+   ```
+
+2. (Optional) To use the OpenTelemetry collector, run the Docker Compose file:
+
+   ```sh
+   docker compose up -d
+   ```
+
+3. Install NPM dependencies:
+
+   ```sh
+   npm install   # or `pnpm` or `yarn`
+   ```
+
+### Output telemetry records to the console
+
+By default, the project is configured to output tracing data to the console.
+This is convenient for demonstation purpose.
+
+To run the sample:
 
-1. `temporal server start-dev` to start [Temporal Server](https://github.com/temporalio/cli/#installation).
-1. `npm install` to install dependencies.
 1. `npm run start.watch` to start the Worker.
-1. In another shell, `npm run workflow` to run the Workflow.
+2. In another shell, `npm run workflow` to run the Workflow.
 
-Example output:
+You will observe in your console various telemetry records:
 
 ```
-Hello, Temporal!
 {
-  traceId: '74bf8e700df7dd3d8e140feebe70927b',
-  parentId: undefined,
-  name: 'StartWorkflow:example',
-  id: 'ab6e93cce7ed7a15',
-  kind: 0,
-  timestamp: 1641749671800859,
-  duration: 515453,
-  attributes: { run_id: '00cf070e-b691-4943-a8e9-9696c1baef4a' },
-  status: { code: 1 },
-  events: []
+  resource: { attributes: { 'service.name': 'interceptors-sample' } },
+  traceId: '8613431a77bcf95cdfcbbe40f2cdc934',
+  id: '15f2ca795e852236'
+  parentId: undefined
+  name: 'RunWorkflow:example',
+  [...]
+}
+{
+  resource: { attributes: { 'service.name': 'interceptors-sample' } },
+  traceId: '8613431a77bcf95cdfcbbe40f2cdc934',
+  id: '945c3e4ee7ae9b4d'
+  parentId: '15f2ca795e852236',
+  name: 'StartActivity:greet',
+  [...]
+}
+{
+  resource: { attributes: { 'service.name': 'interceptors-sample' } },
+  traceId: '8613431a77bcf95cdfcbbe40f2cdc934',
+  parentId: "945c3e4ee7ae9b4d"
+  id: '056ec5cce08a1796'
+  name: 'RunActivity:greet',
+  [...]
 }
 ```
+
+The following subsections describe other configurations that you may experiment with.
+
+### Expose native runtime metrics as a Prometheus endpoint
+
+To configure the sample to expose native runtime metrics as Prometheus endpoints:
+
+1. In file `instrumentation.ts`:
+   1. In function `setupTraceExporter()`, comment out all blocks;
+   2. In function `setupMetricReader()`, uncomment block `(4)`, and comment out all other blocks;
+2. In file `worker.ts`:
+   1. In function `initializeRuntime()`, uncomment block `(2)`, and comment out all other blocks;
+3. `npm run start.watch` to start the Worker.
+4. In another shell, `npm run workflow` to run the Workflow.
+
+To view metrics:
+
+1. Point a web browser on http://0.0.0.0:9091 to view native runtime metrics.
+2. Point a web browser on http://0.0.0.0:9092 to view Node's metrics (there might possibly be none).
+
+### Export telemetry data to an OpenTelemetry collector
+
+To configure the sample to send metrics and tracing data to an OpenTelemetry collector:
+
+1. In file `instrumentation.ts`:
+   1. In function `setupTraceExporter()`, uncomment block `(2)`, and comment out all other blocks;
+   2. In function `setupMetricReader()`, uncomment block `(2)`, and comment out all other blocks;
+2. In file `worker.ts`:
+   1. In function `initializeRuntime()`, uncomment block `(1)`, and comment out all other blocks;
+3. `npm run start.watch` to start the Worker.
+4. In another shell, `npm run workflow` to run the Workflow.
+
+To view collected traces using Jaeger UI (requires the use of the provided Docker Compose file):
+
+1. Point a web browser on http://0.0.0.0:16686.
+2. In the search panel on the left side, under the _Service_ drop down menu, select "interceptors-sample",
+   then click on "Find Traces".
+
+To view collected traces using Zipkin (requires the use of the provided Docker Compose file)
+
+1. Point a web browser on http://0.0.0.0:9411.
+2. In the query bar aat the top, search for `serviceName=interceptors-sample`,
+   then click on "Run Query".
+
+To view collected metrics using Prometheus (requires the use of the provided Docker Compose file)
+
+1. Point a web browser on http://0.0.0.0:9090.
+2. In the query field, search for `temporal_sticky_cache_size` or similar.

interceptors-opentelemetry/docker-compose.yml

@@ -0,0 +1,40 @@
+name: opentelemetry
+
+services:
+  jaeger:
+    image: jaegertracing/all-in-one:latest
+    ports:
+      - '16686:16686' # Web UI
+      - '14268:14268' # HTTP Port for spans
+      - '6831:6831/udp' # UDP port for Jaeger agent
+
+  zipkin:
+    image: openzipkin/zipkin:latest
+    environment:
+      - JAVA_OPTS=-Xms1024m -Xmx1024m -XX:+ExitOnOutOfMemoryError
+    restart: always
+    ports:
+      - '9411:9411'
+
+  otel-collector:
+    image: otel/opentelemetry-collector:latest
+    command: ['--config=/etc/otel-collector-config.yaml']
+    volumes:
+      - ./etc/otel-collector-config.yaml:/etc/otel-collector-config.yaml
+    ports:
+      - '13133:13133' # health_check extension
+      - '4317:4317' # OTLP gRPC receiver
+      - '4318:4318' # OTLP HTTP receiver
+      - '55679:55679' # zpages extension
+    depends_on:
+      - jaeger
+      - zipkin
+
+  prometheus:
+    image: prom/prometheus:latest
+    volumes:
+      - ./etc/prometheus.yaml:/etc/prometheus/prometheus.yml
+    ports:
+      - '9090:9090'
+    depends_on:
+      - otel-collector

interceptors-opentelemetry/etc/otel-collector-config.yaml

@@ -0,0 +1,47 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+
+      http: # Not used in this sample
+        endpoint: 0.0.0.0:4318
+
+processors:
+  # In a normal setup, enabling the batch processor is recommended for better performance. To enable the
+  # batch processor, you must uncomment the following line AND add it to the service pipelines below.
+  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md
+  # - batch:
+
+extensions:
+  health_check:
+  # pprof:
+  #   endpoint: :1888
+  # zpages:
+  #   endpoint: :55679
+
+exporters:
+  # A PUSH exporter supporting 'traces'
+  otlp/jaeger:
+    endpoint: jaeger:4317
+    tls:
+      insecure: true
+
+  # A PULL exporter supporting 'metrics'
+  # Prometheus will peiodically scrape that exporter
+  prometheus:
+    endpoint: '0.0.0.0:9090'
+
+  # A PUSH exporter supporting 'traces'
+  zipkin:
+    endpoint: http://zipkin:9411/api/v2/spans
+
+service:
+  pipelines:
+    metrics:
+      receivers: [otlp]
+      exporters: [prometheus]
+
+    traces:
+      receivers: [otlp]
+      exporters: [otlp/jaeger, zipkin]

interceptors-opentelemetry/etc/prometheus.yaml

@@ -0,0 +1,5 @@
+scrape_configs:
+  - job_name: 'otel-collector'
+    scrape_interval: 3s
+    static_configs:
+      - targets: ['otel-collector:9090']

interceptors-opentelemetry/package.json

@@ -8,9 +8,9 @@
     "format": "prettier --write .",
     "format:check": "prettier --check .",
     "lint": "eslint .",
-    "start": "ts-node src/worker.ts",
-    "start.watch": "nodemon src/worker.ts",
-    "workflow": "ts-node src/client.ts"
+    "start": "ts-node -r ./src/instrumentation.ts src/worker.ts",
+    "start.watch": "nodemon -r ./src/instrumentation.ts src/worker.ts",
+    "workflow": "ts-node -r ./src/instrumentation.ts ./src/client.ts"
   },
   "nodemonConfig": {
     "execMap": {
@@ -22,11 +22,18 @@
     ]
   },
   "dependencies": {
-    "@opentelemetry/core": "^1.26.0",
-    "@opentelemetry/resources": "^1.26.0",
-    "@opentelemetry/sdk-node": "^0.53.0",
-    "@opentelemetry/sdk-trace-base": "^1.26.0",
-    "@opentelemetry/semantic-conventions": "^1.26.0",
+    "@opentelemetry/core": "^1.30.0",
+    "@opentelemetry/exporter-trace-otlp-grpc": "^0.57.0",
+    "@opentelemetry/exporter-trace-otlp-http": "^0.57.0",
+    "@opentelemetry/exporter-metrics-otlp-grpc": "^0.57.0",
+    "@opentelemetry/exporter-metrics-otlp-http": "^0.57.0",
+    "@opentelemetry/exporter-prometheus": "^0.57.0",
+    "@opentelemetry/resources": "^1.30.0",
+    "@opentelemetry/sdk-metrics": "^1.30.0",
+    "@opentelemetry/auto-instrumentations-node": "^0.55.0",
+    "@opentelemetry/sdk-node": "^0.57.0",
+    "@opentelemetry/sdk-trace-node": "^1.30.0",
+    "@opentelemetry/semantic-conventions": "^1.28.0",
     "@temporalio/activity": "^1.11.6",
     "@temporalio/client": "^1.11.6",
     "@temporalio/interceptors-opentelemetry": "^1.11.6",

interceptors-opentelemetry/src/client.ts

@@ -1,40 +1,28 @@
-import { Connection, Client } from '@temporalio/client';
-import { Resource } from '@opentelemetry/resources';
-import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
-import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-base';
-import { NodeSDK } from '@opentelemetry/sdk-node';
+import { randomUUID } from 'crypto';
 import { OpenTelemetryWorkflowClientInterceptor } from '@temporalio/interceptors-opentelemetry';
+import { Connection, Client } from '@temporalio/client';
 import { example } from './workflows';
 
 async function run() {
-  const resource = new Resource({
-    [ATTR_SERVICE_NAME]: 'interceptors-sample-client',
-  });
-  // Export spans to console for simplicity
-  const exporter = new ConsoleSpanExporter();
-
-  const otel = new NodeSDK({ traceExporter: exporter, resource });
-  await otel.start();
-  // Connect to localhost with default ConnectionOptions,
-  // pass options to the Connection constructor to configure TLS and other settings.
+  // Connect to localhost with default ConnectionOptions.
   const connection = await Connection.connect();
-  // Attach the OpenTelemetryClientCallsInterceptor to the client.
+
+  // Attach the OpenTelemetryWorkflowClientInterceptor to the client.
   const client = new Client({
     connection,
+
+    // Registers OpenTelemetry Tracing interceptor for Client calls
     interceptors: {
       workflow: [new OpenTelemetryWorkflowClientInterceptor()],
     },
   });
-  try {
-    const result = await client.workflow.execute(example, {
-      taskQueue: 'interceptors-opentelemetry-example',
-      workflowId: 'otel-example-0',
-      args: ['Temporal'],
-    });
-    console.log(result); // Hello, Temporal!
-  } finally {
-    await otel.shutdown();
-  }
+
+  const result = await client.workflow.execute(example, {
+    taskQueue: 'interceptors-opentelemetry-example',
+    workflowId: randomUUID(),
+    args: ['Temporal'],
+  });
+  console.log(result); // Hello, Temporal!
 }
 
 run().catch((err) => {