Java Logger API does not compliant with the OTel Logs Data Model

[yijiem]

Maybe this is related to #6581.

In the OTel Logs Data Model, it is specified that the Body field must support any type to preserve the semantics of structured logs emitted by the application: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-body. This is specifically useful for the application which logs a JSON-structured data i.e. map<string, any> as the Body. But the OTel Java LogRecordBuilder API does not support such a case, it only supports setting a String in the body: https://github.com/open-telemetry/opentelemetry-java/blob/main/api/all/src/main/java/io/opentelemetry/api/logs/LogRecordBuilder.java#L69-L70.

Ideally, if the LogRecordBuilder API preserves the semantics of structured data, the structure should be preserved across the stack (e.g. from application to OTel collector via OTLP and eventually the structured data gets exported to the backend log store).

Seems like there is also an Event API for logging structured data. But it is still a WIP and is a specialized Log API and would have the same issue in opentelemetry-java since the current API does not support structured data.

Also it seems like OTLP is able to carry structured data in LogRecord.body.

I filed a similar issue with OTel-Cpp at: open-telemetry/opentelemetry-cpp#3000.

[jkwatson]

Yes, please see #6591

[jack-berg]

Just some additional context, any value log bodies are available via opentelemetry-api-incubator today. See example here

opentelemetry-java/api/incubator/src/test/java/io/opentelemetry/api/incubator/logs/ExtendedLogsBridgeApiUsageTest.java

Lines 85 to 135 in 5be177d

	void extendedLogRecordBuilderUsage() {
	// Setup SdkLoggerProvider
	InMemoryLogRecordExporter exporter = InMemoryLogRecordExporter.create();
	SdkLoggerProvider loggerProvider =
	SdkLoggerProvider.builder()
	// Default resource used for demonstration purposes
	.setResource(Resource.getDefault())
	// Simple processor w/ in-memory exporter used for demonstration purposes
	.addLogRecordProcessor(SimpleLogRecordProcessor.create(exporter))
	.build();
	// Get a Logger for a scope
	Logger logger = loggerProvider.get("org.foo.my-scope");
	// Cast to ExtendedLogRecordBuilder, and emit a log
	((ExtendedLogRecordBuilder) logger.logRecordBuilder())
	// ...can set AnyValue log record body, allowing for arbitrarily complex data
	.setBody(
	AnyValue.of(
	ImmutableMap.of(
	"key1",
	AnyValue.of("value1"),
	"key2",
	AnyValue.of(
	ImmutableMap.of(
	"childKey1",
	AnyValue.of("value2"),
	"childKey2",
	AnyValue.of("value3"))))))
	.emit();
	// SDK can access AnyValue body by casting to AnyValueBody
	loggerProvider.forceFlush().join(10, TimeUnit.SECONDS);
	assertThat(exporter.getFinishedLogRecordItems())
	.satisfiesExactly(
	logData ->
	assertThat(((AnyValueBody) logData.getBody()).asAnyValue())
	.isEqualTo(
	AnyValue.of(
	ImmutableMap.of(
	"key1",
	AnyValue.of("value1"),
	"key2",
	AnyValue.of(
	ImmutableMap.of(
	"childKey1",
	AnyValue.of("value2"),
	"childKey2",
	AnyValue.of("value3")))))));
	}
	}

[jack-berg]

Closing as duplicate of #6591.

[yijiem]

Thanks! I'm wondering when exporting AnyValue log record body in OTLP format, will the structure of the log record body be preserved in OTLP's LogRecord::body field? https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto#L167-L170

[jack-berg]

Thanks! I'm wondering when exporting AnyValue log record body in OTLP format, will the structure of the log record body be preserved in OTLP's LogRecord::body field?

Yup. Check out this integration test which verifies this:

opentelemetry-java/integration-tests/otlp/src/main/java/io/opentelemetry/integrationtest/OtlpExporterIntegrationTest.java

Lines 654 to 794 in 5be177d

	.setBody(
	of(
	KeyAnyValue.of("str_key", of("value")),
	KeyAnyValue.of("bool_key", of(true)),
	KeyAnyValue.of("int_key", of(1L)),
	KeyAnyValue.of("double_key", of(1.1)),
	KeyAnyValue.of("bytes_key", of("value".getBytes(StandardCharsets.UTF_8))),
	KeyAnyValue.of("arr_key", of(of("value"), of(1L))),
	KeyAnyValue.of(
	"kv_list",
	of(
	KeyAnyValue.of("child_str_key", of("value")),
	KeyAnyValue.of(
	"child_kv_list",
	of(KeyAnyValue.of("grandchild_str_key", of("value"))))))))
	.setTimestamp(100, TimeUnit.NANOSECONDS)
	.setAllAttributes(Attributes.builder().put("key", "value").build())
	.setSeverity(Severity.DEBUG)
	.setSeverityText("DEBUG")
	.setContext(Context.current())
	.emit();
	eventLogger.builder("namespace.event-name").put("key", "value").emit();
	}
	// Closing triggers flush of processor
	loggerProvider.close();
	await()
	.atMost(Duration.ofSeconds(30))
	.untilAsserted(() -> assertThat(grpcServer.logRequests).hasSize(1));
	ExportLogsServiceRequest request = grpcServer.logRequests.get(0);
	assertThat(request.getResourceLogsCount()).isEqualTo(1);
	ResourceLogs resourceLogs = request.getResourceLogs(0);
	assertThat(resourceLogs.getResource().getAttributesList())
	.contains(
	KeyValue.newBuilder()
	.setKey(SERVICE_NAME.getKey())
	.setValue(AnyValue.newBuilder().setStringValue("integration test").build())
	.build());
	assertThat(resourceLogs.getScopeLogsCount()).isEqualTo(1);
	ScopeLogs ilLogs = resourceLogs.getScopeLogs(0);
	assertThat(ilLogs.getScope().getName()).isEqualTo(OtlpExporterIntegrationTest.class.getName());
	assertThat(ilLogs.getLogRecordsCount()).isEqualTo(2);
	// LogRecord via Logger.logRecordBuilder()...emit()
	io.opentelemetry.proto.logs.v1.LogRecord protoLog1 = ilLogs.getLogRecords(0);
	assertThat(protoLog1.getBody())
	.isEqualTo(
	AnyValue.newBuilder()
	.setKvlistValue(
	KeyValueList.newBuilder()
	.addValues(
	KeyValue.newBuilder()
	.setKey("str_key")
	.setValue(AnyValue.newBuilder().setStringValue("value").build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("bool_key")
	.setValue(AnyValue.newBuilder().setBoolValue(true).build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("int_key")
	.setValue(AnyValue.newBuilder().setIntValue(1).build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("double_key")
	.setValue(AnyValue.newBuilder().setDoubleValue(1.1).build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("bytes_key")
	.setValue(
	AnyValue.newBuilder()
	.setBytesValue(
	ByteString.copyFrom(
	"value".getBytes(StandardCharsets.UTF_8)))
	.build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("arr_key")
	.setValue(
	AnyValue.newBuilder()
	.setArrayValue(
	ArrayValue.newBuilder()
	.addValues(
	AnyValue.newBuilder()
	.setStringValue("value")
	.build())
	.addValues(
	AnyValue.newBuilder().setIntValue(1).build())
	.build())
	.build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("kv_list")
	.setValue(
	AnyValue.newBuilder()
	.setKvlistValue(
	KeyValueList.newBuilder()
	.addValues(
	KeyValue.newBuilder()
	.setKey("child_str_key")
	.setValue(
	AnyValue.newBuilder()
	.setStringValue("value")
	.build())
	.build())
	.addValues(
	KeyValue.newBuilder()
	.setKey("child_kv_list")
	.setValue(
	AnyValue.newBuilder()
	.setKvlistValue(
	KeyValueList.newBuilder()
	.addValues(
	KeyValue.newBuilder()
	.setKey(
	"grandchild_str_key")
	.setValue(
	AnyValue
	.newBuilder()
	.setStringValue(
	"value")
	.build())
	.build())
	.build())
	.build())
	.build())
	.build())
	.build())
	.build())
	.build())
	.build());