Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs API to be user-facing #4225

Closed
wants to merge 24 commits into from
Closed

Logs API to be user-facing #4225

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
45f6847
Use Logs API to emit Events
pellared Sep 25, 2024
ffe4a1b
Fix lint issues
pellared Sep 25, 2024
4dadc75
Update changelog
pellared Sep 25, 2024
90610d2
Fix changelog
pellared Sep 25, 2024
452f61b
Apply suggestions from code review
pellared Sep 25, 2024
4ab9a0a
Apply suggestions from code review
pellared Sep 26, 2024
76c4644
Merge branch 'main' into log-api
pellared Sep 26, 2024
daaaa1f
Apply suggestions from code review
pellared Sep 27, 2024
7c609c0
Merge branch 'main' into log-api
pellared Sep 27, 2024
847b691
Update specification/logs/api.md
pellared Sep 27, 2024
3ea0a68
Merge branch 'main' into log-api
pellared Sep 27, 2024
1409d01
Bring back Events API and SDK
pellared Sep 30, 2024
bbc8ab1
Revert spec complaince matrix
pellared Sep 30, 2024
3b32a2d
Events API and Logs API side by side
pellared Sep 30, 2024
71c0986
Update changelog
pellared Sep 30, 2024
22d628d
Fix hyperlink
pellared Sep 30, 2024
de72b99
Fix links
pellared Sep 30, 2024
beff5c5
Update README.md
pellared Sep 30, 2024
6e2511d
Remove a badly formulated statement
pellared Sep 30, 2024
ba78baa
Refine versioning and stability
pellared Sep 30, 2024
d3aefef
Update api.md
pellared Sep 30, 2024
cc0f262
Update api.md
pellared Sep 30, 2024
47468d4
Merge branch 'main' into log-api
pellared Sep 30, 2024
d286fc2
Update specification/logs/event-api.md
cijothomas Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ release.
([#4203](https://github.com/open-telemetry/opentelemetry-specification/pull/4203))
- Make all fields as identifying for Logger. Previously attributes were omitted from being identifying.
([#4161](https://github.com/open-telemetry/opentelemetry-specification/pull/4161))
- Remove Events API and SDK. Rename Logs Bridge API to Logs API and do not discourage from direct usage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge Events API and SDK

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to simply bring back the Events API and SDK. We can address it later as a separate PR.

([#4225](https://github.com/open-telemetry/opentelemetry-specification/pull/4225))

### Events

Expand Down
20 changes: 0 additions & 20 deletions spec-compliance-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,26 +203,6 @@ Disclaimer: this list of features is still a work in progress, please refer to t
| Can plug custom LogRecordExporter | | | + | | + | | | + | | + | | |
| Trace Context Injection | | | + | | + | | | + | | + | + | |

## Events

Features for the [Events API](specification/logs/event-api.md) and the [Events SDK](specification/logs/event-sdk.md).
Disclaimer: Events are currently in Development status - work in progress.

| Feature | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table is (was) incomplete, several languages already have this capability and users of this API.

JS, Java (Android) and recently Python

|----------------------------------------------------------------------------|----------|----|------|----|--------|------|--------|-----|------|-----|------|-------|
| [EventLoggerProvider](specification/logs/event-api.md#eventloggerprovider) | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
| Get EventLogger | | | | | | | | | | | | |
| Get EventLogger accepts version | X | | | | | | | | | | | |
| Get EventLogger accepts schema_url | X | | | | | | | | | | | |
| Get EventLogger accepts attributes | X | | | | | | | | | | | |
| [EventLogger](specification/logs/event-api.md#eventlogger) | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
| Emit event accepts name | | | | | | | | | | | | |
| Emit event accepts AnyValue body | X | | | | | | | | | | | |
| Emit event accepts severity | X | | | | | | | | | | | |
| Emit event accepts timestamp | X | | | | | | | | | | | |
| Emit event accepts attributes | X | | | | | | | | | | | |
| Emit event accepts context | X | | | | | | | | | | | |

## Resource

| Feature | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
Expand Down
4 changes: 1 addition & 3 deletions specification/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ path_base_for_github_subdir:
- [Baggage](baggage/api.md)
- [Tracing](trace/api.md)
- [Metrics](metrics/api.md)
- [Logs](logs/README.md)
- [Bridge API](logs/bridge-api.md)
- [Event API](logs/event-api.md)
- [Logs](logs/api.md)
- SDK Specification
- [Tracing](trace/sdk.md)
- [Metrics](metrics/sdk.md)
Expand Down
4 changes: 2 additions & 2 deletions specification/glossary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ concept the attributes of the scope cannot change at runtime.

This refers to the `name` and (optional) `version` arguments specified when
creating a new `Tracer` or `Meter` (see
[Obtaining a Tracer](trace/api.md#tracerprovider)/[Obtaining a Meter](metrics/api.md#meterprovider))/[Obtaining a Logger](logs/bridge-api.md#loggerprovider).
[Obtaining a Tracer](trace/api.md#tracerprovider)/[Obtaining a Meter](metrics/api.md#meterprovider))/[Obtaining a Logger](logs/api.md#loggerprovider).
The name/version pair identifies the
[Instrumentation Scope](#instrumentation-scope), for example the
[Instrumentation Library](#instrumentation-library) or another unit of
Expand Down Expand Up @@ -250,7 +250,7 @@ recommended to call it out specifically.
### Log Appender / Bridge

A log appender or bridge is a component which bridges logs from an existing log
API into OpenTelemetry using the [Log Bridge API](./logs/bridge-api.md). The
API into OpenTelemetry using the [Logs (Bridge) API](./logs/api.md). The
terms "log bridge" and "log appender" are used interchangeably, reflecting that
these components bridge data into OpenTelemetry, but are often called appenders
in the logging domain.
34 changes: 17 additions & 17 deletions specification/logs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,16 +140,18 @@ Given the above state of the logging space we took the following approach:
OpenTelemetry log data model. OpenTelemetry Collector can read such logs and
translate them to OpenTelemetry log data model.

- OpenTelemetry defines a Logs Bridge API
for [emitting LogRecords](./bridge-api.md#emit-a-logrecord). Application developers are
NOT encouraged to call this API directly. It is provided for library authors
to build [log appender](../glossary.md#log-appender--bridge), which use
the API to bridge between existing logging libraries and the OpenTelemetry log
data model. Existing logging libraries generally provide a much richer set of
features than what is defined in OpenTelemetry. It is NOT a goal of
OpenTelemetry to ship a feature-rich logging library.

- OpenTelemetry defines an [SDK](./sdk.md) implementation of the [Bridge API](./bridge-api.md),
- OpenTelemetry defines a Logs API for [emitting LogRecords](./api.md#emit-a-logrecord).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This NEEDS to include comments about also emitting events as that is the intention

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is described in API and Data Model. Do we need to repeat it? Can you suggest a change?

It is provided for library authors to build [log appender](../glossary.md#log-appender--bridge),
pellared marked this conversation as resolved.
Show resolved Hide resolved
which use the API to bridge between existing logging libraries and the
OpenTelemetry log data model.
The [API](./api.md) is also provided for instrumentation libraries to emit log
records following semantic conventions (also known as OpenTelemetry Events).
pellared marked this conversation as resolved.
Show resolved Hide resolved
Existing logging libraries generally provide a much richer set of features
than what is defined in OpenTelemetry. It is NOT a goal of OpenTelemetry to
ship a feature-rich logging library. However, nothing prevents application
cijothomas marked this conversation as resolved.
Show resolved Hide resolved
developers to use it directly.

- OpenTelemetry defines an [SDK](./sdk.md) implementation of the [API](./api.md),
which enables configuration of [processing](./sdk.md#logrecordprocessor)
and [exporting](./sdk.md#logrecordexporter) LogRecords.

Expand Down Expand Up @@ -230,10 +232,9 @@ processor.
### Infrastructure Logs

These are logs generated by various infrastructure components, such as
Kubernetes events (if you are wondering why events are discussed in the context
of logs see [Event API Data model](./event-api.md#event-data-model)). Like system logs, the
infrastructure logs lack a trace context and can be enriched by the resource
context - information about the node, pod, container, etc.
Kubernetes events. Like system logs, the infrastructure logs lack a trace
context and can be enriched by the resource context - information about
the node, pod, container, etc.

OpenTelemetry Collector or other agents can be used to query logs from most
common infrastructure controllers.
Expand Down Expand Up @@ -359,7 +360,7 @@ as parsers, log tailing and rotation. It also enables the possibility to send
logs directly to the logging backend without using a log collection agent.

To facilitate both approaches described above OpenTelemetry provides
a [Bridge API](./bridge-api.md) and [SDK](./sdk.md), which can be used together with existing
a [API](./api.md) and [SDK](./sdk.md), which can be used together with existing
logging libraries to automatically inject the trace context in the emitted logs,
and provide an easy way to send the logs via OTLP. Instead of
modifying each logging statement, [log appenders](../glossary.md#log-appender--bridge)
Expand Down Expand Up @@ -444,10 +445,9 @@ standard output.

## Specifications

* [Logs Bridge API](./bridge-api.md)
* [Logs API](./api.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about having links to the old API's stating that the this new API will encompass the existing functionality.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not want to create yet another API.

* [Logs SDK](./sdk.md)
* [Logs Data Model](./data-model.md)
* [Event API](./event-api.md)
* [Trace Context in non-OTLP Log Formats](../compatibility/logging_trace_context.md)

## References
Expand Down
40 changes: 22 additions & 18 deletions specification/logs/bridge-api.md → specification/logs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Logs Bridge API
# Logs API

**Status**: [Stable](../document-status.md), except where otherwise specified

Expand All @@ -9,6 +9,7 @@

<!-- toc -->

- [Overview](#overview)
- [LoggerProvider](#loggerprovider)
* [LoggerProvider operations](#loggerprovider-operations)
+ [Get a Logger](#get-a-logger)
Expand All @@ -18,21 +19,33 @@
+ [Enabled](#enabled)
- [Optional and required parameters](#optional-and-required-parameters)
- [Concurrency requirements](#concurrency-requirements)
- [Artifact Naming](#artifact-naming)
- [References](#references)

<!-- tocstop -->

</details>

<b>Note: this document defines a log *backend* API. The API is not intended
to be called by application developers directly. It is provided for logging
library authors to build
[log appenders](./supplementary-guidelines.md#how-to-create-a-log4j-log-appender),
## Overview

This document defines a log *backend* API.

pellared marked this conversation as resolved.
Show resolved Hide resolved
The API serves following use cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many additional features / requirements that exist and are yet to be defined as we move this API forward (some of which was starting to occur in the Event API) as it was moving (or trying to move) towards stability.
Some of these will include different processing requirements for the different types and as it moves towards a stabilized state it may cause implementation changes, so for now (as a small step) I'll try to define a starting point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should rather require refining the SDK and not the API.

It is provided for instrumentation libraries to emit events (log records following
OpenTelemetry Semantic Conventions).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should include an emitEvent function which MUST include the name as a first-class concept as the specific value of the name defines the Semantic-Definition of any provided body for the specific event.

As the name is a required argument / property for this function it MUST also enforce that the name conforms to the OpenTelemetry general semantic conventions (ie. it is fully qualified -- just like an Attribute name (this doesn't mean a complete list of valid "registered" events), e.g. it just needs to have a leading domain and then name domain.eventName where the domain may have multiple separators.

And it should also

  • Be used only to emit a well-structured, well defined event as per the definitions from an OpenTelemetry and application perspective.
  • These events will include the fundamental log types like "Application", "User", "Client", "Device", "Security".
  • Which "It signals that "this thing happened at this time" and provides additional specifics about the occurrence"
  • Should be used by Applications (Adopters) and Instrumentations only.
  • Because the exact structure and semantic meaning of the body of the event is driven and defined by the name, this allows for SDK's (or applications) to extend and provide (if required)
    • explicit validation of the content
    • explicit sampling / rules governing that specific "instance"
    • explicit delivery mechanism (Send NOW, Don't Drop, Create Metric, as some possible examples)
    • The exact location of the "where" and "when" of any rules is up to the specific application / implementation (eg. it "MAY" occur on the client, in the SDK, at the aggregation point (Collector) or at rest and queried as required.

And (if) future versions of OpenTelemetry evolve to change the internal representation of the event as a LogRecord all of the "changes" can be limited to the SDK implementation/version of the API, so that all existing users of the API have zero change when it's just an internal "representation" within the LogRecord (not-withstanding that this would have downstream implementations)

It is provided for logging library authors to build
[log appenders/bridges](./supplementary-guidelines.md#how-to-create-a-log4j-log-appender),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • emitBridgeLog (just making the name different from above -- needs a better one), just here to have the distinction explicit.
    • Exists to "bridge" into the OpenTelemetry Logs pipeline only
    • Used by log Appenders ONLY to create OpenTelemetry LogRecords and have them processed
    • NO special processing (beyond existing Log Processors ??)*
      • (*)There may be some special use cases where some Log Processors may be required to be skipped (like a processor that links OTel Logs -> Native Logging)
      • Should we change this and have "explicit" Appender/Bridge Log Processors
    • IF an existing language / logging library supports the creation of "Events" or specific "Structured" Logs then this API MAY be used to "Simulate" (generate) a LogRecord that looks like those created by the above API's.
      • HOWEVER, this WILL NOT re-apply / perform any special processing that may be configured by the SDK instance for the other API methods
      • So it's entirely 100% up to the appender (caller of this API) to ensure that it is complying with any necessary semantic conventions to avoid any downstream complications around deterministically identifying / processing the resulting LogRecord.
      • It also MUST NOT just convert everything into an Event UNLESS
        • it can EXPLICITLY identify that the intent of the LogRecord was to identify (signal) that the specifically named event occurred vs general Detailed Logging
        • or it create a general semantic convention that identifies that this LogRecord that looks like an event was created by or follows the semantic conventions of X. ie. it has a specific name like log.*, applog.* or event.ilogger again so the intent (type) can be deterministically evaluated AND it would avoid any "clashes" with defined semantic conventions.
        • To phrase another way as long as the Appender "Knows" that the user emitting the LogRecord WANTS this specific log to be called XXXXX then this is 100% acceptable as long as the control (setting) of this name is explicitly defined by the user. eg. Application has their own logger for their own events, but they want to explicitly emit an OpenTelemetry defined event with a Specific name via their own logger and not via the OpenTelemetry emitEvent with the understanding (implicit or otherwise) that any configured OpenTelemetry "Event" processing will NOT be performed via this path.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MSNev can you highlight any specific blocking concerns you have about this PR beyond #4225 (comment), and open tracking issues for the rest? Once we get this initial restructuring PR landed then we'll be able to start sending PRs to try to address those. Thanks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is up to the bridge implementation how the logging library would be mapped to OTel Logs Data Model. I think we should rather create some "guidance" / "recommendations" for log bridge authors. We cannot control how the bridges are going to be implemented.

For sure this should be out of scope of this PR.

which use this API to bridge between existing logging libraries and the
OpenTelemetry log data model.</b>
OpenTelemetry log data model.

The Logs Bridge API consist of these main components:
It is provided for application developers to emit structured log recors
(e.g. custom events).
pellared marked this conversation as resolved.
Show resolved Hide resolved

Languges MAY provide a separate Logs Bridge API if they need different
ergonimics for consumers that are building log appenders/bridges.
pellared marked this conversation as resolved.
Show resolved Hide resolved

pellared marked this conversation as resolved.
Show resolved Hide resolved
The Logs API consist of these main components:

pellared marked this conversation as resolved.
Show resolved Hide resolved
* [LoggerProvider](#loggerprovider) is the entry point of the API. It provides access to `Logger`s.
* [Logger](#logger) is responsible for emitting logs as
Expand Down Expand Up @@ -167,22 +180,13 @@ provide it.

## Concurrency requirements

For languages which support concurrent execution the Logs Bridge APIs provide
For languages which support concurrent execution the Logs APIs provide
specific guarantees and safeties.

**LoggerProvider** - all methods are safe to be called concurrently.

**Logger** - all methods are safe to be called concurrently.

## Artifact Naming

The Logs Bridge API is not intended to be called by application developers
directly, and SHOULD include documentation that discourages direct use. However,
in the event OpenTelemetry were to add a user facing API, the Logs Bridge API would
be a natural starting point. Therefore, Log Bridge API artifact, package, and class
names MUST NOT include the terms "bridge", "appender", or any other qualifier
that would prevent evolution into a user facing API.

## References

- [OTEP0150 Logging Library SDK Prototype Specification](https://github.com/open-telemetry/oteps/blob/main/text/logs/0150-logging-library-sdk.md)
162 changes: 0 additions & 162 deletions specification/logs/event-api.md
View file
Open in desktop

This file was deleted.

Loading
Loading