Skip to content

Commit

Permalink
Merge pull request #1066 from splunk/repo-sync
Browse files Browse the repository at this point in the history
Pulling refs/heads/main into main
  • Loading branch information
aurbiztondo-splunk authored Nov 21, 2023
2 parents 514bd88 + 771ae2f commit 404a616
Show file tree
Hide file tree
Showing 5 changed files with 66 additions and 28 deletions.
Binary file modified _images/logs/chrome-secure1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified _images/logs/chrome-secure2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
39 changes: 36 additions & 3 deletions infrastructure/intro-to-infrastructure.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,42 @@ Splunk Infrastructure Monitoring hierarchy

The following diagram gives you an overview of the different components and their hierarchical relationship in Splunk Infrastructure Monitoring.

.. image:: /_images/infrastructure/imm-hierarchy.png
:width: 80%
:alt: This image describes the hierarchy in Splunk Infrastructure Monitoring.
.. mermaid::

%%{
init: {
'theme': 'base',
'themeVariables': {
'primaryColor': '#FFFFFF',
'primaryTextColor': '#000000',
'primaryBorderColor': '#000000',
'nodeBorder':'#000000',
'lineColor': '#000000',
}
}
}%%


flowchart TB
accTitle: Splunk Infrastructure Monitoring hierarchy
accDescr: In Splunk Infrastructure Monitoring, realm is the all-encompassing top level. A realm contains multiple organizations. Each organization contains dashboard groups, navigators, detectors, and teams. Teams contain users. Both dashboard groups and navigators contain dashboards. Dashboards contain charts. Charts and detectors use metrics to operate. Detectors can generate alerts and send notifications.

%% LR indicates the direction (left-to-right)

%% You can define classes to style nodes and other elements
classDef default fill:#FFFFFF, stroke:#000

subgraph Splunk Infrastructure Monitoring hierarchy
direction TB
realm[Realm]--contains-->org[Organizations]--contain-->dashboardGroup[Dashboard groups] & navigator[Navigators] & detector[Detectors] & teams[Teams]
dashboardGroup --contain-->dashboard[Dashboards]
navigator--contain-->dashboard--contain-->chart[Charts]
chart--use-->metric[Metrics]
detector--use-->metric
detector--generate-->alert[Alerts]
teams--contain-->users[Users]
alert--send-->notification[Notifications]
end

Metrics are the driving components for Splunk Infrastructure Monitoring. To learn more about metrics, see :ref:`get-started-metrics`.

Expand Down
37 changes: 19 additions & 18 deletions metrics-and-metadata/enablerelatedcontent.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,24 @@
Enable Related Content in Splunk Observability Cloud
*****************************************************************

Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by OpenTelemetry and Splunk Observability Cloud.
Splunk Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by OpenTelemetry and Splunk Observability Cloud.

When you deploy Splunk Distribution of Open Telemetry Collector to send your telemetry data to Observability Cloud, your metadata key names are automatically mapped correctly. When you do not use the Splunk Distribution of OpenTelemetry Collector, your telemetry data might have metadata key names that are not consistent with those used by Observability Cloud and OpenTelemetry. In that case, you must change your metadata key names.
When you deploy the Splunk Distribution of OpenTelemetry Collector to send your telemetry data to Observability Cloud, your metadata key names are automatically mapped correctly. If you do not use the Splunk Distribution of OpenTelemetry Collector, your telemetry data might have metadata key names that are not consistent with those used by Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names.

For example, say Observability Cloud receives the following telemetry data:
For example, say Splunk Observability Cloud receives the following telemetry data:

- Splunk APM receives a trace with the metadata key ``trace_id: 2b78e7c951497655``

- Splunk Log Observer receives a log with the metadata key ``trace.id:2b78e7c951497655``

Although these refer to the same trace ID value, the log and the trace cannot be correlated in Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. In this case, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Observability Cloud. Then when you are viewing the trace in APM, you can click directly into the log with the same trace ID value and view the correlated log in Log Observer.
Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. In this case, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Observability Cloud. Then, when you are viewing the trace in APM, you can select directly into the log with the same trace ID value and view the correlated log in Log Observer.

How to change your metadata key names
=================================================================
Metrics and traces

Change metric and traces names
-----------------------------------------------------------------

Use the Splunk Distribution of OpenTelemetry Collector to ensure that your metrics and traces have the metadata key names required to use Observability Cloud's Related Content feature. If you did not use the Collector and your metrics or traces do not include the required metadata key names, you can instrument your applications and serverless functions to include them. See the following pages to learn how:

- :ref:`get-started-application`
Expand All @@ -28,16 +30,16 @@ Use the Splunk Distribution of OpenTelemetry Collector to ensure that your metri

- :ref:`rum-gdi`


Logs
Change logs names
-----------------------------------------------------------------

If the required key names use different names in your log fields, remap them using one of the methods listed in :ref:`remap-log-fields`.

The remainder of this page provides details on the required metadata fields for each view in Observability Cloud.


Splunk APM
=================================================================

To ensure full functionality of Related Content, do not change any of the metadata key names or span tags provided by the Splunk Distribution of OpenTelemetry Collector. To learn more about span tags in Splunk APM, see :ref:`apm-traces-spans`.

The Splunk Distribution of OpenTelemetry Collector provides the following APM span tags that enable Related Content:
Expand All @@ -49,6 +51,7 @@ To learn more about deployment environments in Splunk APM, see :ref:`apm-environ

Leverage Related Content for pod-specific Kubernetes data
-----------------------------------------------------------------

For a Related Content tile in APM to link to data for a specific Kubernetes pod (k8s.pod.name), you must first filter on a specific Kubernetes cluster (k8s.cluster.name). APM cannot guarantee an accurate Related Content Kubernetes pod destination in Infrastructure Monitoring without both values because Kubernetes pod names are not required to be unique across clusters.

For example, consider a scenario in which Related Content needs to return data for a Kubernetes pod named :strong:`Pod-B`. As shown the following diagram, a Kubernetes implementation can have multiple pods with the same name. For Related Content to return the data for the correct :strong:`Pod-B`, you must also provide the name of the Kubernetes cluster the pod resides in. In this case, that name would be either :strong:`Cluster-West` or :strong:`Cluster-East`. This combination of filtering on cluster and pod names creates the unique combination that Related Content needs to link to the correct pod data in Infrastructure Monitoring.
Expand All @@ -59,11 +62,11 @@ For example, consider a scenario in which Related Content needs to return data f
:width: 80%
:alt: This diagram shows two uniquely named Kubernetes clusters, each containing pods that share names across clusters.


.. _enablerelatedcontent-imm:

Splunk Infrastructure Monitoring
=================================================================

To ensure full functionality of Related Content, do not change any of the metadata key names provided by the Splunk Distribution of OpenTelemetry Collector.

The Splunk Distribution of OpenTelemetry Collector provides the following Infrastructure Monitoring metadata keys that enable Related Content:
Expand All @@ -80,6 +83,7 @@ The Splunk Distribution of OpenTelemetry Collector provides the following Infras

Splunk Log Observer
=================================================================

To ensure full functionality of both Log Observer and Related Content, confirm that your log events fields are correctly mapped. Correct log field mappings enable built-in log filtering, embed logs in APM and
Infrastructure Monitoring functionality, and enable fast searches as well as the Related Content bar.

Expand All @@ -93,11 +97,13 @@ The following key names are required to enable Related Content for Log Observer:

If the key names in the preceding list use different names in your log fields, remap them to the key names listed here. For example, if you do not see values for :strong:`host.name` in the Log Observer UI, check to see whether your logs use a different field name, such as :strong:`host_name`. If your logs do not contain the default field names exactly as they appear in the preceding list, remap your logs using one of the methods in the following section.

.. include:: /_includes/log-observer-transition.rst

.. _remap-log-fields:

Methods of remapping log fields
--------------------------------------------------------------------------

The following table describes the four methods for remapping log fields:

.. list-table::
Expand All @@ -107,7 +113,7 @@ The following table describes the four methods for remapping log fields:
* - :strong:`Remapping Method`
- :strong:`Instructions`

* - Observability Cloud Logs Pipeline Management
* - Splunk Observability Cloud Logs Pipeline Management
- Create and apply a field copy processor. See the
:strong:`Field copy processors` section in
:ref:`logs-processors` to learn how.
Expand Down Expand Up @@ -139,7 +145,8 @@ Use Log Field Aliasing to remap fields in Observability Cloud when you cannot or

Kubernetes log fields
--------------------------------------------------------------------------
Do not change the following fields, which Splunk Distribution of OpenTelemetry Collector injects into your Kubernetes logs:

Do not change the following fields, which the Splunk Distribution of OpenTelemetry Collector injects into your Kubernetes logs:

- ``k8s.cluster.name``
- ``k8s.node.name``
Expand All @@ -148,12 +155,6 @@ Do not change the following fields, which Splunk Distribution of OpenTelemetry C
- ``k8s.namespace.name``
- ``kubernetes.workload.name``


Using Observability Collector for Kubernetes
----------------------------------------------------------------------------

For Kubernetes environments, instead of changing existing Fluentd configuration, you can install a configured agent provided as a helm chart. It goes with a configured Fluentd agent and OpenTelemetry collector for collecting logs, metrics, and traces with all metadata relevant to Kubernetes.

To learn more about Observability Collector for Kubernetes, see :new-page:`Observability Collector for Kubernetes helm chart <https://github.com/signalfx/o11y-collector-for-kubernetes>` on GitHub.
Learn more about the Collector for Kubernetes at :ref:`otel-install-k8s`.


18 changes: 11 additions & 7 deletions metrics-and-metadata/relatedcontent.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,22 @@ Related Content in Splunk Observability Cloud

The Related Content feature automatically correlates data between different views within Splunk Observability Cloud by presenting related data at the bottom of the screen.

Select tiles in the Related Content bar to seamlessly navigate from one view to another in Observability Cloud. The following animation shows a user navigating from APM to Infrastructure Monitoring to Log Observer.
Select tiles in the Related Content bar to seamlessly navigate from one view to another in Splunk Observability Cloud. The following animation shows a user navigating from APM to Infrastructure Monitoring to Log Observer.

.. image:: /_images/get-started/Related1.gif
:alt: Using Related Content in Observability Cloud.
:alt: Using Related Content in Splunk Observability Cloud.

In the preceding example, the user navigates through the following sequence:

1. The user starts in APM by exploring the service dependency map. They select the :strong:`Frontend` service because it shows a high error rate.

In the Related Content bar at the bottom of the screen, the user sees an Infrastructure tile showing related EC2 instances and selects it. Results are grouped by component. For example, Infrastructure, Logs, APM. Hovering over the tile indicates whether there are any Related Content results to view.

2. Observability Cloud takes the user to Infrastructure where they click the first EC2 instance because it shows the highest CPU utilization.
2. Splunk Observability Cloud takes the user to Infrastructure where they select the first EC2 instance because it shows the highest CPU utilization.

In the Related Content bar, the user sees a tile showing logs related to the EC2 instance, so they click it.

3. Observability Cloud takes them to Log Observer where they can drill down into the related logs to find the root cause of the problem.
3. Splunk Observability Cloud takes them to Log Observer where they can drill down into the related logs to find the root cause of the problem.

.. note:: Related Content is different from data links, a separate capability, which lets you dynamically transfer contextual information about the property you're viewing to the resource, helping you get to relevant information faster. To learn more about data links, see :ref:`apm-create-data-links`.

Expand All @@ -33,7 +33,7 @@ Prerequisites

Related Content relies on specific metadata that allow APM, Infrastructure Monitoring, and Log Observer to pass filters around Observability Cloud.

The following sections list the metadata key names required to enable Related Content for each view in Observability Cloud. If your data does not have the field names listed here, Observability Cloud cannot correlate your related data.
The following sections list the metadata key names required to enable Related Content for each view in Splunk Observability Cloud. If your data does not have the field names listed here, Splunk Observability Cloud cannot correlate your related data.

APM
-----------------------------------------------------------------
Expand All @@ -60,7 +60,9 @@ The following Infrastructure Monitoring metadata keys are required to enable Rel
- ``k8s.namespace.name``
- ``kubernetes.workload.name``

The Splunk Distribution of OpenTelemetry Collector already provides the Infrastructure Monitoring metadata.
If you're using the Splunk Distribution of the OpenTelemetry Collector for Kubernetes, the required Infrastructure Monitoring metadata is provided. See more at :ref:`otel-install-k8s`.

If you're using other configurations to collect infrastructure data, Related Content won't work out of the box.

Log Observer
-----------------------------------------------------------------
Expand All @@ -73,6 +75,8 @@ The following metadata keys are required to enable Related Content for Log Obser
- ``trace_id``
- ``span_id``

.. include:: /_includes/log-observer-transition.rst

Enable Related Content
=================================================================

Expand All @@ -81,7 +85,7 @@ See :ref:`get-started-enablerelatedcontent` to learn how you can make any necess
Where can I see Related Content?
=================================================================

The following table describes when and where in Observability Cloud you can see Related Content:
The following table describes when and where in Splunk Observability Cloud you can see Related Content:

.. list-table::
:header-rows: 1
Expand Down

0 comments on commit 404a616

Please sign in to comment.