Skip to content

Commit

Permalink
Merge pull request #1535 from splunk/mbechtld-dg23652-zeroconfig-impr…
Browse files Browse the repository at this point in the history 
…ovements-2

[DOCGUILD-23652] Updates for zero config Java + Node.js
  • Loading branch information
@mbechtold-splunk
mbechtold-splunk authored Nov 28, 2023
2 parents f409a99 + 62c5cba commit 1095a00
Show file tree
Hide file tree
Showing 2 changed files with 241 additions and 107 deletions.
204 changes: 135 additions & 69 deletions gdi/opentelemetry/auto-instrumentation/auto-instrumentation-java-k8s.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ Zero Config Auto Instrumentation for Java requires the following components:
* A Kubernetes instrumentation object ``opentelemetry.io/v1alpha1``, which configures auto-instrumentation settings for applications.

Deploy the Helm Chart with the Operator enabled
------------------------------------------------------------
=========================================================

Add certifications and deploy the Helm Chart
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
--------------------------------------------------------

The Operator requires certain TLS certificates to work. Use the following command to check whether a certification manager is available:

Expand All @@ -36,21 +36,96 @@ If a certification manager (or any other TLS certificate source) is not availabl
.. code-block:: yaml
# If cert-manager is not deployed.
helm install splunk-otel-collector -f ./my_values.yaml --set certmanager.enabled=true,operator.enabled=true,environment=dev -n monitoring helm-charts/splunk-otel-collector
helm install splunk-otel-collector -f ./my_values.yaml --set certmanager.enabled=true,operator.enabled=true,environment=prd -n monitoring splunk-otel-collector-chart/splunk-otel-collector
# If cert-manager is already deployed.
helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,environment=dev -n monitoring helm-charts/splunk-otel-collector
helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,environment=prd -n monitoring splunk-otel-collector-chart/splunk-otel-collector
.. _zeroconfig-java-traces:

Ingest traces
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
------------------------------------------------

To properly ingest trace telemetry data, the attribute ``deployment.environment`` must be onboard the exported traces. The following table demonstrates the different methods for setting this attribute:

.. list-table::
:header-rows: 1
:width: 100%
:widths: 33 33 33

* - Method
- Scope
- Implementation
* - Through the ``values.yaml`` file ``environment`` configuration
- Applies the attribute to all telemetry data (metrics, logs, traces) exported through the collector.
- The chart will set an attribute processor to add ``deployment.environment=prd`` to all telemetry data processed by the collector.
* - Through the ``values.yaml`` file and ``operator.instrumentation.spec.env`` or ``operator.instrumentation.spec.{instrumentation_library}.env`` configuration
- Allows you to set ``deployment.environment`` either for all auto-instrumented applications collectively or per auto-instrumentation language.
- Add the ``OTEL_RESOURCE_ATTRIBUTES`` environment variable, setting its value to ``deployment.environment=prd``.
* - Through your Kubernetes application deployment, daemonset, or pod specification
- Allows you to set ``deployment.environment`` at the level of individual deployments, daemonsets, or pods.
- Employ the ``OTEL_RESOURCE_ATTRIBUTES`` environment variable, assigning the value ``deployment.environment=prd``.

The following examples demonstrate how to set the attribute using each method:

.. tabs::

.. tab:: ``values.yaml``

Add the environment variable using the ``extraEnvs`` option in the ``values.yaml`` file. This adds the ``deployment.environment`` attribute to all telemetry data the Collector receives, including data from automatically-instrumented pods.

.. code-block:: yaml
extraEnvs: [OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prd]
In order to be properly ingest trace telemetry data, the attribute ``environment`` must be on board the exported traces. There are two ways to set this attribute:
.. tab:: ``values.yaml`` (Instrumentation spec)

* Use the ``values.yaml`` optional environment configuration.
* Use the Instrumentation spec with the environment variable ``OTEL_RESOURCE_ATTRIBUTES``.
Add the environment variable to the ``instrumentation`` spec as shown in the following example code. This method adds the ``deployment.environment`` attribute to all telemetry data from automatically-instrumented pods.

.. code-block:: yaml
operator:
enabled: true
instrumentation:
spec:
env:
- name: OTEL_RESOURCE_ATTRIBUTES
value: "deployment.environment=prd"
java:
env:
- name: OTEL_RESOURCE_ATTRIBUTES
value: "deployment.environment=prd-canary-java"
.. tab:: Deployment ``.yaml`` file

Update the application deployment YAML file. This method adds the ``deployment.environment`` attribute to all telemetry data from pods that contain the specified environment variable.

.. code-block:: yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-java-app
spec:
template:
spec:
containers:
- name: my-java-app
image: my-java-app:latest
env:
- name: OTEL_RESOURCE_ATTRIBUTES
value: "deployment.environment=prd"
.. tab:: ``kubectl``

Update the environment variable ``OTEL_RESOURCE_ATTRIBUTES`` using ``kubectl set env``. For example:

.. code-block:: bash
kubectl set env deployment/<my-deployment> OTEL_RESOURCE_ATTRIBUTES=environment=prd
Verify all the OpenTelemetry resources are deployed successfully
---------------------------------------------------------------------------
==========================================================================

Resources include the Collector, the Operator, webhook, an instrumentation.

Expand Down Expand Up @@ -78,73 +153,56 @@ Run the following to verify the resources are deployed correctly:
# splunk-instrumentation 3m http://$(SPLUNK_OTEL_AGENT):4317
Set annotations to instrument Java applications
-------------------------------------------------------------------

You can activate auto instrumentation for Java applications before or during runtime.
===================================================================

Activate and deactivate auto instrumentation before runtime
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can activate auto instrumentation for Java applications before runtime.

If the deployment is not deployed, add the ``otel.splunk.com/inject-java`` annotation to the application deployment YAML file.
If the related Kubernetes object (deployment, daemonset, or pod) is not deployed, add the ``otel.splunk.com/inject-java`` annotation to the application object YAML.

For example, given the following deployment YAML:

.. code-block:: yaml
.. code-block:: yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-java-app
spec:
template:
spec:
containers:
- name: my-java-app
image: my-java-app:latest
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-java-app
namespace: monitoring
spec:
template:
spec:
containers:
- name: my-java-app
image: my-java-app:latest
Activate auto instrumentation by adding ``otel.splunk.com/inject-java: "true"`` to the ``spec``:

.. code-block:: yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-java-app
spec:
template:
metadata:
annotations:
otel.splunk.com/inject-java: "true"
spec:
containers:
- name: my-java-app
image: my-java-app:latest
The Collector operator activates automatic instrumentation for any Java applications in the deployment.

To deactivate automatic instrumentation, remove the annotation or set its value to ``false``.

Activate and deactivate auto instrumentation for Java on a running workload
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To activate auto instrumentation for your Java deployment, run the following command. Replace ``<my-deployment>`` with the deployment name and ``<my-namespace>`` with the name of the target application namespace.

.. code-block:: bash
kubectl patch deployment <my-deployment> -n <my-namespace> -p '{"spec": {"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"<splunk_otel_collector_namespace>/splunk-otel-collector"}}}} }'
.. note::
* The deployment pod will restart after running this command.
* If the chart is not installed in the "default" namespace, modify the annotation value to be "{chart_namespace}/splunk-otel-collector".

To deactivate auto instrumentation for your Java deployment, run the same command but change the annotation value to ``false``:

.. code-block:: bash
kubectl patch deployment <my-deployment> -n <my-namespace> --type=json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-java"}]'
.. code-block:: yaml
:emphasize-lines: 10
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-java-app
namespace: monitoring
spec:
template:
metadata:
annotations:
otel.splunk.com/inject-java: "true"
spec:
containers:
- name: my-java-app
image: my-java-app:latest
To deactivate automatic instrumentation, remove the annotation. The following command removes the annotation for automatic instrumentation, deactivating it:

.. code-block:: bash
kubectl patch deployment <my-deployment> -n <my-namespace> --type=json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-java"}]'
Verify instrumentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-----------------------------------------------

To verify that the instrumentation was successful, run the following command on an individual pod. Your instrumented pod should contain an initContainer named ``opentelemetry-auto-instrumentation`` and the target application container should have several ``OTEL_*`` environment variables similar to those in the demo output below.

Expand All @@ -153,7 +211,7 @@ To verify that the instrumentation was successful, run the following command on
kubectl describe pod -n otel-demo -l app.kubernetes.io/name=opentelemetry-demo-frontend
# Name: opentelemetry-demo-frontend-57488c7b9c-4qbfb
# Namespace: otel-demo
# Annotations: instrumentation.opentelemetry.io/inject-java: default/splunk-otel-collector
# Annotations: instrumentation.opentelemetry.io/inject-java: true
# Status: Running
# Init Containers:
# opentelemetry-auto-instrumentation:
Expand Down Expand Up @@ -196,16 +254,24 @@ To verify that the instrumentation was successful, run the following command on
# Type: EmptyDir (a temporary directory that shares a pod's lifetime)
View results at Splunk Observability APM
------------------------------------------------------------
===========================================================

Allow the Operator to do the work. The Operator intercepts and alters the Kubernetes API requests to create and update annotated pods, the internal pod application containers are instrumented, and trace and metrics data populates the :ref:`APM dashboard <apm-dashboards>`.

(Optional) Configure the instrumentation
------------------------------------------------------------
===========================================================

You can configure the Splunk Distribution of OpenTelemetry Java to suit your instrumentation needs. In most cases, modifying the basic configuration is enough to get started.

To learn more, see :ref:`java-manual-instrumentation`.
You can add advanced configuration like activating custom sampling and including custom data in the reported spans with environment variables and Java system properties. To do so, use the ``values.yaml`` file and ``operator.instrumentation.sampler`` configuration. For more information, see the :new-page:`documentation in GitHub <https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#instrumentationspecsampler>` and :new-page:`example in GitHub <https://github.com/signalfx/splunk-otel-collector-chart/blob/main/examples/enable-operator-and-auto-instrumentation/instrumentation/instrumentation-add-trace-sampler.yaml>`.

You can also use the methods shown in :ref:`zeroconfig-java-traces` to configure your instrumentation with the ``OTEL_RESOURCE_ATTRIBUTES`` environment variable and other environment variables. For example, if you want every span to include the key-value pair ``build.id=feb2023_v2``, set the ``OTEL_RESOURCE_ATTRIBUTES`` environment variable:

.. code-block:: bash
kubectl set env deployment/<my-deployment> OTEL_RESOURCE_ATTRIBUTES=build.id=feb2023_v2
See :ref:`advanced-java-otel-configuration` for the full list of supported environment variables.

Learn more
===========================================================================
Expand Down
Loading

0 comments on commit 1095a00

Please sign in to comment.