Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Installing OpenSearch, the prompt "Unable to read the file root-ca.pem." appears. #642

Open
tianxin8206 opened this issue Jan 10, 2025 · 13 comments
Labels
bug Something isn't working

Comments

@tianxin8206
Copy link

tianxin8206 commented Jan 10, 2025

What is the bug?
Installing opensearch with helm did not start properly.

java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions]

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Install a default opensearch setup using a helm chart = 2.30.0

What is the expected behavior?
The program starts normally.

What is your host/environment?

  • OS: AlmaLinux 9.5 / Kubernetes V1.30
  • Helm Charts 2.30.0

Do you have any screenshots?

java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?]
	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
	at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
	at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
	at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:95) ~[netty-handler-4.1.114.Final.jar:4.1.114.Final]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin$4.run(OpenSearchSecuritySSLPlugin.java:218) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:213) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.18.0.jar:2.18.0]
[2025-01-10T07:15:02,827][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
	... 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions
	at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?]
	at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
	... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions]
	at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165)
	at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85)
	at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137)
	at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93)
	at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
	at org.opensearch.node.Node.<init>(Node.java:523)
	at org.opensearch.node.Node.<init>(Node.java:450)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)

Do you have any additional context?
It looks like opensearch-project/security#4923 .Please confirm that the BUG has been fixed.

@tianxin8206 tianxin8206 added bug Something isn't working untriaged Issues that have not yet been triaged labels Jan 10, 2025
@cwperks cwperks transferred this issue from opensearch-project/security Jan 13, 2025
@cwperks
Copy link
Member

cwperks commented Jan 13, 2025

Transferring to helm-charts repo.

@DandyDeveloper
Copy link
Collaborator

@tianxin8206 Are you deploying with changes to the values? Can you provide the values you're using?

Specifically, are you disabling the demo installation / demo stuff via the env?

If you do this, it will not run the demo configuration script, resulting in you needing to provide your own certificates for transport SSL between the nodes.

@tianxin8206
Copy link
Author

@DandyDeveloper I modified the value of OPENSEARCH_INITIAL_ADMIN_PASSWORD as required in the document, and made no other changes.These are my values.

@scobit
Copy link

scobit commented Jan 17, 2025

I have the same problem.

I added only:

OPENSEARCH_INITIAL_ADMIN_PASSWORD

and

storageClass: MyNameHere

Chart version: 2.30.1

Also I have tested charts 2.27.1 and 2.26.1 - similar situation.

Probably it some kind of infra issue, but i have no idea what kind of.

In my case one of three pods is running, but another two have status CrashLoopBackOff.

NAME                          READY   STATUS             RESTARTS     AGE
opensearch-cluster-master-0   0/1     CrashLoopBackOff   2 (6s ago)   55s
opensearch-cluster-master-1   0/1     CrashLoopBackOff   2 (8s ago)   55s
opensearch-cluster-master-2   1/1     Running            0            55s

@DandyDeveloper
Copy link
Collaborator

DandyDeveloper commented Jan 17, 2025

@scobit Can you tell me a little more about your k8s cluster? The fact its other nodes, might mean:

  1. You have Opensearch nodes deploying to the same K8s node, which on the backside means they might be trying to share volumes.
  2. You have a node that does have a relevant local FS it can mount things against, other nodes might be using a tempfs and the permissions are not allowing access to the CA certificate in question.

Just as an example, I've got a rootless implementation of Minikube running locally and I've used your values and it's working fine:

C02:12:21 dandy@Laptop ~ → kgp -w
NAME                          READY   STATUS    RESTARTS   AGE
opensearch-cluster-master-0   1/1     Running   0          33s
opensearch-cluster-master-1   1/1     Running   0          33s
opensearch-cluster-master-2   1/1     Running   0          33s

Something to try - Override the command in the statefulset with something like comment: ["bash", "-c", "sleep 10000000"]

Then, exec into the failing nodes and see what's in dir: /usr/share/opensearch/config/root-ca.pem

@gurlov
Copy link

gurlov commented Jan 17, 2025

I have the same problem.
@DandyDeveloper Then, exec into the failing nodes and see what's in dir: /usr/share/opensearch/config/root-ca.pem :

[opensearch@opensearch-cluster-master-0 ~]$ ls -l /usr/share/opensearch/config/
total 40
-rw-rw----. 1 opensearch opensearch 3041 Jan 15 19:14 jvm.options
drwxr-x---. 2 opensearch opensearch 4096 Oct 28 19:44 jvm.options.d
-rw-rw----. 1 opensearch opensearch  514 Jan 15 19:13 log4j2.properties
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-notifications
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-notifications-core
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-observability
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-performance-analyzer
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-reports-scheduler
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-security
-rw-r--r--. 1 opensearch opensearch 1771 Jan 17 14:41 opensearch.yml

@DandyDeveloper
Copy link
Collaborator

@gurlov Throw me a complete node log please, looks to me like the demo stuff is failing for some reason.

@gurlov
Copy link

gurlov commented Jan 17, 2025

@DandyDeveloper with the section plugins: commented in opensearch.yml (This is the default for custom-values.yaml).
When uncommented - visually the errors were the same. But if necessary, I can form them separately and send them.
opensearch.log

@scobit
Copy link

scobit commented Jan 20, 2025

@DandyDeveloper in my case, using this code for StatefulSet:

      - command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash
           cp -r /tmp/configfolder/*  /tmp/config/
        - sleep 10000

And running these commands:

k exec -it opensearch-cluster-master-1 -- cat /usr/share/opensearch/config/root-ca.pem
Defaulted container "opensearch" out of: opensearch, configfile (init)
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

k exec -it opensearch-cluster-master-1 -- cat /usr/share/opensearch/config/root-ca.pem
k exec -it opensearch-cluster-master-2 -- cat /usr/share/opensearch/config/root-ca.pem

I figured out that every pod at start has this file. But after some time one of the pods changed status to failed.

NAME                          READY   STATUS             RESTARTS        AGE
opensearch-cluster-master-0   1/1     Running            0               14m
opensearch-cluster-master-1   1/1     Running            0               14m
opensearch-cluster-master-2   0/1     CrashLoopBackOff   7 (2m38s ago)   14m

I am using k8s by RKE v2.10. Above test was without persistence storage.

opensearch-cluster-master-2.log

@gaiksaya gaiksaya removed the untriaged Issues that have not yet been triaged label Jan 22, 2025
@DandyDeveloper
Copy link
Collaborator

@gurlov

/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.

This line suggests to me that you're doing something with the opensearch.yml config OR somehow sharing volumes between the kubernetes nodes.

I don't get this on my local env, and specifically this will be the reason no certificates are being provisioned.

Are you absolutely sure you're not editing the opensearch.yml in some way? Maybe you have an older persisted volume that shouldn't be there and inherited some bad values?

@cyxinda
Copy link

cyxinda commented Jan 24, 2025

I encountered the same problem using the helm chart of the version 2.30.1.
I have change the config :

# such as opensearch.yml and log4j2.properties
config:
  opensearch.yml: |
    cluster.name: opensearch-cluster

    # Bind to all interfaces because we don't know what IP address Docker will assign to us.
    network.host: 0.0.0.0

    # Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
    # Implicitly done if ".singleNode" is set to "true".
    # discovery.type: single-node

    # Start OpenSearch Security Demo Configuration
    # WARNING: revise all the lines below before you go into production
    plugins:
      security:
        ssl:
    #       transport:
    #         pemcert_filepath: esnode.pem
    #         pemkey_filepath: esnode-key.pem
    #         pemtrustedcas_filepath: root-ca.pem
    #         enforce_hostname_verification: false
          http:
            enabled: false
    #         pemcert_filepath: esnode.pem
    #         pemkey_filepath: esnode-key.pem
    #         pemtrustedcas_filepath: root-ca.pem
    #     allow_unsafe_democertificates: true

And :

extraEnvs:
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
  value: Knowdee666

Excluding the config above ,I didn't change any other configuration.

@DandyDeveloper
Copy link
Collaborator

@cyxinda In your example, the problem makes sense: You've configured things that would exclude the demo provisioning and tried to disable the TLS transport options. You cannot disable these. You must have SSL transport enabled.

With this, you'll need to provision your own self-signed certificates and mount them into the nodes for transport SSL.

@gurlov
Copy link

gurlov commented Jan 24, 2025

@DandyDeveloper Then, exec into the failing nodes and see what's in dir: /usr/share/opensearch/config/root-ca.pem

Where does opensearch look for root-ca.pem? at /usr/share/opensearch/config/ or /usr/share/opensearch/config/certs/ ?
Helm creates my StatefulSet with /usr/share/opensearch/config/certs/

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: opensearch-cluster-master
spec:
  template:
    spec:
      containers:
      - name: opensearch
        volumeMounts:
        - mountPath: /usr/share/opensearch/config/certs
          name: opensearch-ssl
          readOnly: true
      volumes:
      - name: opensearch-ssl
        secret:
          defaultMode: 420
          secretName: opensearch-ssl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 🆕 New
Development

No branches or pull requests

7 participants