feat!: Enable pprof server on demand (#16594) #17068

JorTurFer · 2024-02-01T14:09:47Z

UPDATE: pprof server was enabled as default ALWAYS, this is a security risk so this PR has removed the default always on integration in favor of an on demand integration.

In order to profile current ArgoCD components, users have to modify the code and deploy their own image into their clusters. Although this works perfectly, it's not really user-friendly.
controller-runtime added sometime ago a native support for pprof via registering a http server on demand to enable and expose pprof. There are some blockers to integrate that solution (as I shared in the issue), so I've created a pkg that can be used in all ArgoCD components to support this.

Fixes #16594

Checklist:

codecov · 2024-02-01T14:17:47Z

Codecov Report

Attention: Patch coverage is 9.63855% with 75 lines in your changes missing coverage. Please review.

Project coverage is 52.68%. Comparing base (d0313c5) to head (e109d52).
Report is 927 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/pprof/server.go	14.63%	35 Missing ⚠️
...ntroller/commands/argocd_application_controller.go	0.00%	7 Missing ⚠️
...md/argocd-cmp-server/commands/argocd_cmp_server.go	0.00%	7 Missing ⚠️
cmd/argocd-notification/commands/controller.go	0.00%	7 Missing ⚠️
cmd/argocd-server/commands/argocd_server.go	0.00%	7 Missing ⚠️
...t-controller/commands/applicationset_controller.go	0.00%	5 Missing and 1 partial ⚠️
.../argocd-repo-server/commands/argocd_repo_server.go	25.00%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17068      +/-   ##
==========================================
+ Coverage   50.70%   52.68%   +1.98%     
==========================================
  Files         316      316              
  Lines       43479    43656     +177     
==========================================
+ Hits        22046    23001     +955     
+ Misses      18920    18102     -818     
- Partials     2513     2553      +40

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

JorTurFer · 2024-02-01T14:18:29Z

I'm totally sure that this requires at least a note in documentation, but I'm not totally sure about where I should write down the info.
Could you give any suggestion about the correct place @blakepettersson ?

EDIT: Is this commit enough?

blakepettersson · 2024-02-02T08:24:32Z

@JorTurFer IMO there should be a new page under docs/operator-manual describing how to profile the app. The location of the new doc would also need to be added to mkdocs.yml

todaywasawesome · 2024-02-06T16:57:35Z

@JorTurFer I converted this to a draft until the docs are ready.

JorTurFer · 2024-02-06T17:00:49Z

@JorTurFer I converted this to a draft until the docs are ready.

yeah! Sorry, I want to finish this during the week but I forgot to make it a draft

agilgur5

I'd suggest putting this behind an env var flag, as we have in Workflows. That way it's not running profilers (that use CPU and memory themselves and reveal debug info) by default, as the vast majority of usage would not need it

JorTurFer · 2024-02-08T17:13:28Z

I'd suggest putting this behind an env var flag, as we have in Workflows. That way it's not running profilers (that use CPU and memory themselves and reveal debug info) by default, as the vast majority of usage would not need it

Thanks for the feedback! The profiler isn't enabled as default, you have to set a command arg to enable it. As default this line is false: https://github.com/argoproj/argo-cd/pull/17068/files#diff-d9546c5501e320265b2650f92869618fd0c75f9b2caab740461b0b44223ad9c9R21

Using command arg or env isn't a problem, I can switch to (true/false) env to be aligned, but in that case the port has to be hardcoded or provided from other source. That's why to avoid 2 parameters I took the same approach as controller-runtime took: kubernetes-sigs/controller-runtime@482ed05#diff-d500fbd6a2aa620607ca5e2a7c3ac4f1a4c82309d1a549561e92abfcb18f2f0eR594-R604

Given that, I want to work on the pending doc this weekend, so it'd be nice if we decide the enabling approach xD

agilgur5 · 2024-02-08T17:35:50Z

Thanks for the feedback! The profiler isn't enabled as default, you have to set a command arg to enable it. As default this line is false: https://github.com/argoproj/argo-cd/pull/17068/files#diff-d9546c5501e320265b2650f92869618fd0c75f9b2caab740461b0b44223ad9c9R21

Oh. That seems like an unintuitive way of handling it. It also does still create and run a server{} if the address isn't provided?

Using command arg or env isn't a problem, I can switch to (true/false) env to be aligned, but in that case the port has to be hardcoded or provided from other source. That's why to avoid 2 parameters I took the same approach as controller-runtime took: kubernetes-sigs/controller-runtime@482ed05#diff-d500fbd6a2aa620607ca5e2a7c3ac4f1a4c82309d1a549561e92abfcb18f2f0eR594-R604

Surprised to see controller-runtime doing it that way. They do return nil though.

With an env var you could do something similar, i.e. ARGO_PPROF="6060". If the env var exists and the port number is valid, start the pprof server, otherwise skip

agilgur5 · 2024-02-08T17:37:23Z

pkg/pprof/server.go

+	mux.HandleFunc("/debug/pprof/", pprof.Index)
+	mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
+	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
+	mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
+	mux.HandleFunc("/debug/pprof/trace", pprof.Trace)


might want to remove this other code that added handlers previously: #7533

lol, I didn't realize that the code was there 🤦
but, does ArgoCD already have support for enabling the profiler using vars?

I mean, maybe I've added a feature that was already supported

Yea I was just responding to someone on Slack about this and found the old PR and your PR and issue. Complete coincidence that I stumbled upon this, especially as I primarily contribute to Workflows.

but, does ArgoCD already have support for enabling the profiler using vars?

it seems to be unconditionally enabled on the metrics port(?) if I'm reading correctly. that would be a security issue if so (albeit a minor one, as it's just debug info of an OSS project), and a perf issue too.

also that means the port isn't customizable

Okey, we will replace that with my current approach, right? I mean, I have to remove the old code and just keep my PR

Yes, the title in particular is used in the changelog and the prefix is typically used for cherry-picking (although CD has a bit different process for that). This might be a fix! since it's technically breaking?

is the ! required too in the title?

In conventional commits, the ! is an indication of a breaking change. This does remove the previously default enabled access to the pprof endpoints and so is technically breaking.

So the previous support for pprof actually only added it to the Application Controller and Server (some of the components may not have existed at the time; I think only support for the Repo Server was missing at the time of #7533), so this PR also adds it to the rest... which is technically a feature... 😖

So maybe this makes sense as a feat!: that is still backported due to the security issue? I'll let the CD Maintainers decide how to handle that 😅

No problem , feat!:! xD

JorTurFer · 2024-02-08T17:41:23Z

They do return nil though.

That's true xD

With an env var you could do something similar, i.e. ARGO_PPROF="6060". If the env var exists and the port number is valid, start the pprof server, otherwise skip

Yeah, for sure. I'm happy changing to ARGO_PPROF=Port. My concern was just with ARGO_PPROF=true/false as it hardcodes the port, but ARGO_PPROF=Port seems aligned with other projects within the org, so it's nice.

agilgur5 · 2024-02-08T20:58:56Z

but ARGO_PPROF=Port seems aligned with other projects within the org, so it's nice.

yea I could also change Workflows to support a port number as well. now that I'm looking at it in-detail, would be a good time

agilgur5 · 2024-02-08T21:02:44Z

@JorTurFer IMO there should be a new page under docs/operator-manual describing how to profile the app. The location of the new doc would also need to be added to mkdocs.yml

Could probably put this just after the "Metrics" page in order so that they're right next to each other as they're related

JorTurFer · 2024-02-09T00:05:24Z

Hi!
I've just addressed the comments and updated docs. PTAL, from my side, it is ready to review

agilgur5

All docs comments below

mkdocs.yml

docs/operator-manual/pprof.md

JorTurFer · 2024-02-09T07:53:20Z

Wow, thanks for the in depth review! ❤️

docs/operator-manual/profiling.md

agilgur5

There's a short-hand for log.Error followed by os.Exit(1) -- log.Fatal

Otherwise LGTM other than the PR title and description per in-line comments.

Will pass off to CD folks

cmd/argocd-application-controller/commands/argocd_application_controller.go

cmd/argocd-applicationset-controller/commands/applicationset_controller.go

cmd/argocd-cmp-server/commands/argocd_cmp_server.go

cmd/argocd-application-controller/commands/argocd_application_controller.go

cmd/argocd-server/commands/argocd_server.go

agilgur5 · 2024-02-09T22:18:51Z

pkg/pprof/server.go

+	mux.HandleFunc("/debug/pprof/", pprof.Index)
+	mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
+	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
+	mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
+	mux.HandleFunc("/debug/pprof/trace", pprof.Trace)


Yes, the title in particular is used in the changelog and the prefix is typically used for cherry-picking (although CD has a bit different process for that). This might be a fix! since it's technically breaking?

JorTurFer · 2024-02-09T23:00:13Z

There's a short-hand for log.Error followed by os.Exit(1) -- log.Fatal

Otherwise LGTM other than the PR title and description per in-line comments.

Will pass off to CD folks

Thanks a lot for the guidance!

agilgur5 · 2024-02-09T23:09:37Z

docs/operator-manual/profiling.md

+`pprof` has two modes: interactive and non-interactive. Non-interactive mode generates profiling data for future analysis. Interactive mode launches a web server to visualize the profiling data in real-time.
+
+!!! Note
+    In the below commands, replace http://localhost:6060 with the appropriate URL of your Argo component.


Just realized, we might want to recommend that users port-forward the pprof port to their localhost, since the port should generally not be accessible via Ingress (unlike the UI, for example)

I used localhost to implicitly suggest the port-forward, but you are right and in this case, more explicit is better as exposing this server could be risky

JorTurFer · 2024-02-13T21:34:07Z

The vendor can explain how to use pprof to get e.g. stack trace to troubleshoot an issue.

So, if the vendor will explain how to use pprof, why can't they explain how to enable it too? At the end of the day, they have to explain the process

alexmt · 2024-02-13T21:51:43Z

@agilgur5

I don't know why this needs a strong reason, nor why security is a "weak" reason.

Because "vendors will need to instruct customers to re-enable pprof to efficiently debug issues." . Just to avoid any confusion: vendor, of course, doesn't have access to the customer's argocd. They will have to ask customer to customize their argocd installation to re-enable pprof. The change is small, but it is yet another customization that needs to be communicated, explained and maintained. So I feel it worths to have a discussion before making a change.

As I mentioned, both metrics and the pprof endpoint are not accessible outside of the cluster. If we agree that hosting pprof handlers on the metrics port constitutes a CVE, then does it mean that an unprotected metrics port is a CVE as well? If yes, how do we fix it?

I don't know what all the resistance is to a very simple patch in this PR

I literally asked one question.

I would really, strongly advise against repeatedly encouraging poor security practice, in public no less.

Thank you for advice. I would advice to keep conversation friendly and constructive.

crenshaw-dev · 2024-02-13T22:01:28Z

@JorTurFer I'll add you to the GHSA so we can discuss details without risking revealing anything sensitive. Others on this thread are already on the GHSA.

agilgur5 · 2024-02-13T22:24:38Z

The change is small, but it is yet another customization that needs to be communicated, explained and maintained

I mean, you could say the same thing about the entirety of Argo CD's configuration (or any configuration at all, for that matter). Many of those toggles are used significantly more frequently than pprof. Such an argument is a strawman.

I'll repeat again, that we did literally create the docs in this PR and that this was undocumented before. Both the author and I literally arrived here because we had no idea about this insecure default. We are doing that very communication and explanation in this PR, and improving the maintenance with a security patch.

then does it mean that an unprotected metrics port is a CVE as well?

Once more, the metrics should be authenticated -- and therefore not "unprotected" -- and the metrics are not debug-level information.

I literally asked one question.

You rejected the PR on behalf of all Approvers. You are also a Lead on CD, your comments and actions have more authority than any other person -- please recognize your own position.

It's one thing to ask a question in a comment, it is a different thing to reject a security patch outright, suggest it is not a vulnerability at all, and superimpose your opinion, one in which you repeatedly suggested poor security practices ("secure enough"), made incorrect statements ("no one can access it"), and asked to publicly reveal what else might be possible. The amount of security red flags in your statements is quite high.
You also are superimposing your opinion over a widely recognized CWE as well as SIG Security.

and constructive

A constructive discussion also involves engaging with all the material. You rejected this outright in a few short sentences. My disclosure is substantially longer than that, and I provided some details here as well. You did not address the CWE, related CVEs, nor Workflows's own patch what-so-ever. You had several opportunities to do so, and you did not engage with the material, particularly not before rejecting the patch.
I provided plenty of information; I would ask that you engage thoroughly with all the details of a disclosure from the very beginning.

alexmt · 2024-02-13T22:55:36Z

@agilgur5 I've asked for clarification: I'm just trying to understand how exposing the pprof endpoint is more dangerous than exposing the metrics endpoint.

You rejected the PR on behalf of all Approvers.
You also are superimposing your opinion over a widely recognized CWE as well as SIG Security.
You rejected this outright in a few short sentences.

I disagree with these statements. I believe I've just joined a conversation and asked a question. I must do it as a maintainer. So many personal accusations make it difficult to focus on the point of this PR. I'm proposing to keep the discussion technical. It would really help to keep moving this change forward.

JorTurFer · 2024-02-13T23:03:34Z

I strongly suggest moving this conversation to the GHSA to prevent sharing sensitive information or projecting a wrong message

jannfis · 2024-02-14T18:52:32Z

Can we please keep the discussion technical, and not make any personal attacks here?

agilgur5 · 2024-02-14T19:11:41Z

I strongly suggest moving this conversation to the GHSA to prevent sharing sensitive information or projecting a wrong message

💯 that's why I made it, referred to it, and declined to go too far into it here. Thanks!

Can we please keep the discussion technical, and not make any personal attacks here?

To be clear, I did not "attack" anyone, I strongly pushed back against repeated suggestions for poor security practices. I believe we all have an ethical obligation to do the same (as Engineers, as Maintainers, etc). Standing against poor security practices is not an "attack", and in fact is quite technical.

I disagree with these statements. I believe I've just joined a conversation and asked a question.

To be clear, rejecting a PR is rejecting a PR -- I don't know how else to say that. That's not my interpretation, that's what the action in GitHub does.
You don't have to reject a PR and request insecure changes if you're asking a question. You don't have to approve it nor reject it. You can just leave a comment. GH's code review even has a third option which is leaving a comment.
It is also literally "on behalf of all Approvers":

Those are not "personal accusations", those are statements of fact. Those actions were in fact made and I merely pointed out that they were made...

gdsoumya · 2024-02-15T06:30:46Z

To be clear, rejecting a PR is rejecting a PR -- I don't know how else to say that. That's not my interpretation, that's what the action in GitHub does.

@agilgur5 I think you are confusing a change request with rejection. There is no point in continuing this discussion but fyi a rejection is outright closing the PR which none of the maintainers did. This was a very civil discussion where maintainers of the project put out their opinions(which they are very much entitled to) keeping the best in mind for the project and the users. You might have a differing opinion but there's no justifying your rude behaviour.

It is also literally "on behalf of all Approvers":

That's because all maintainers are part of the approvers github group, my review comment also had the same tag. It does not represent the opinions of all the maintainers of the project neither does it mean a PR is rejected.

agilgur5 · 2024-02-15T16:21:22Z

a rejection is outright closing the PR which none of the maintainers did

I've never worked in any organization where a code review rejection was not addressed -- they have always been considered blocking denials. In this case two of them said to intentionally make something insecure, which is the most important part of this PR. To be clear, I was appalled, horrified to see that not once, but twice with increasing authority and even more insecure suggestions. The ramifications of that are deeply, deeply concerning to me, so much so that I have been reconsidering contributing to Argo entirely. The security culture is critical to any organization -- it is a pre-requisite for patches to land and land timely. Insecure statements from any position of authority reverberate through an organization and its followers/users. I have an ethical obligation to correct and limit that effect as soon as possible, which is why this has been at the top of my priorities for days (despite this not being a very high risk vuln IMO -- I explained the difference between severity and risk in the disclosure, effectively CVSS v3 vs v4). I am ethically obligated to defend the security of any system I am involved with, as much as humanly possible.

In any organization I've ever worked in, one would make those changes or the PR would indeed eventually be closed.
In this case, making those changes would be to extend a vulnerability, which I am vehemently against and have an ethical obligation to prevent as much as humanly possible. The security context is of utmost importance. Without it, I would not care much and, frankly, would have agreed with the changes to make it non-breaking -- heck I would have said that myself before an Approver even responded.
Context is key.

You might have a differing opinion but there's no justifying your rude behaviour.

I don't think I was rude. I do think my tone was harsh & forceful, no doubt, I do not disagree. That very much reflects my aghast, shocked reaction plus my ethical obligation.

Notably, this does very much feel like tone policing, which is a form of ad hominem attack. Ad hominem attacks are, notably, exclusively non-technical, by definition.

Certainly had I known that would be the case beforehand, I may have approached it differently, but that was so unexpected for me at the time that I began panicking ("sh*tting my pants" in American idioms) at the ramifications of such statements. Security statements are of grave importance, particularly when made in public in a position of authority, and should not be made lightly.

Further, it very much seemed that the supporting material I provided was not engaged with (whereas it was by SIG Security), which felt disrespectful and incredibly dismissive to me and those authors. If you want to bring in "rudeness", I would say such dismissal is not only very rude but also the first instance of such "rudeness" in this thread, and could very well be considered a violation of the CoC. The material is all exclusively technical, yet was not even addressed here.

my review comment

I recognized that you did not have access to the disclosure and may not be involved in security matters as such. I was still forceful in my reaction to your comment, per my ethical obligation, but not nearly as much as later since I recognized you did not have context and may not have in-depth experience with these matters. I was intensely more aghast when someone with higher authority who had seen the disclosure (commented there before here) doubled down on it and proceeded to make other insecure statements as well. That made me go from pretty concerned to downright horrified panicking.

terrytangyuan · 2024-02-15T19:21:26Z

You rejected the PR on behalf of all Approvers.

@agilgur5 This is the way GitHub works since Alex is part of the approvers team. When anyone from the approvers team requests changes on the PR, GitHub adds "requested changes on behalf of argocd-approvers" automatically.

You don't have to reject a PR and request insecure changes if you're asking a question. You don't have to approve it nor reject it. You can just leave a comment. GH's code review even has a third option which is leaving a comment.

I think you are misinterpreting this. "Requesting changes" is not the same thing as "rejecting". I think Alex's intention is simply to ask more questions and gather feedback to reach agreement before approving and merging by other approvers.

Let's keep this discussion technical.

woehrl01 · 2024-02-19T05:58:57Z

@JorTurFer this looks very promising. Could you please add the capability to enable block and mutex profiling? By setting some envs maybe?

I'd say this makes sense as there are lot of locking operations in argocd and it's services.

See: https://github.com/DataDog/go-profiler-notes/blob/main/guide%2FREADME.md

JorTurFer · 2024-02-21T10:29:33Z

@JorTurFer this looks very promising. Could you please add the capability to enable block and mutex profiling? By setting some envs maybe?

I'd say this makes sense as there are lot of locking operations in argocd and it's services.

See: DataDog/go-profiler-notes@main/guide%2FREADME.md

I'll check it. This week I'm quite disconnected due to a business trip but I'll tackle this asap and update the PR with them (or a reason for not adding them if it's not possible xD)

- merge conflicts fixed by agilgur5 Signed-off-by: Jorge Turrado <[email protected]>

Signed-off-by: Jorge Turrado <[email protected]>

Co-authored-by: Anton Gilgur <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

Signed-off-by: Jorge Turrado <[email protected]>

Co-authored-by: Anton Gilgur <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

Signed-off-by: Jorge Turrado <[email protected]>

Co-authored-by: Anton Gilgur <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

blakepettersson · 2025-01-21T14:59:59Z

@JorTurFer what happened to this issue? I understand there previously was some heated discussion..

JorTurFer · 2025-01-22T22:17:22Z

Hello
There was a discussion about this and I think that it was solved with other PR. IIRC, this can be closed

JorTurFer changed the title ~~feat: Add an integrated Pprof server enabled on demand~~ feat: Add an integrated Pprof server enabled on demand (#16594) Feb 1, 2024

JorTurFer mentioned this pull request Feb 1, 2024

Support controller-runtime integration for pprof #16594

Open

JorTurFer marked this pull request as ready for review February 1, 2024 15:33

JorTurFer requested review from a team as code owners February 1, 2024 15:33

todaywasawesome marked this pull request as draft February 6, 2024 16:57

agilgur5 reviewed Feb 8, 2024

View reviewed changes

JorTurFer marked this pull request as ready for review February 9, 2024 00:04

agilgur5 reviewed Feb 9, 2024

View reviewed changes

agilgur5 added security Security related type:scalability Issues related to scalability and performance related issues labels Feb 9, 2024

agilgur5 reviewed Feb 9, 2024

View reviewed changes

docs/operator-manual/profiling.md Outdated Show resolved Hide resolved

agilgur5 reviewed Feb 9, 2024

View reviewed changes

JorTurFer changed the title ~~feat: Add an integrated Pprof server enabled on demand (#16594)~~ fix: Enable pprof server only on demand (#16594) Feb 9, 2024

agilgur5 reviewed Feb 9, 2024

View reviewed changes

JorTurFer changed the title ~~fix: Enable pprof server only on demand (#16594)~~ fix!: Enable pprof server only on demand (#16594) Feb 9, 2024

JorTurFer and others added 12 commits July 24, 2024 21:34

feat: Add an integrated Pprof server enabled on demand

2f76bb7

- merge conflicts fixed by agilgur5 Signed-off-by: Jorge Turrado <[email protected]>

update docs

57f916c

- merge conflicts fixed by agilgur5 Signed-off-by: Jorge Turrado <[email protected]>

Apply feedback

1461aa6

- merge conflicts fixed by agilgur5 Signed-off-by: Jorge Turrado <[email protected]>

revert args

6238d8c

- merge conflicts fixed by agilgur5 Signed-off-by: Jorge Turrado <[email protected]>

Add documentation about pprof

4b139ca

Signed-off-by: Jorge Turrado <[email protected]>

remove var

85b3a3d

Signed-off-by: Jorge Turrado <[email protected]>

Apply suggestions from code review

87aa849

Co-authored-by: Anton Gilgur <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

rename file

3c0ee25

Signed-off-by: Jorge Turrado <[email protected]>

Update docs/operator-manual/profiling.md

878f181

Co-authored-by: Anton Gilgur <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

apply feedback

3ae08b4

Signed-off-by: Jorge Turrado <[email protected]>

update comment

b71bc4d

Signed-off-by: Jorge Turrado <[email protected]>

Update docs/operator-manual/profiling.md

e109d52

Co-authored-by: Anton Gilgur <[email protected]> Signed-off-by: Jorge Turrado Ferrero <[email protected]>

agilgur5 force-pushed the pprof branch from 93ef38f to e109d52 Compare July 31, 2024 21:33

blakepettersson mentioned this pull request Jan 22, 2025

feat!: Enable pprof server on demand (#16594) #21622

Draft

14 tasks

JorTurFer closed this Jan 22, 2025

feat!: Enable pprof server on demand (#16594) #17068

feat!: Enable pprof server on demand (#16594) #17068

Conversation

JorTurFer commented Feb 1, 2024 • edited Loading

codecov bot commented Feb 1, 2024 • edited Loading

Codecov Report

JorTurFer commented Feb 1, 2024 • edited Loading

blakepettersson commented Feb 2, 2024

todaywasawesome commented Feb 6, 2024

JorTurFer commented Feb 6, 2024

agilgur5 left a comment • edited Loading

Choose a reason for hiding this comment

JorTurFer commented Feb 8, 2024 • edited Loading

agilgur5 commented Feb 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agilgur5 Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JorTurFer commented Feb 8, 2024

agilgur5 commented Feb 8, 2024

agilgur5 commented Feb 8, 2024

JorTurFer commented Feb 9, 2024

agilgur5 left a comment

Choose a reason for hiding this comment

JorTurFer commented Feb 9, 2024

agilgur5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JorTurFer commented Feb 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JorTurFer commented Feb 13, 2024

alexmt commented Feb 13, 2024

crenshaw-dev commented Feb 13, 2024

agilgur5 commented Feb 13, 2024 • edited Loading

alexmt commented Feb 13, 2024

JorTurFer commented Feb 13, 2024 • edited Loading

jannfis commented Feb 14, 2024

agilgur5 commented Feb 14, 2024 • edited Loading

gdsoumya commented Feb 15, 2024 • edited Loading

agilgur5 commented Feb 15, 2024 • edited Loading

terrytangyuan commented Feb 15, 2024 • edited Loading

woehrl01 commented Feb 19, 2024

JorTurFer commented Feb 21, 2024

blakepettersson commented Jan 21, 2025

JorTurFer commented Jan 22, 2025

JorTurFer commented Feb 1, 2024 •

edited

Loading

codecov bot commented Feb 1, 2024 •

edited

Loading

JorTurFer commented Feb 1, 2024 •

edited

Loading

agilgur5 left a comment •

edited

Loading

JorTurFer commented Feb 8, 2024 •

edited

Loading

agilgur5 Feb 8, 2024 •

edited

Loading

agilgur5 commented Feb 13, 2024 •

edited

Loading

JorTurFer commented Feb 13, 2024 •

edited

Loading

agilgur5 commented Feb 14, 2024 •

edited

Loading

gdsoumya commented Feb 15, 2024 •

edited

Loading

agilgur5 commented Feb 15, 2024 •

edited

Loading

terrytangyuan commented Feb 15, 2024 •

edited

Loading