Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reproducibility of pipeline outputs #34

Open
4 tasks
ashsong-nv opened this issue Dec 12, 2024 · 1 comment
Open
4 tasks

Improve reproducibility of pipeline outputs #34

ashsong-nv opened this issue Dec 12, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@ashsong-nv
Copy link
Collaborator

Add fields to the pipeline output that would help improve pipeline reproducibility. This would help with accuracy evaluation and improvement efforts.

Some ideas of fields to add to the output:

  • Input config (such as model parameters)
  • Pipeline config (such as whether to use the code search tool, whether the QA tool should return source documents)
  • Rendered prompts
  • Specific git commit hash when the input ref is a tag or branch
@ashsong-nv
Copy link
Collaborator Author

ashsong-nv commented Dec 17, 2024

@shawn-davis per discussion today, I've tested that it's possible to get the current checked out commit hash by running git rev-parse HEAD in the cloned repo directory after the checkout step. However, we might need to do some testing to find the equivalent command for doing this in the git python library. We will also need to add a new attribute to the output schema to store this.

Since there are quite a lot of changes needed to improve reproducibility, it would be good to break the tasks above into separate issues/PRs to make the PRs smaller.

@ashsong-nv ashsong-nv added the enhancement New feature or request label Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants