Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --force option to fetch sources even if they are already present #81

Merged
merged 5 commits into from
Jan 6, 2025
Merged

Add --force option to fetch sources even if they are already present #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9fe7de3
add `--force` option to fetch script
m-fila Dec 17, 2024
2e738a8
split missing file and forceful fetch scenarios
m-fila Dec 18, 2024
bc1600a
update readme
m-fila Dec 18, 2024
86badcb
print usage on argparse error
m-fila Dec 18, 2024
6441659
print script basename instead of full path in usage
m-fila Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 47 additions & 14 deletions .github/scripts/fetch_external_sources.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,37 @@
#!/usr/bin/env bash
# Script that tries to automatically fetch further documentation from other
# github repositories, where it simply assumes that they are available under the
# same name. Runs through our ususal suspects of github organizations while
# same name. Runs through our usual suspects of github organizations while
# trying to do so

# Initialize force option
FORCE=false

# Display usage message
display_usage() {
echo "Usage: $(basename $0) [--force|-f] [--help|-h]"
}

# Parse command line arguments
for arg in "$@"; do
case $arg in
--force|-f) FORCE=true ;;
--help|-h)
display_usage
echo ""
echo "Options:"
echo " --force, -f Force fetching files even if they already exist"
echo " --help, -h Display this help message"
exit 0
;;
*)
echo "Unknown parameter passed: $arg"
display_usage
exit 1
;;
esac
done

# Try to fetch a file from a github repository
try_fetch() {
local org=${1}
Expand All @@ -30,22 +58,27 @@ fetch_for_file() {
if [ -n "${line}" ] && [[ "${line}" == *.md ]] || [[ "${line}" == *.png ]]; then
# If the file exists do nothing, otherwise pull it in from github
local file_to_fetch=${file_dir}/${line}
if ! ls "${file_to_fetch}" > /dev/null 2>&1; then
if [ "${FORCE}" = true ]; then
echo "Force option enabled. Trying to fetch '${line}' from github"
elif ! ls "${file_to_fetch}" > /dev/null 2>&1; then
echo "${line} does not exist. Trying to fetch it from github"
local outputdir=$(dirname ${file_to_fetch})
mkdir -p ${outputdir} # make the directory for the output

# Try a few github organizations
for org in key4hep HEP-FCC AIDASoft iLCSoft; do
echo "Trying to fetch from github organization: '${org}'"
if try_fetch ${org} ${line} ${file_dir}; then
echo "Fetched succesfully from organization '${org}'"
break
fi
done
else
continue
fi

# Check again if we hav succesfully fetched the file
local outputdir=$(dirname ${file_to_fetch})
mkdir -p ${outputdir} # make the directory for the output

# Try a few github organizations
for org in key4hep HEP-FCC AIDASoft iLCSoft; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, I think it should be fairly straight forward to make it possible to get content from (user) forks here by just pre-pending this list with more "organizations" (aka user names) where we could also take input via CL arguments.

The only downside to the dumb approach then is that we will give precedence to all user forks, when users potentially just want to give precedence to one repository.

Also we would need to make try_fetch actually consider more branches than main and master.

Maybe it's not so straight forward after all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I think I'd be fine with fetching official documentation from the main/master branches of approved repositories and manually injecting some WIP pages for preview if needed.
I'd like to avoid a situation that by accident the fetch prefers to take some page from my outdated fork instead of upstream repo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that was my concern as well. If we want an override here it needs to be package specific. This is already a step forward as it is.

echo "Trying to fetch from github organization: '${org}'"
if try_fetch ${org} ${line} ${file_dir}; then
echo "Fetched successfully from organization '${org}'"
break
fi
done

# Check again if we have successfully fetched the file
if ! ls "${file_to_fetch}" > /dev/null 2>&1; then
echo "Could not fetch file '${line}' from external sources" 1>&2
exit 1
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ First fetch the documentation pages from other key4hep repositories:
.github/scripts/fetch_external_sources.sh
```

If the sources already exist but you want to update them, use the `--force` option.

Then build the site locally:

```sh
Expand Down
Loading