Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
just analysis-logs
Browse files Browse the repository at this point in the history
EtomicBomb committed Oct 20, 2024
1 parent 08d9af8 commit a905400
Showing 14 changed files with 80 additions and 68 deletions.
13 changes: 13 additions & 0 deletions analysis-logs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Instructions

Ensure that you have obtained `analysis-logs/input/access.log`.

The script `analysis-logs/input.sh` downloads this file with the Kaggle API. In
order to use the Kaggle API, you must have the API token: `~/.kaggle/kaggle.json`.

From <https://www.kaggle.com/docs/api>:

> In order to use the Kaggle’s public API, you must first authenticate using an
> API token. Go to the 'Account' tab of your user profile and select 'Create New
> Token'. This will trigger the download of kaggle.json, a file containing your
> API credentials.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

REPO_TOP=$(git rev-parse --show-toplevel)
results_dir="${REPO_TOP}/covid-mts/results"
results_dir="${REPO_TOP}/analysis-logs/results"

echo "Cleaning up outputs..."
rm -rf $results_dir
1 change: 1 addition & 0 deletions analysis-logs/hashes/results.full.md5sum
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
105b333d6b49399cc10763a5f629343a results.full/out
1 change: 1 addition & 0 deletions analysis-logs/hashes/results.small.md5sum
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
105b333d6b49399cc10763a5f629343a results.small/out
17 changes: 17 additions & 0 deletions analysis-logs/input.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

REPO_TOP=$(git rev-parse --show-toplevel)
DIR=$REPO_TOP/analysis-logs/input
mkdir -p $DIR

# Set up Kaggle API
if [[ ! -d ~/.kaggle ]]; then
mkdir ~/.kaggle
echo "Place your kaggle.json in the ~/.kaggle directory."
fi
chmod 600 ~/.kaggle/kaggle.json

cd $DIR
kaggle datasets download -d eliasdabbas/web-server-access-logs
unzip web-server-access-logs
rm -f web-server-access-logs.zip client_hostname.csv
1 change: 1 addition & 0 deletions analysis-logs/input/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

4 changes: 0 additions & 4 deletions analysis-logs/input/.kaggle/kaggle.json

This file was deleted.

1 change: 0 additions & 1 deletion analysis-logs/input/checksum.md5

This file was deleted.

30 changes: 0 additions & 30 deletions analysis-logs/input/input.sh

This file was deleted.

17 changes: 0 additions & 17 deletions analysis-logs/input/run.sh

This file was deleted.

15 changes: 0 additions & 15 deletions analysis-logs/input/verify.sh

This file was deleted.

20 changes: 20 additions & 0 deletions analysis-logs/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/bash

REPO_TOP=$(git rev-parse --show-toplevel)

eval_dir="${REPO_TOP}/analysis-logs"
results_dir="${eval_dir}/results"
scripts_dir="${eval_dir}/scripts"
input_dir="${eval_dir}/input"
mkdir -p $results_dir

export INPUT=${input_dir}/access.log

suffix=".full"
if [[ "$@" == *"--small"* ]]; then
suffix=".small"
fi

log_dir="$results_dir/results$suffix"
mkdir -p $log_dir
$scripts_dir/nginx.sh > $log_dir/out
File renamed without changes.
26 changes: 26 additions & 0 deletions analysis-logs/verify.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash

REPO_TOP=$(git rev-parse --show-toplevel)

eval_dir="${REPO_TOP}/analysis-logs"
input_dir="${eval_dir}/input"
hashes_dir="${eval_dir}/hashes"
results_dir="${eval_dir}/results"
mkdir -p $results_dir

suffix=".full"
if [[ "$@" == *"--small"* ]]; then
suffix=".small"
fi

cd $results_dir # md5sum computes paths relative to cd
if [[ "$@" == *"--generate"* ]]; then
md5sum results$suffix/* > $hashes_dir/results$suffix.md5sum
fi

okay=0
if ! md5sum --check --quiet $hashes_dir/results$suffix.md5sum; then
okay=1
echo "img_convert $suffix failed verification"
fi
exit $okay

0 comments on commit a905400

Please sign in to comment.