-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
1 parent
08d9af8
commit a905400
Showing
14 changed files
with
80 additions
and
68 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# Instructions | ||
|
||
Ensure that you have obtained `analysis-logs/input/access.log`. | ||
|
||
The script `analysis-logs/input.sh` downloads this file with the Kaggle API. In | ||
order to use the Kaggle API, you must have the API token: `~/.kaggle/kaggle.json`. | ||
|
||
From <https://www.kaggle.com/docs/api>: | ||
|
||
> In order to use the Kaggle’s public API, you must first authenticate using an | ||
> API token. Go to the 'Account' tab of your user profile and select 'Create New | ||
> Token'. This will trigger the download of kaggle.json, a file containing your | ||
> API credentials. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
105b333d6b49399cc10763a5f629343a results.full/out |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
105b333d6b49399cc10763a5f629343a results.small/out |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
#!/bin/bash | ||
|
||
REPO_TOP=$(git rev-parse --show-toplevel) | ||
DIR=$REPO_TOP/analysis-logs/input | ||
mkdir -p $DIR | ||
|
||
# Set up Kaggle API | ||
if [[ ! -d ~/.kaggle ]]; then | ||
mkdir ~/.kaggle | ||
echo "Place your kaggle.json in the ~/.kaggle directory." | ||
fi | ||
chmod 600 ~/.kaggle/kaggle.json | ||
|
||
cd $DIR | ||
kaggle datasets download -d eliasdabbas/web-server-access-logs | ||
unzip web-server-access-logs | ||
rm -f web-server-access-logs.zip client_hostname.csv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
|
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
#!/bin/bash | ||
|
||
REPO_TOP=$(git rev-parse --show-toplevel) | ||
|
||
eval_dir="${REPO_TOP}/analysis-logs" | ||
results_dir="${eval_dir}/results" | ||
scripts_dir="${eval_dir}/scripts" | ||
input_dir="${eval_dir}/input" | ||
mkdir -p $results_dir | ||
|
||
export INPUT=${input_dir}/access.log | ||
|
||
suffix=".full" | ||
if [[ "$@" == *"--small"* ]]; then | ||
suffix=".small" | ||
fi | ||
|
||
log_dir="$results_dir/results$suffix" | ||
mkdir -p $log_dir | ||
$scripts_dir/nginx.sh > $log_dir/out |
0
analysis-logs/nginx.sh → analysis-logs/scripts/nginx.sh
100644 → 100755
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
#!/bin/bash | ||
|
||
REPO_TOP=$(git rev-parse --show-toplevel) | ||
|
||
eval_dir="${REPO_TOP}/analysis-logs" | ||
input_dir="${eval_dir}/input" | ||
hashes_dir="${eval_dir}/hashes" | ||
results_dir="${eval_dir}/results" | ||
mkdir -p $results_dir | ||
|
||
suffix=".full" | ||
if [[ "$@" == *"--small"* ]]; then | ||
suffix=".small" | ||
fi | ||
|
||
cd $results_dir # md5sum computes paths relative to cd | ||
if [[ "$@" == *"--generate"* ]]; then | ||
md5sum results$suffix/* > $hashes_dir/results$suffix.md5sum | ||
fi | ||
|
||
okay=0 | ||
if ! md5sum --check --quiet $hashes_dir/results$suffix.md5sum; then | ||
okay=1 | ||
echo "img_convert $suffix failed verification" | ||
fi | ||
exit $okay |