blog: initial sharding (ishard)

Co-authored-by: Alex Aizman <[email protected]> Signed-off-by: Alex Aizman <[email protected]>
NVIDIA · Aug 16, 2024 · 35bca58 · 35bca58
1 parent 18dc104
commit 35bca58
Show file tree

Hide file tree

Showing 7 changed files with 424 additions and 0 deletions.
diff --git a/docs/_posts/2024-08-16-ishard.md b/docs/_posts/2024-08-16-ishard.md
diff --git a/docs/assets/ishard/ishard-base_file_name.png b/docs/assets/ishard/ishard-base_file_name.png
diff --git a/docs/assets/ishard/ishard-full_name.png b/docs/assets/ishard/ishard-full_name.png
diff --git a/docs/assets/ishard/ishard-ml-buckets.gif b/docs/assets/ishard/ishard-ml-buckets.gif
diff --git a/docs/assets/ishard/ishard-throughput-plot.png b/docs/assets/ishard/ishard-throughput-plot.png
diff --git a/docs/assets/ishard/ishard_workflow.gif b/docs/assets/ishard/ishard_workflow.gif
diff --git a/docs/examples/ishard-imagenet/download.sh b/docs/examples/ishard-imagenet/download.sh
@@ -0,0 +1,37 @@
+# Download all training data from ImageNet official website
+# This takes approximately 200 GB disk space
+nohup wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_train.tar
+tar xf $IMAGENET_HOME/ILSVRC2012_img_train.tar -C $IMAGENET_HOME/train
+cd $IMAGENET_HOME/train
+for f in *.tar; do
+ d=`basename $f .tar`
+ mkdir $d
+ tar xf $f -C $d
+ rm $f
+done
+
+# Prepare validation data
+wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
+tar xf $IMAGENET_HOME/ILSVRC2012_img_val.tar -C $IMAGENET_HOME/validation
+
+# Prepare features
+wget https://image-net.org/data/ILSVRC/2010/ILSVRC2010_feature_sbow_train.tar
+tar xf $IMAGENET_HOME/ILSVRC2010_feature_sbow_train.tar -C $IMAGENET_HOME/label
+
+# Prepare Annotations
+wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_bbox_train_v2.tar.gz
+tar xf $IMAGENET_HOME/ILSVRC2012_bbox_train_v2.tar.gz -C $IMAGENET_HOME/train_annotation
+
+wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_bbox_val_v3.tgz
+tar xf $IMAGENET_HOME/ILSVRC2012_bbox_train_v2.tar.gz -C $IMAGENET_HOME/validation_annotation
+
+cd $IMAGENET_HOME/annotation
+for f in *.tar.gz; do
+ d=`basename $f .tar.gz`
+ mkdir $d
+ tar xf $f -C $d
+ rm $f
+done
+
+# Remove original tar files to free up disk space
+rm ILSVRC2012_bbox_train_v2  ILSVRC2012_bbox_val_v3.tgz  ILSVRC2010_feature_sbow_train.tar  ILSVRC2012_img_val.tar  ILSVRC2012_img_train.tar  nohup.out