Skip to content

Latest commit

 

History

History
50 lines (39 loc) · 1.17 KB

training.md

File metadata and controls

50 lines (39 loc) · 1.17 KB

Training

You need an additional package train.

pip install -U 'pilota[ja-line,train] @ git+https://github.com/megagonlabs/pilota'

Training for dialogs

  • Needed corpus
    • asdc
    • (optional, internal only) scud_internal
OUTPUT=/path/to/output
make -j1 -f ./train.mk \
    OUTPUT="${OUTPUT}" \
    T5BASE=megagonlabs/t5-base-japanese-web-8k \
    BATCH=100 BATCH_DEV=100 EPOCH=20 IN_LEN=128 OUT_LEN=64 BATCH_PRED=100 \
    all

Training for Jalan reviews

OUTPUT=/path/to/output
make -j1 -f ./train.mk \
    OUTPUT="${OUTPUT}" \
    T5BASE=megagonlabs/t5-base-japanese-web-8k \
    BATCH=86 BATCH_DEV=112 EPOCH=20 IN_LEN=128 OUT_LEN=64 BATCH_PRED=120 JALAN=1 \
    all

Training for Scud2Query

OUTPUT=/path/to/output
make -j1 -f ./train.mk \
    OUTPUT="${OUTPUT}" \
    T5BASE=megagonlabs/t5-base-japanese-web-8k \
    BATCH=86 BATCH_DEV=112 EPOCH=20 IN_LEN=128 OUT_LEN=64 BATCH_PRED=120 SCUD2QUERY=1 \
    all