models

History

…
..
experiments	…
README.md	…
__init__.py	…
roformer.py	…
roformer_attention.py	…
roformer_attention_test.py	…
roformer_encoder.py	…
roformer_encoder_block.py	…
roformer_encoder_block_test.py	…
roformer_encoder_test.py	…
roformer_experiments.py	…
train.py	…

README.md

Code for Roformer.

Run with

DATA_PATH=???
OUTPUT_DIR=???
python3 train.py \
  --experiment=roformer/pretraining \
  --config_file=experiments/roformer_base.yaml \
  --params_override="task.validation_data.input_path=${DATA_PATH},runtime.distribution_strategy=tpu" \
  --tpu=local \
  --model_dir=${OUTPUT_DIR} \
  --mode=train_and_eval