If using a vanilla task, just make sure any file(s) loaded by your dataset_fn are accessible to the TPU (i.e., are in a GCS bucket), and you should be good to go! TfdsTask Using a Taskĭepending on your data source (see above), you will need to prepare your data appropriately. You may either use a new or pre-existing Task, or you may load examples from a preprocessed TSV file. You can use these instructions to reproduce our results, fine-tune one of our released checkpoints with your own data and/or hyperparameters, or pre-train a model from scratch.
Bulk convert stat transfer how to#
The easiest way to try out T5 is with a free TPU in our Colab Tutorial.īelow we provide examples for how to pre-train, fine-tune, evaluate, and decode from a model from the command-line with our codebase. Since the HfPyTorchModel is experimental, the remainder of this README assumes usage of the MtfModel and its associated binary.Ī usage example of HfPyTorchModel is available here. If you are interested fine-tuning our models on a GPU in PyTorch, you should try the HfPyTorchModel API. If you want to use our largest models on TPUs and/or reproduce the results in our paper, you should use the MtfModel API and the t5_mesh_transformer binary. The Hugging Face API is currently experimental and subject to change, but provides a simple and easy way to load, fine-tune, and evaluate our pre-trained models using PyTorch on a single GPU. T5.models contains shims for connecting T5 Tasks and Mixtures to a model implementation for training, evaluation, and inference.Ĭurrently there are two shims available: One for the Mesh TensorFlow Transformer that we used in our paper and another for the Hugging Face Transformers library. utilities for applying these metrics at evaluation time.T5.evaluation contains two core components: We provide some predefined metrics in t5.trics.įinally, t5.data contains a Mixture class that can be instantiated to combine multiple Task datasets for multi-task training using various functions for specifying the mixture rates. You may also define a postprocess function to convert the target and prediction text to another format before calling the metric. The metric function returns a score given the target and prediction from the model. If you create your own, you must use the flags -pad_id=0 -eos_id=1 -unk_id=2 -bos_id=-1 with spm_train to be compatible with our model code. You can create your own model with the google/sentencepiece library, or use our default one at t5.data.DEFAULT_SPM_PATH. The SentencePiece model is used to tokenize the input strings and decode the output tokens. We provide many predefined preprocessors in t5.data.preprocessors, but you may also define your own. We implemented our unsupervised pre-training objectives using these token preprocessors. In addition to text preprocessing, you can also use one or more token preprocessors to modify the inputs post-tokenization. For example, the predefined t5. preprocessor converts inputs in the form The text preprocessor converts the examples in the source dataset into the appropriate format for a text-to-text model with fields for inputs and targets. The data source can be an arbitrary function that provides a tf.data.Dataset, but we also provide simpler wrappers for datasets available in TensorFlow Datasets (TFDS) (a TfdsTask) or stored as text files with one example per line (a TextLineTask). T5.data is a package for defining Task objects that provide tf.data.Datasets.Īdditionally, you may optionally provide: The t5 library can be used for future model development by providing useful modules for training and fine-tuning (potentially huge) models on mixtures of text-to-text tasks. It also provides a way to fine-tune the pre-trained models released alongside the publication.
Bulk convert stat transfer code#
The bulk of the code in this repository is used for loading, preprocessing, mixing, and evaluating datasets. In the paper, we demonstrate how to achieve state-of-the-art results on multiple NLP tasks using a text-to-text transformer pre-trained on a large text corpus. The t5 library serves primarily as code for reproducing the experiments in Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
0 kommentar(er)
