Language Model Training with π ezpz
and π€ HF Trainer¶
The
src/ezpz/hf_trainer.py
module provides a mechanism for distributed training with π€ huggingface /
transformers.
In particular, it allows for distributed training using the
transformers.Trainer
object with any1 (compatible) combination of
{models
,
datasets
}.
π£ Getting Started¶
-
π‘ Setup environment (on ANY {Intel, NVIDIA, AMD} accelerator)
-
π¦ Install dependencies:
-
Install π
ezpz
(from GitHub):-
Update {
tiktoken
,sentencepiece
,transformers
,evaluate
}:
-
-
βοΈ Build DeepSpeed config:
-
π Launch training:
-
πͺ Magic:
Behind the scenes, this will πͺ automagically determine the specifics of the running job, and use this information to construct (and subsequently run) the appropriate:
across all of our available accelerators.
-
β Tip:
Call:
to see the full list of supported arguments.
In particular, any
transformers.TrainingArguments
should be supported.
-
-
π DeepSpeed Support¶
Additionally, DeepSpeed is fully supported and can be configured by specifying the path to a compatible DeepSpeed config json file, e.g.:
- Build a DeepSpeed config:
- Train:
π 2 ez
-
See the full list of supported models at: https://hf.co/models?filter=text-generation ↩