You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Stanislas0 800e5b3cf1 Add dataset for megatron 2 years ago
..
__init__.py Add dataset for megatron 2 years ago
indexed_dataset.py Add dataset for megatron 2 years ago
process_pretrain_dataset.py Add dataset for megatron 2 years ago
processor.py Add dataset for megatron 2 years ago
types.py Add dataset for megatron 2 years ago
utils.py Add dataset for megatron 2 years ago