Scaling LLMs for Science and Ongoing Collaborations

Sam Foreman

foremans@anl.gov

Argonne National Laboratory

2023-08-17

Scaling LLMs for Science
(& Ongoing Collaborations)

 
Sam Foreman
Venkat Vishwanath
 

saforem2/{scaling4science, Megatron-DS-Benchmarking}

Loooooooooong Sequence Lengths

• Working with Microsoft DeepSpeed team to enable longer sequence lengths (context windows) for LLMs

\hspace{30pt}

Figure 1: Maximum (achievable) SEQ_LEN for both 25B and 33B models [WIP]

Ongoing Work & Collaborations

Scaling LLMs

• saforem2/Megatron-DS-Benchmarking

Climate Modeling

• ViT for Climate Models [WIP]

• ClimRR: Climate Risk & Resiliency Portal

Lattice QCD

• • saforem2/l2hmc-qcd
• MLMC @ Lattice 23
  • saforem2/lattice23
• LQCD @ PASC 23
  • saforem2/lqcd-pasc23

Thank you!

• Link to slides

• Huge shout out to

  • Venkat Vishwanath
  • James Osborn
  • Xiao-Yong Jin
  • Rao Kotamarthi
  • Romit Maulik

  • Troy Arcomano
  • Microsoft DeepSpeed Team
  • ALCF Data Science Team (everyone!)
    • ALCF Staff (Ops, Performance, Software, User Support / Documentation, …)

Acknowledgements

This research used resources of the Argonne Leadership Computing Facility,
which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357.