Skip to content

🙋 Frequently Asked Questions

⚠️ Common Issues

  1. ImportError: <path-to-kernel.so>: undefined symbol: [...]

This can happen when (for whatever reason) you have the wrong modules loaded.

For example, on Aurora, some of the newer environments have a version of PyTorch which was built with a newer version of the Intel OneAPI software stack. This relies on a newer set of modules than those provided by e.g. the module load frameworks.

Therefore, if you try and python3 -c 'import torch' in this environment, without having loaded the correct set of (newer) modules, you will encounter something like:

code: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
#[🐍 2025-08-pt29]
#[~/d/f/p/s/ezpz][🌱 saforem2/yeet-env][📦📝🤷✓]
#[08/26/25 @ 07:44:31][x4204c4s2b0n0]
; ezpz-test
Traceback (most recent call last):
  File "/tmp/2025-08-pt29/bin/ezpz-test", line 4, in <module>
    from ezpz.test import main
  File "/lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/src/ezpz/__init__.py", line 19, in <module>
    import torch
  File "/lus/flare/projects/datascience/foremans/micromamba/envs/2025-08-pt29/lib/python3.11/site-packages/torch/__init__.py", line 407, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: /lus/flare/projects/datascience/foremans/micromamba/envs/2025-08-pt29/lib/python3.11/site-packages/torch/lib/libtorch-xpu-ops-sycl-ZetaKernel.so: undefined symbol: _ZN4sycl3_V17handler28extractArgsAndReqsFromLambdaEPcRKSt6vectorINS0_6detail19kernel_param_desc_tESaIS5_EEb
[1]    24429 exit 1     ezpz-test
took: 0h:00m:16s
This can be resolved by loading the correct set of modules. - On Aurora, for example, we can: 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
#[🐍 2025-08-pt29]
#[~/d/f/p/s/ezpz][🌱 saforem2/yeet-env][📦📝🤷✓] [⏱️ 8m43s]
#[08/26/25 @ 07:57:03][x4204c4s2b0n0]
; ezpz-test
[W826 07:57:09.062669769 OperatorEntry.cpp:218] Warning: Warning only once for all operators,  other operators may also be overridden.
  Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
    registered at /lus/flare/projects/datascience/foremans/projects/argonne-lcf/frameworks-standalone/builds/nightly/2025-07-24-095425/pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: XPU
  previous kernel: registered at /lus/flare/projects/datascience/foremans/projects/argonne-lcf/frameworks-standalone/builds/nightly/2025-07-24-095425/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:37
       new kernel: registered at /lus/flare/projects/datascience/foremans/projects/argonne-lcf/frameworks-standalone/builds/nightly/2025-07-24-095425/intel-extension-for-pytorch/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:172 (function operator())
[2025-08-26 07:57:12,902747][I][__init__/ezpz:265:__init__.ezpz] Setting logging level to 'INFO' on 'RANK == 0'
[2025-08-26 07:57:12,904622][I][__init__/ezpz:266:__init__.ezpz] Setting logging level to 'CRITICAL' on all others 'RANK != 0'


[2025-08-26 07:57:12,910880][I][launch/ezpz:356:launch] ----[🍋 ezpz.launch][started][2025-08-26-075712]----
[2025-08-26 07:57:16,122020][I][launch/ezpz:361:launch] Job ID: 7419283
[2025-08-26 07:57:16,122674][I][launch/ezpz:362:launch] nodelist: ['x4204c4s2b0n0', 'x4204c4s3b0n0']
[2025-08-26 07:57:16,122983][I][launch/ezpz:363:launch] hostfile: /var/spool/pbs/aux/7419283.aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov
[2025-08-26 07:57:16,124166][I][pbs/ezpz:228:pbs] ✅ Using [24/24] GPUs [2 hosts] x [12 GPU/host]
[2025-08-26 07:57:16,124950][I][launch/ezpz:332:launch] Building command to execute by piecing together:
[2025-08-26 07:57:16,125270][I][launch/ezpz:333:launch] (1.) : mpiexec --verbose --envall --np=24 --ppn=12 --hostfile=/var/spool/pbs/aux/7419283.aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov --no-vni --cpu-bind=verbose,list:2-4:10-12:18-20:26-28:34-36:42-44:54-56:62-64:70-72:78-80:86-88:94-96
[2025-08-26 07:57:16,125795][I][launch/ezpz:334:launch] (2.) cmd_to_launch: /lus/flare/projects/datascience/foremans/micromamba/envs/2025-08-pt29/bin/python3 -m ezpz.test_dist
[2025-08-26 07:57:16,126304][I][launch/ezpz:441:launch] Took: 3.22 seconds to build command.
[2025-08-26 07:57:16,126571][I][launch/ezpz:444:launch] Executing:
mpiexec
  --verbose
  --envall
  --np=24
  --ppn=12
  --hostfile=/var/spool/pbs/aux/7419283.aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov
  --no-vni
  --cpu-bind=verbose,list:2-4:10-12:18-20:26-28:34-36:42-44:54-56:62-64:70-72:78-80:86-88:94-96
  /lus/flare/projects/datascience/foremans/micromamba/envs/2025-08-pt29/bin/python3
  -m
  ezpz.test_dist
[2025-08-26 07:57:16,127609][I][launch/ezpz:176:launch] Filtering for Aurora-specific messages. To view list of filters, run with EZPZ_LOG_LEVEL=DEBUG
[2025-08-26 07:57:16,127984][I][launch/ezpz:460:launch] Execution started @ 2025-08-26-075716...
[2025-08-26 07:57:16,128310][I][launch/ezpz:463:launch] ----[🍋 ezpz.launch][stop][2025-08-26-075716]----
[2025-08-26 07:57:16,128648][I][launch/ezpz:99:launch] Caught 20 filters
[2025-08-26 07:57:16,128901][I][launch/ezpz:100:launch] Running command:
 mpiexec --verbose --envall --np=24 --ppn=12 --hostfile=/var/spool/pbs/aux/7419283.aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov --no-vni --cpu-bind=verbose,list:2-4:10-12:18-20:26-28:34-36:42-44:54-56:62-64:70-72:78-80:86-88:94-96 /lus/flare/projects/datascience/foremans/micromamba/envs/2025-08-pt29/bin/python3 -m ezpz.test_dist
Disabling local launch: multi-node application
Connected to tcp://x4204c4s2b0n0.hsn.cm.aurora.alcf.anl.gov:7919
Launching application 1eb3bdcb-f2c8-4bf9-b79e-25a490bda5c7
cpubind:list x4204c4s3b0n0 pid 203966 rank 12 0: mask 0x1c
cpubind:list x4204c4s3b0n0 pid 203967 rank 13 1: mask 0x1c00
cpubind:list x4204c4s3b0n0 pid 203968 rank 14 2: mask 0x1c0000
cpubind:list x4204c4s3b0n0 pid 203969 rank 15 3: mask 0x1c000000
cpubind:list x4204c4s3b0n0 pid 203970 rank 16 4: mask 0x1c00000000
cpubind:list x4204c4s3b0n0 pid 203971 rank 17 5: mask 0x1c0000000000
cpubind:list x4204c4s3b0n0 pid 203972 rank 18 6: mask 0x1c0000000000000
cpubind:list x4204c4s3b0n0 pid 203973 rank 19 7: mask 0x1c000000000000000
cpubind:list x4204c4s3b0n0 pid 203974 rank 20 8: mask 0x1c00000000000000000
cpubind:list x4204c4s3b0n0 pid 203975 rank 21 9: mask 0x1c0000000000000000000
cpubind:list x4204c4s3b0n0 pid 203976 rank 22 10: mask 0x1c000000000000000000000
cpubind:list x4204c4s3b0n0 pid 203977 rank 23 11: mask 0x1c00000000000000000000000
cpubind:list x4204c4s2b0n0 pid 27818 rank 0 0: mask 0x1c
cpubind:list x4204c4s2b0n0 pid 27819 rank 1 1: mask 0x1c00
cpubind:list x4204c4s2b0n0 pid 27820 rank 2 2: mask 0x1c0000
cpubind:list x4204c4s2b0n0 pid 27821 rank 3 3: mask 0x1c000000
cpubind:list x4204c4s2b0n0 pid 27822 rank 4 4: mask 0x1c00000000
cpubind:list x4204c4s2b0n0 pid 27823 rank 5 5: mask 0x1c0000000000
cpubind:list x4204c4s2b0n0 pid 27824 rank 6 6: mask 0x1c0000000000000
cpubind:list x4204c4s2b0n0 pid 27825 rank 7 7: mask 0x1c000000000000000
cpubind:list x4204c4s2b0n0 pid 27826 rank 8 8: mask 0x1c00000000000000000
cpubind:list x4204c4s2b0n0 pid 27827 rank 9 9: mask 0x1c0000000000000000000
cpubind:list x4204c4s2b0n0 pid 27828 rank 10 10: mask 0x1c000000000000000000000
cpubind:list x4204c4s2b0n0 pid 27829 rank 11 11: mask 0x1c00000000000000000000000
  operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
    registered at /lus/flare/projects/datascience/foremans/projects/argonne-lcf/frameworks-standalone/builds/nightly/2025-07-24-095425/pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
[2025-08-26 07:57:27,562570][I][__init__/ezpz:265:__init__.ezpz] Setting logging level to 'INFO' on 'RANK == 0'
[2025-08-26 07:57:27,564617][I][__init__/ezpz:266:__init__.ezpz] Setting logging level to 'CRITICAL' on all others 'RANK != 0'
[2025-08-26 07:57:27,572099][I][test_dist/ezpz:111:test_dist.__main__] Outputs will be saved to /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727
[2025-08-26 07:57:27,572677][I][dist/ezpz:1171:dist] Using fw='ddp' with torch_{device,backend}= {xpu, xccl}
[2025-08-26 07:57:27,573394][I][dist/ezpz:1035:dist] Caught MASTER_PORT=57707 from environment!
[2025-08-26 07:57:27,573850][I][dist/ezpz:1051:dist] Using torch.distributed.init_process_group with
- master_addr='x4204c4s2b0n0.hsn.cm.aurora.alcf.anl.gov'
- master_port='57707'
- world_size=24
- rank=0
- local_rank=0
- timeout=datetime.timedelta(seconds=3600)
- backend='xccl'
[2025-08-26 07:57:27,574703][I][dist/ezpz:768:dist] Calling torch.distributed.init_process_group_with: rank=0 world_size=24 backend=xccl
[2025-08-26 07:57:36,394953][I][pbs/ezpz:228:pbs] ✅ Using [24/24] GPUs [2 hosts] x [12 GPU/host]
2025:08:26-07:57:36:(27818) |CCL_WARN| value of CCL_LOG_LEVEL changed to be error (default:warn)
[2025-08-26 07:57:36,961298][I][dist/ezpz:1389:dist] Using device='xpu' with backend='xccl' + 'xccl' for distributed training.
[2025-08-26 07:57:36,962014][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 0/23]
[2025-08-26 07:57:36,960983][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][10/23]
[2025-08-26 07:57:36,961070][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 1/23]
[2025-08-26 07:57:36,961062][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][13/23]
[2025-08-26 07:57:36,961117][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][14/23]
[2025-08-26 07:57:36,961042][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][16/23]
[2025-08-26 07:57:36,961237][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 3/23]
[2025-08-26 07:57:36,961100][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][17/23]
[2025-08-26 07:57:36,961070][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 5/23]
[2025-08-26 07:57:36,961030][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 6/23]
[2025-08-26 07:57:36,961081][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 7/23]
[2025-08-26 07:57:36,960983][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 8/23]
[2025-08-26 07:57:36,961062][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 9/23]
[2025-08-26 07:57:36,961014][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][11/23]
[2025-08-26 07:57:36,961069][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 2/23]
[2025-08-26 07:57:36,961063][I][dist/ezpz:1434:dist] ['x4204c4s2b0n0'][ 4/23]
[2025-08-26 07:57:36,961094][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][18/23]
[2025-08-26 07:57:36,961052][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][19/23]
[2025-08-26 07:57:36,961057][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][21/23]
[2025-08-26 07:57:36,961057][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][22/23]
[2025-08-26 07:57:36,961097][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][23/23]
[2025-08-26 07:57:36,961057][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][12/23]
[2025-08-26 07:57:36,961125][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][15/23]
[2025-08-26 07:57:36,961153][I][dist/ezpz:1434:dist] ['x4204c4s3b0n0'][20/23]
[2025-08-26 07:57:37,049349][I][test_dist/ezpz:639:test_dist.__main__] Took: 9.48 seconds to setup torch
[2025-08-26 07:57:38,429479][I][test_dist/ezpz:274:test_dist.__main__] Model size: 352364544 parameters
[2025-08-26 07:57:38,431186][I][test_dist/ezpz:278:test_dist.__main__]
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
SequentialLinearNet                      --
├─Sequential: 1-1                        352,364,544
=================================================================
Total params: 352,364,544
Trainable params: 352,364,544
Non-trainable params: 0
=================================================================
[2025-08-26 07:57:38,432150][I][test_dist/ezpz:286:test_dist.__main__] Took: 1.37825637194328 seconds to build model
[2025-08-26 07:57:38,470760][I][test_dist/ezpz:574:test_dist.__main__] model=
SequentialLinearNet(
  (layers): Sequential(
    (0): Linear(in_features=2048, out_features=4096, bias=True)
    (1): ReLU()
    (2): Linear(in_features=4096, out_features=8192, bias=True)
    (3): ReLU()
    (4): Linear(in_features=8192, out_features=16384, bias=True)
    (5): ReLU()
    (6): Linear(in_features=16384, out_features=8192, bias=True)
    (7): ReLU()
    (8): Linear(in_features=8192, out_features=4096, bias=True)
    (9): ReLU()
    (10): Linear(in_features=4096, out_features=2048, bias=True)
  )
)
[2025-08-26 07:57:52,714675][I][test_dist/ezpz:290:test_dist.__main__] Took: 14.28 seconds to build optimizer
[2025-08-26 07:57:52,747214][I][pbs/ezpz:228:pbs] ✅ Using [24/24] GPUs [2 hosts] x [12 GPU/host]
[2025-08-26 07:57:52,753235][I][dist/ezpz:1664:dist] Setting up wandb from rank=0
[2025-08-26 07:57:52,753677][I][dist/ezpz:1665:dist] Using WB_PROJECT=ezpz.test_dist
wandb: Currently logged in as: foremans (aurora_gpt) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.21.1
wandb: Run data is saved locally in /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/wandb/run-20250826_075752-5wgkqmrd
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run revived-snow-5851
wandb:  View project at https://wandb.ai/aurora_gpt/ezpz.test_dist
wandb:  View run at https://wandb.ai/aurora_gpt/ezpz.test_dist/runs/5wgkqmrd
[2025-08-26 07:57:53,974792][I][dist/ezpz:1694:dist] wandb.run=[revived-snow-5851](https://wandb.ai/aurora_gpt/ezpz.test_dist/runs/5wgkqmrd)
[2025-08-26 07:57:53,977590][I][pbs/ezpz:228:pbs] ✅ Using [24/24] GPUs [2 hosts] x [12 GPU/host]
[2025-08-26 07:57:53,981659][I][dist/ezpz:1738:dist] Running on machine='Aurora'
[2025-08-26 07:57:53,998002][I][test_dist/ezpz:293:test_dist.__main__] Took: 1.28 seconds to build trainer
[2025-08-26 07:57:53,998843][I][test_dist/ezpz:297:test_dist.__main__] config:
{
  "acc_events": false,
  "backend": "DDP",
  "batch_size": 64,
  "cp": 1,
  "dtype": "bfloat16",
  "input_size": 2048,
  "layer_sizes": [
    4096,
    8192,
    16384,
    8192,
    4096
  ],
  "log_freq": 1,
  "output_size": 2048,
  "pp": 1,
  "print_freq": 100,
  "profile_memory": true,
  "pyinstrument_profiler": false,
  "pytorch_profiler": false,
  "pytorch_profiler_active": 3,
  "pytorch_profiler_repeat": 5,
  "pytorch_profiler_wait": 1,
  "pytorch_profiler_warmup": 2,
  "rank_zero_only": false,
  "record_shapes": true,
  "tp": 1,
  "train_iters": 1000,
  "warmup": 50,
  "with_flops": true,
  "with_modules": true,
  "with_stack": true
}
[2025-08-26 07:57:54,000396][I][test_dist/ezpz:299:test_dist.__main__] Took: 26.43 to get here.
[2025-08-26 07:57:55,508021][I][test_dist/ezpz:247:test_dist.__main__] Warmup complete at step 50
[2025-08-26 07:57:56,477134][I][test_dist/ezpz:218:test_dist.__main__] iter=100 loss=11008.000000 dtf=0.000675 dtb=0.001717
[2025-08-26 07:57:58,426199][I][test_dist/ezpz:218:test_dist.__main__] iter=200 loss=10944.000000 dtf=0.000665 dtb=0.001735
[2025-08-26 07:58:00,363985][I][test_dist/ezpz:218:test_dist.__main__] iter=300 loss=10944.000000 dtf=0.000684 dtb=0.001741
[2025-08-26 07:58:02,310994][I][test_dist/ezpz:218:test_dist.__main__] iter=400 loss=10944.000000 dtf=0.000674 dtb=0.001729
[2025-08-26 07:58:04,253050][I][test_dist/ezpz:218:test_dist.__main__] iter=500 loss=10944.000000 dtf=0.000669 dtb=0.001730
[2025-08-26 07:58:06,195840][I][test_dist/ezpz:218:test_dist.__main__] iter=600 loss=10944.000000 dtf=0.000672 dtb=0.001740
[2025-08-26 07:58:08,139168][I][test_dist/ezpz:218:test_dist.__main__] iter=700 loss=10944.000000 dtf=0.000664 dtb=0.001736
[2025-08-26 07:58:10,084974][I][test_dist/ezpz:218:test_dist.__main__] iter=800 loss=10944.000000 dtf=0.000677 dtb=0.001750
[2025-08-26 07:58:12,005505][I][test_dist/ezpz:218:test_dist.__main__] iter=900 loss=10944.000000 dtf=0.000667 dtb=0.001736
[2025-08-26 07:58:20,023565][I][history/ezpz:824:history] Saving iter plot to: /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/mplot
[2025-08-26 07:58:20,211523][I][history/ezpz:824:history] Saving loss plot to: /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/mplot
[2025-08-26 07:58:20,372340][I][history/ezpz:824:history] Saving dtf plot to: /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/mplot
[2025-08-26 07:58:20,532649][I][history/ezpz:824:history] Saving dtb plot to: /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/mplot
[2025-08-26 07:58:20,685178][I][history/ezpz:720:history] Saving tplots to /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/tplot
                    loss [2025-08-26-075820]
     ┌─────────────────────────────────────────────────────┐
11072┤    ▌    ▌ ▐  ▐ ▐  ▐                                 │
     │    ▌    ▌ ▐  ▐ ▐  ▐                                 │
11040┤    ▌    ▌ ▐  ▐ ▐  ▐                                 │
     │    ▌    ▌ ▐  ▐ ▐  ▐                                 │
     │    ▌    ▌ ▐  ▐ ▐  ▐                                 │
11008┤████████████████████████████████▌▐███████████▐▐██████│
     │████████████████████████████████▌▐███████████▐▐██████│
10976┤████████████████████████████████▌▐███████████▐▐██████│
     │████████████████████████████████▌▐███████████▐▐██████│
10944┤████████████████████████████████▙▟███████████▟▟██████│
     │▐▌        ▌ ▌         ▌ ▌   ▐▐ ▌ ▌▐▌▐▐█▌▐█▌▌▐█▌ ▐ █  │
     │▐▌        ▌ ▌         ▌ ▌   ▐▐ ▌ ▌▐▌▐▐█▌▐█▌▌▐█▌ ▐ █  │
10912┤▐▌        ▌ ▌         ▌ ▌   ▐▐ ▌ ▌▐▌▐▐█▌▐█▌▌▐█▌ ▐ █  │
     │▐▌        ▌ ▌         ▌ ▌   ▐▐ ▌ ▌▐▌▐▐█▌▐█▌▌▐█▌ ▐ █  │
10880┤▝▌        ▌ ▌         ▌ ▌   ▐▐ ▌ ▌▐▌▐▝▛▌▐▛▌▌▐█▌ ▐ █  │
     └──┬──┬────┬─────┬───┬─────┬───┬────┬─────┬──────┬────┘
       44 84   185   296 366   471 551  647   750    867
loss                          iter
text saved in /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/tplot/loss.txt
                       dtf [2025-08-26-075820]
        ┌──────────────────────────────────────────────────┐
0.001042┤                                                 ▌│
        │                                                 ▌│
0.000974┤                                                 ▌│
        │                                                 ▌│
        │                                                 ▌│
0.000905┤   ▌                          ▐                  ▌│
        │  ▐▌        ▗                 ▐                  ▌│
0.000837┤  ▐▌   ▗    ▐▖         ▖    ▗ ▐  ▗               ▌│
        │  ▐▌   ▐▖   ▐▌    ▖    ▌    ▐ ▐  ▐▖    ▖    ▌    ▙│
0.000768┤  ▐▌   ▐▌   ▐▌    █    █▗   ▐ ▐  ▐▌   ▐▌▖   █    █│
        │ ▗▐█  ▐▐▌▄ ▗▐▌▌  ▌█▐  ▖█▐  ▗▐▗█ ▄▐▌▌ ▙▐▌▌  ▖█▐   █│
        │▖▐▐█  ▐▐▌█ ▐▐▌▌  ▌█▟ ▖▌█▐  █▐▐█ █▐▌▌ █▐▌▌  ▌█▐ ▗▙▌│
0.000699┤▌▟▐█▄ ▐▐██▗▐▟▌▙▖▄▙██▖▌▌█▟▖▖█▐▟█▄█▟▙▙▖█▐▌▙▖▄▙█▟▗▐█▌│
        │███████████▜▛███████████████████▛▛▛▛█████████████ │
0.000631┤   ▘                                   ▐          │
        └─┬──┬─────┬────┬───┬────┬───┬────┬─────┬─────┬────┘
         25 84    185  296 366  471 551  647   750   867
dtf                             iter
text saved in /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/tplot/dtf.txt
                     dtf [2025-08-26-075820]
     ┌─────────────────────────────────────────────────────┐
567.0┤█████                                                │
     │█████                                                │
472.5┤█████                                                │
     │█████                                                │
     │█████                                                │
378.0┤█████                                                │
     │███████████                                          │
283.5┤███████████                                          │
     │███████████                                          │
189.0┤███████████                                          │
     │███████████                                          │
     │███████████                                          │
 94.5┤███████████                                          │
     │████████████████                                     │
  0.0┤██████████████████████████ ██████████           █████│
     └┬────────────┬────────────┬────────────┬────────────┬┘
   0.00061      0.00072      0.00084      0.00095   0.00106
freq                           dtf
text saved in /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/tplot/dtf-hist.txt
                      dtb [2025-08-26-075820]
       ┌───────────────────────────────────────────────────┐
0.00265┤                                                  ▟│
       │                                                  █│
0.00242┤                                                  █│
       │                                                  █│
       │   ▖                                              █│
0.00219┤   ▌                           ▐                  █│
       │   ▌                           ▐                  █│
0.00196┤   ▌▗                          ▐                  █│
       │ ▖ ▌▐▗ ▖ ▗▗▖▗   ▟▗▖ ▖ ▄▖ ▐▙ ▗  ▐▄▗ ▖ ▗ ▐ ▗▗    ▄▗▐█│
0.00173┤█▙▙███▙▙▄▟█▙▟▄▄▟██▙▙▙▙██▟██▙█▟▟██▟▄▙▟██▟▄▟███▄▙██▟█│
       │████████████▌██▌█▌█████▛███████▜██████████▜████████│
       │▛████▜██████▌██▌█▌▐▐███▘██▜███ ▐████▜█▌███▐███████▌│
0.00150┤▌▛█▜█▐█▘██▜█▌▜▛▘▛▌▐▝▀▀▛ █▛▐▛▛█ ▐█▛█▝▐█▌▐█▀▝█▛█▛▜▜▛▘│
       │▌ ▐          ▝           ▘   ▝  ▘   ▐  ▐▌    ▐  ▝  │
0.00128┤▌ ▐                                     ▌    ▝     │
       └─┬──┬─────┬─────┬─────┬──────┬────┬─────┬─────┬────┘
        25 84    185   296   420    551  647   750   867
dtb                            iter
text saved in /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/tplot/dtb.txt
                    dtb [2025-08-26-075820]
   ┌───────────────────────────────────────────────────────┐
714┤                 █████                                 │
   │                 █████                                 │
595┤                 █████                                 │
   │                 █████                                 │
   │                 █████                                 │
476┤                 █████                                 │
   │                 █████                                 │
357┤                 █████                                 │
   │                 █████                                 │
238┤                 █████                                 │
   │                 █████                                 │
   │                 █████                                 │
119┤      █████      █████                                 │
   │      ██████████ ██████████                            │
  0┤█████ ██████████ ██████████      █████            █████│
   └┬─────────────┬────────────┬─────────────┬────────────┬┘
  0.00121      0.00159      0.00196       0.00234   0.00271
freq                          dtb
text saved in /lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/outputs/ezpz.test_dist/2025-08-26-075727/ezpz.test_dist/plots/tplot/dtb-hist.txt
[2025-08-26 07:58:20,765711][I][test_dist/ezpz:238:test_dist.__main__] dataset=<xarray.Dataset> Size: 34kB
Dimensions:  (draw: 949)
Coordinates:
  * draw     (draw) int64 8kB 0 1 2 3 4 5 6 7 ... 942 943 944 945 946 947 948
Data variables:
    iter     (draw) int64 8kB 51 52 53 54 55 56 57 ... 994 995 996 997 998 999
    loss     (draw) float32 4kB 1.094e+04 1.101e+04 ... 1.101e+04 1.094e+04
    dtf      (draw) float64 8kB 0.0007204 0.0006706 ... 0.000786 0.0007987
    dtb      (draw) float64 8kB 0.0018 0.001277 0.001722 ... 0.00186 0.001675
[2025-08-26 07:58:20,767159][I][test_dist/ezpz:311:test_dist.__main__] Took: 26.77 seconds to finish training
[2025-08-26 07:58:20,767684][I][test_dist/ezpz:655:test_dist.__main__] Took: 53.20 seconds
wandb:
wandb: 🚀 View run revived-snow-5851 at: https://wandb.ai/aurora_gpt/ezpz.test_dist/runs/5wgkqmrd
wandb: Find logs at: ../../../../../../../lus/flare/projects/datascience/foremans/projects/saforem2/ezpz/wandb/run-20250826_075752-5wgkqmrd/logs
Application 1eb3bdcb resources: utime=1292s stime=200s maxrss=8414632KB inblock=0 oublock=2200 minflt=11887747 majflt=20605 nvcsw=494913 nivcsw=7778
[2025-08-26 07:58:23,184273][I][launch/ezpz:467:launch] Execution finished with 0.
[2025-08-26 07:58:23,185063][I][launch/ezpz:468:launch] Executing finished in 67.06 seconds.
[2025-08-26 07:58:23,185548][I][launch/ezpz:469:launch] Took 67.06 seconds to run. Exiting.
took: 0h:01m:18s