forked from SchedMD/slurm
-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathNEWS
15335 lines (14988 loc) · 871 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.
* Changes in Slurm 24.05.6
==========================
-- data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when
dumping data with v0.0.40+complex data parser.
-- Fix sattach when using auth/slurm.
-- scrun - Add support '--all' argument for kill subcommand.
-- Fix performance regression while packing larger RPCs.
-- Fix crash and issues evaluating job's suitability for running in
nodes with already suspended job(s) there.
-- Fixed a job requeuing issue that merged job entries into the
same SLUID when all nodes in a job failed simultaneously.
-- switch/hpe_slingshot - Fix compatibility with newer cxi
drivers, specifically when specifying disable_rdzv_get.
-- Add ABORT_ON_FATAL environment variable to capture a backtrace
from any fatal() message.
* Changes in Slurm 24.05.5
==========================
-- Fix issue signaling cron jobs resulting in unintended requeues.
-- Fix slurmctld memory leak in implementation of HealthCheckNodeState=CYCLE.
-- job_container/tmpfs - Fix SLURM_CONF env variable not being properly set.
-- sched/backfill - Fix job's time_limit being overwritten by time_min for job
arrays in some situations.
-- RoutePart - fix segfault from incorrect memory allocation when node doesn't
exist in any partition.
-- slurmctld - Fix crash when a job is evaluated for a reservation after
removal of a dynamic node.
-- gpu/nvml - Attempt loading libnvidia-ml.so.1 as a fallback for failure in
loading libnvidia-ml.so.
-- slurmrestd - Fix populating non-required object fields of objects as '{}' in
JSON/YAML instead of 'null' causing compiled OpenAPI clients to reject
the response to 'GET /slurm/v0.0.40/jobs' due to validation failure of
'.jobs[].job_resources'.
-- Fix sstat/sattach protocol errors for steps on higher version slurmd's
(regressions since 20.11.0rc1 and 16.05.1rc1 respectively).
-- slurmd - Avoid a crash when starting slurmd version 24.05 with
SlurmdSpoolDir files that have been upgraded to a newer major version of
Slurm. Log warnings instead.
-- Fix race condition in stepmgr step completion handling.
-- Fix slurmctld segfault with stepmgr and MpiParams when running a job array.
-- Fix requeued jobs keeping their priority until the decay thread happens.
-- slurmctld - Fix crash and possible split brain issue if the
backup controller handles an scontrol reconfigure while in control
before the primary resumes operation.
-- Fix stepmgr not getting dynamic node addrs from the controller
-- stepmgr - avoid "Unexpected missing socket" errors.
-- Fix `scontrol show steps` with dynamic stepmgr
-- Support IPv6 in configless mode.
* Changes in Slurm 24.05.4
==========================
-- Fix generic int sort functions.
-- Fix user look up using possible unrealized uid in the dbd.
-- Fix FreeBSD compile issue with tls/none plugin.
-- slurmrestd - Fix regressions that allowed slurmrestd to be run as SlurmUser
when SlurmUser was not root.
-- mpi/pmix fix race conditions with het jobs at step start/end which could
make srun to hang.
-- Fix not showing some SelectTypeParameters in scontrol show config.
-- Avoid assert when dumping removed certain fields in JSON/YAML.
-- Improve how shards are scheduled with affinity in mind.
-- Fix MaxJobsAccruePU not being respected when MaxJobsAccruePA is set
in the same QOS.
-- Prevent backfill from planning jobs that use overlapping resources for the
same time slot if the job's time limit is less than bf_resolution.
-- Fix memory leak when requesting typed gres and --[cpus|mem]-per-gpu.
-- Prevent backfill from breaking out due to "system state changed" every 30
seconds if reservations use REPLACE or REPLACE_DOWN flags.
-- slurmrestd - Make sure that scheduler_unset parameter defaults to true even
when the following flags are also set: show_duplicates, skip_steps,
disable_truncate_usage_time, run_away_jobs, whole_hetjob,
disable_whole_hetjob, disable_wait_for_result, usage_time_as_submit_time,
show_batch_script, and or show_job_environment. Additionaly, always make
sure show_duplicates and disable_truncate_usage_time default to true when
the following flags are also set: scheduler_unset, scheduled_on_submit,
scheduled_by_main, scheduled_by_backfill, and or job_started. This effects
the following endpoints:
'GET /slurmdb/v0.0.40/jobs'
'GET /slurmdb/v0.0.41/jobs'
-- Ignore --json and --yaml options for scontrol show config to prevent mixing
output types.
-- Fix not considering nodes in reservations with Maintenance or Overlap flags
when creating new reservations with nodecnt or when they replace down nodes.
-- Fix suspending/resuming steps running under a 23.02 slurmstepd process.
-- Fix options like sprio --me and squeue --me for users with a uid greater
than 2147483647.
-- fatal() if BlockSizes=0. This value is invalid and would otherwise cause the
slurmctld to crash.
-- sacctmgr - Fix issue where clearing out a preemption list using
preempt='' would cause the given qos to no longer be preempt-able until set
again.
-- Fix stepmgr creating job steps concurrently.
-- data_parser/v0.0.40 - Avoid dumping "Infinity" for NO_VAL tagged "number"
fields.
-- data_parser/v0.0.41 - Avoid dumping "Infinity" for NO_VAL tagged "number"
fields.
-- slurmctld - Fix a potential leak while updating a reservation.
-- slurmctld - Fix state save with reservation flags when a update fails.
-- Fix reservation update issues with parameters Accounts and Users, when
using +/- signs.
-- slurmrestd - Don't dump warning on empty wckeys in:
'GET /slurmdb/v0.0.40/config'
'GET /slurmdb/v0.0.41/config'
-- Fix slurmd possibly leaving zombie processes on start up in configless when
the initial attempt to fetch the config fails.
-- Fix crash when trying to drain a non-existing node (possibly deleted
before).
-- slurmctld - fix segfault when calculating limit decay for jobs with an
invalid association.
-- Fix IPMI energy gathering with multiple sensors.
-- data_parser/v0.0.39 - Remove xassert requiring errors and warnings to have a
source string.
-- slurmrestd - Prevent potential segfault when there is an error parsing an
array field which could lead to a double xfree. This applies to several
endpoints in data_parser v0.0.39, v0.0.40 and v0.0.41.
-- scancel - Fix a regression from 23.11.6 where using both the --ctld and
--sibling options would cancel the federated job on all clusters instead of
only the cluster(s) specified by --sibling.
-- accounting_storage/mysql - Fix bug when removing an association
specified with an empty partition.
-- Fix setting multiple partition state restore on a job correctly.
-- Fix difference in behavior when swapping partition order in job submission.
-- Fix security issue in stepmgr that could permit an attacker to execute
processes under other users' jobs. CVE-2024-48936.
* Changes in Slurm 24.05.3
==========================
-- data_parser/v0.0.40 - Added field descriptions
-- slurmrestd - Avoid creating new slurmdbd connection per request to
'* /slurm/slurmctld/*/*' endpoints.
-- Fix compilation issue with switch/hpe_slingshot plugin.
-- Fix gres per task allocation with threads-per-core.
-- data_parser/v0.0.41 - Added field descriptions
-- slurmrestd - Change back generated OpenAPI schema for
`DELETE /slurm/v0.0.40/jobs/` to RequestBody instead of using parameters
for request. slurmrestd will continue accept endpoint requests via
RequestBody or HTTP query.
-- topology/tree - Fix issues with switch distance optimization.
-- Fix potential segfault of secondary slurmctld when falling back to the
primary when running with a JobComp plugin.
-- Enable --json/--yaml=v0.0.39 options on client commands to dump data using
data_parser/v0.0.39 instead or outputting nothing.
-- switch/hpe_slingshot - Fix issue that could result in a 0 length state file.
-- Fix unnecessary message protocol downgrade for unregistered nodes.
-- Fix unnecessarily packing alias addrs when terminating jobs with a mix of
non-cloud/dynamic nodes and powered down cloud/dynamic nodes.
-- accounting_storage/mysql - Fix issue when deleting a qos that could remove
too many commas from the qos and/or delta_qos fields of the assoc table.
-- slurmctld - Fix memory leak when using RestrictedCoresPerGPU.
-- Fix allowing access to reservations without MaxStartDelay set.
-- Fix regression introduced in 24.05.0rc1 breaking srun --send-libs parsing.
-- Fix slurmd vsize memory leak when using job submission/allocation commands
that implicitly or explicitly use --get-user-env.
-- slurmd - Fix node going into invalid state when using CPUSpecList and
setting CPUs to the # of cores on a multithreaded node
-- Fix reboot asap nodes being considered in backfill after a restart.
-- Fix --clusters/-M queries for clusters outside of a federation when
fed_display is configured.
-- Fix scontrol allowing updating job with bad cpus-per-task value.
-- sattach - Fix regression from 24.05.2 security fix leading to crash.
-- mpi/pmix - Fix assertion when built under --enable-debug.
* Changes in Slurm 24.05.2
==========================
-- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when
more than 10 threads try to get energy at the same time. This prevented
the possibility to get energy from slurmd by any step until slurmd was
restarted, so losing energy accounting metrics in the node.
-- accounting_storage/mysql - Fix issue where new user with wckey did not
have a default wckey sent to the slurmctld.
-- slurmrestd - Prevent slurmrestd segfault when handling the following
endpoints when none of the optional parameters are specified:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
'GET /slurm/v0.0.40/shares'
'GET /slurm/v0.0.41/shares'
'GET /slurmdb/v0.0.40/instance'
'GET /slurmdb/v0.0.41/instance'
'GET /slurmdb/v0.0.40/instances'
'GET /slurmdb/v0.0.41/instances'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.41/job/{job_id}'
-- Fix IPMI energy gathering when no IPMIPowerSensors are specified in
acct_gather.conf. This situation resulted in an accounted energy of 0
for job steps.
-- Fix a minor memory leak in slurmctld when updating a job dependency.
-- scontrol,squeue - Fix regression that caused incorrect values for
multisocket nodes at '.jobs[].job_resources.nodes.allocation' for
'scontrol show jobs --(json|yaml)' and 'squeue --(json|yaml)'.
-- slurmrestd - Fix regression that caused incorrect values for
multisocket nodes at '.jobs[].job_resources.nodes.allocation' to be dumped
with endpoints:
'GET /slurm/v0.0.41/job/{job_id}'
'GET /slurm/v0.0.41/jobs'
-- jobcomp/filetxt - Fix truncation of job record lines > 1024 characters.
-- Fixed regression that prevented compilation on FreeBSD hosts.
-- switch/hpe_slingshot - Drain node on failure to delete CXI services.
-- Fix a performance regression from 23.11.0 in cpu frequency handling when no
CpuFreqDef is defined.
-- Fix one-task-per-sharing not working across multiple nodes.
-- Fix inconsistent number of cpus when creating a reservation using the
TRESPerNode option.
-- data_parser/v0.0.40+ - Fix job state parsing which could break filtering.
-- Prevent cpus-per-task to be modified in jobs where a -c value has been
explicitly specified and the requested memory constraints implicitly
increase the number of CPUs to allocate.
-- slurmrestd - Fix regression where args '-s v0.0.39,dbv0.0.39' and
'-d v0.0.39' would result in 'GET /openapi/v3' not registering as a valid
possible query resulting in 404 errors.
-- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the
query parameters specified account, association, cluster, constraints,
format, groups, job_name, partition, qos, reason, reservation, state, users,
or wckey. This affects the following endpoints:
'GET /slurmdb/v0.0.39/jobs'
-- slurmrestd - In the case the slurmdbd does not respond to a persistent
connection init message, prevent the closed fd from being used, and instead
emit an error or warning depending on if the connection was required.
-- Fix 24.05.0 regression that caused the slurmdbd not to send back an error
message if there is an error initializing a persistent connection.
-- Reduce latency of forwarded x11 packets.
-- Add "curr_dependency" (representing the current dependency of the job)
and "orig_dependency" (representing the original requested dependency of
the job) fields to the job record in job_submit.lua (for job update) and
jobcomp.lua.
-- Fix potential segfault of slurmctld configured with
SlurmctldParameters=enable_rpc_queue from happening on reconfigure.
-- Fix potential segfault of slurmctld on its shutdown when rate limitting
is enabled.
-- slurmrestd - Fix missing job environment for SLURM_JOB_NAME,
SLURM_OPEN_MODE, SLURM_JOB_DEPENDENCY, SLURM_PROFILE, SLURM_ACCTG_FREQ,
SLURM_NETWORK and SLURM_CPU_FREQ_REQ to match sbatch.
-- Add missing bash-completions dependency to slurm-smd-client debian package.
-- Fix bash-completions installation in debian pacakges.
-- Fix GRES environment variable indices being incorrect when only using a
subset of all GPUs on a node and the --gres-flags=allow-task-sharing option
-- Add missing mariadb/mysql client package dependency to debian package.
-- Fail the debian package build early if mysql cannot be found.
-- Prevent scontrol from segfaulting when requesting scontrol show reservation
--json or --yaml if there is an error retrieving reservations from the
slurmctld.
-- switch/hpe_slingshot - Fix security issue around managing VNI access.
CVE-2024-42511.
-- switch/nvidia_imex - Fix security issue managing IMEX channel access.
CVE-2024-42511.
-- switch/nvidia_imex - Allow for compatibility with job_container/tmpfs.
* Changes in Slurm 24.05.1
==========================
-- Fix slurmctld and slurmdbd potentially stopping instead of performing a
logrotate when recieving SIGUSR2 when using auth/slurm.
-- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02.
-- Fix "Could not find group" errors from validate_group() when using
AllowGroups with large /etc/group files.
-- Prevent an assertion in debugging builds when triggering log rotation
in a backup slurmctld.
-- Add AccountingStoreFlags=no_stdio which allows to not record the stdio
paths of the job when set.
-- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field,
which was never usable. Now it explicitly ignores the value and emits a
warning if it is used for the following endpoints:
'POST /slurm/v0.0.39/job/{job_id}'
'POST /slurm/v0.0.39/job/submit'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.40/job/submit'
'POST /slurm/v0.0.41/job/{job_id}'
'POST /slurm/v0.0.41/job/submit'
'POST /slurm/v0.0.41/job/allocate'
-- mpi/pmi2 - Fix communication issue leading to task launch failure with
"invalid kvs seq from node".
-- Fix getting user environment when using sbatch with "--get-user-env" or
"--export=" when there is a user profile script that reads /proc.
-- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but
GresTypes is not configured.
-- Do not log the following errors when AcctGatherEnergyType plugins are used
but a node does not have or cannot find sensors:
"error: _get_joules_task: can't get info from slurmd"
"error: slurm_get_node_energy: Zero Bytes were transmitted or received"
However, the following error will continue to be logged:
"error: Can't get energy data. No power sensors are available. Try later"
-- sbatch, srun - Set SLURM_NETWORK environment variable if --network is set.
-- Fix cloud nodes not being able to forward to nodes that restarted with new
IP addresses.
-- Fix cwd not being set correctly when running a SPANK plugin with a
spank_user_init() hook and the new "contain_spank" option set.
-- slurmctld - Avoid deadlock during shutdown when auth/slurm is active.
-- Fix segfault in slurmctld with topology/block.
-- sacct - Fix printing of job group for job steps.
-- scrun - Log when an invalid environment variable causes the job submission
to be rejected.
-- accounting_storage/mysql - Fix problem where listing or modifying an
association when specifying a qos list could hang or take a very long time.
-- gpu/nvml - Fix gpuutil/gpumem only tracking last GPU in step. Now,
gpuutil/gpumem will record sums of all GPUS in the step.
-- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1.
-- Fix slurmctld crash on a batch job submission with "--nodes 0,...".
-- Fix dynamic IP address fanout forwarding when using auth/slurm.
-- Restrict listening sockets in the mpi/pmix plugin and sattach to the
SrunPortRange.
-- slurmrestd - Limit mime types returned from query to 'GET /openapi/v3' to
only return one mime type per serializer plugin to fix issues with OpenAPI
client generators that are unable to handle multiple mime type aliases.
-- Fix many commands possibly reporting an "Unexpected Message Received" when
in reality the connection timed out.
-- Prevent slurmctld from starting if there is not a json serializer present
and the extra_constraints feature is enabled.
-- Fix heterogeneous job components not being signaled with scancel --ctld and
'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given,
the heterogeneous job components match the given filters, and the
heterogeneous job leader does not match the given filters.
-- Fix regression from 23.02 impeding job licenses from being cleared.
-- Move error to log_flag which made _get_joules_task error to be logged to the
user when too many rpcs were queued in slurmd for gathering energy.
-- For scancel --ctld and the associated rest api endpoints:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
Fix canceling the final array task in a job array when the task is pending
and all array tasks have been split into separate job records. Previously
this task was not canceled.
-- Fix power_save operation after recovering from a failed reconfigure.
-- slurmctld - Skip removing the pidfile when running under systemd. In that
situation it is never created in the first place.
-- Fix issue where altering the flags on a Slurm account (UsersAreCoords)
several limits on the account's association would be set to 0 in
Slurm's internal cache.
-- Fix memory leak in the controller when relaying stepmgr step accounting to
the dbd.
-- Fix segfault when submitting stepmgr jobs within an existing allocation.
-- Added "disable_slurm_hydra_bootstrap" as a possible MpiParams parameter in
slurm.conf. Using this will disable env variable injection to allocations
for the following variables: I_MPI_HYDRA_BOOTSTRAP,
I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS, HYDRA_BOOTSTRAP,
HYDRA_LAUNCHER_EXTRA_ARGS.
-- scrun - Delay shutdown until after start requested. This caused scrun
to never start or shutdown and hung forever when using --tty.
-- Fix backup slurmctld potentially not running the agent when taking over as
the primary controller.
-- Fix primary controller not running the agent when a reconfigure of the
slurmctld fails.
-- slurmd - fix premature timeout waiting for REQUEST_LAUNCH_PROLOG with large
array jobs causing node to drain.
-- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time.
-- jobcomp/elasticsearch - Fix slurmctld memory leak from curl usage
-- acct_gather_profile/influxdb - Fix slurmstepd memory leak from curl usage
-- Fix 24.05.0 regression not deleting job hash dirs after MinJobAge.
-- Fix filtering arguments being ignored when using squeue --json.
-- switch/nvidia_imex - Move setup call after spank_init() to allow namespace
manipulation within the SPANK plugin.
-- switch/nvidia_imex - Skip plugin operation if nvidia-caps-imex-channels
device is not present rather than preventing slurmd from starting.
-- switch/nvidia_imex - Skip plugin operation if job_container/tmpfs
is configured due to incompatibility.
-- switch/nvidia_imex - Remove any pre-existing channels when slurmd starts.
-- rpc_queue - Add support for an optional rpc_queue.yaml configuration file.
-- slurmrestd - Add new +prefer_refs flag to data_parser/v0.0.41 plugin. This
flag will avoid inlining single referenced schemas in the OpenAPI schema.
* Changes in Slurm 24.05.0
==========================
-- Fix regresssion in rc1 causing power_save thread to spin continuously.
-- Improve ctld_relay shutdown sequence.
-- Fixed 'make distclean' behavior for contribs/perlapi.
-- slurmrestd - Avoid ignoring numerical only endpoints during startup with
older libjson-c due to type parsing mismatching.
-- Reject non-stepmgr job allocations requesting --resv-ports from the ctld.
-- slurmrestd - Add fields '.job.resv_ports' '.jobs[].resv_ports' to
'POST /slurm/v0.0.41/job/submit' and 'POST /slurm/v0.0.41/job/allocate'.
-- slurmstepd - Fix crash when cleaning up on shutdown with
--enable-memory-leak-debug.
-- Fix segfault in switch/hpe_slingshot plugin due to initialization squence.
-- scrun - Fix regression in rc1 that caused scrun to crash.
-- Prevent unnecessary log statement when free'ing ports.
-- Fix regression in rc1 causing communication problems when sending large
responses from slurmctld.
-- sreport - fix parsing of 'format=Planned' to prevent it from being
misinterpreted as 'PlannedDown'. 'PlannedDown' is now also known as
'PLNDDown' to match what is printed as the column title.
-- topology/block - Always return an error when the segment size does not
match the system or job specification.
-- Add previously missing timers for Prolog and Epilog scripts when RunInJob is
set.
-- Show an error when PrologFlags RunInJob and Serial are used together.
PrologFlags=Serial is not compatible with how RunInJob operates.
-- Fix memory leak on shutdown when using --enable-memory-leak-debug and
freeing cons_tres node usage.
-- Rename src/stepmgr/gres_ctld.[ch] to src/stepmgr/gres_stepmgr.[ch].
-- Fix various cosmetic issues with states in sinfo.
-- slurmrestd - Avoid crash due to associations query.
-- Calculate a job's min_cpus with consideration to --cpus-per-gpu.
-- Fix scancel request when specifying individual array tasks in combination
with filtering options (in both regular and --interactive mode).
-- Enable MaxStepCount in stepmgr.
-- Enable AccountingStorageEnforce=nojobs,nosteps in stepmgr.
-- Add AccountingStorageParameters=max_step_records to limit how many steps
are recorded in the database for each job -- excluding batch, extern, and
interactive steps.
-- switch/hpe_slingshot - allocate VNIs on the controller for stepmgr jobs and
pass to the stepmgr for steps to use.
-- switch/hpe_slingshot - fix assertion when restarting the controller.
-- switch/hpe_slingshot - fix calcuation of free vnis when restarting the
controller with running jobs.
-- Improve default job reserve MPI port allocations that use overcommit or do
not specify a task count for stepmgr enabled jobs.
-- Fix a regression in rc1 resulting in scrun occasionally deadlocking when
the --enable-memory-leak-debug configure option was used.
-- topology/default - Prevent segfault in slurmctld on 'scontrol show topo'.
-- slurmrestd - Avoid creating or requring a connection to slurmdbd for the
'GET /openapi/v3' endpoint, fixing a regression in rc1.
-- scrun - Fix setting and getting environment via SPANK plugins.
-- sview - Fix nodes tab if a node has RestrictedCoresPerGPU configured.
-- slurmrestd - Add --generate-openapi-spec argument.
-- sview - Prevent segfault when retrieving slurmdbd configuration.
-- Avoid canceling rejected heterogeneous jobs without job write lock.
-- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running.
-- Fix slurmctld crash after a job has been resized.
* Changes in Slurm 24.05.0rc1
=============================
-- Make slurmstepd retry REQUEST_COMPLETE_BATCH_SCRIPT indefinitely.
-- Always load serializer/json when using any data_parser plugins.
-- slurmrestd - Reject single http query with multiple path requests.
-- slurmrestd - Add time/planned field to slurmdb/v0.0.41/job/{job_id}.
-- Improve Power Save's Resume/Suspend rate limiting.
-- slurmrestd - Improve reliablity under high memory pressure by closing
connections instead of forcing a fatal exit due to lack of memory.
-- data_parser/v0.0.41 - Avoid aborting when invoking a not implemented
parser.
-- data_parser/v0.0.41 - Fix how nice values are parsed for job submissions.
-- data_parser/v0.0.41 - Fix regression where parsing error did not result in
invalid request being rejected.
-- Print an error message in 'scontrol reboot' when a node reboot request
is ignored due to the current node state.
-- squeue - Add "--notme" option.
-- data_parser/v0.0.41 - change "association.id" to just include the int "id"
rather than include redundant assoc info (cluster, user, partition, account)
that's already included in the "association" object.
-- data_parser/v0.0.41 - Improve parsing of numeric user id.
-- data_parser/v0.0.41 - Improve parsing of numeric group id.
-- slurmrestd - Generated openapi.json will only populate "deprecated" fields
if true. False is the default value and does not require being present.
-- slurmrestd - Populate missing "deprecated" fields in openapi.json.
-- slurmrestd - Corrected deprecated fields in generated openapi.json not
getting populated.
-- slurmrestd - Generated openapi.json will have reduced number of "$ref"
fields. Where there was only 1 reference for the schema, the "$ref" schema
will be directly populated in place.
-- slurmrestd - Rename *_NO_VAL schemas in generated openapi.json to have
_struct and to pass along correct integer sizing when possible.
-- slurmrestd - Correct description fields in generated openapi.json where
descriptions were not present or too generic.
-- Remove support for Cray XC ("cray_aries") systems.
-- Prevent backup slurmctld from taking over if the heartbeat file is
still being updated. Failure to ping may have been due to clock skew.
-- serializer/yaml - Converted to new parsing interface in libyaml to improve
parsing compatibility.
-- Removed TopologyPlugin tree and dragonfly support from select/linear.
If those topology plugins are desired please switch to select/cons_tres.
-- Changed slurmrestd.service to only listen on TCP socket by default.
Environments with existing drop-in units for the service may need
further adjustments to work after upgrading.
-- Fix how gres are allocated per job when using multiple gres types.
-- Log an error when UnkillableStepTimeout is less than five times
MessageTimeout.
-- Avoid step gres dealloc count underflow errors after reconfiguring or
restarting slurmctld.
-- Fix controller not validating periodic dynamic future registrations.
-- Fix dynamic future nodes registering as new node when specifying -N<name>.
-- Fix sbcast (or srun --bcast) --send-libs when it is used multiple times in
the same job. Previously, subsequent calls to sbcast --send-libs would
overwrite the libraries for the first executable.
-- Add support for sbcast --preserve when job_container/tmpfs configured
(previously documented as unsupported).
-- Changed the default value for UnkillableStepTimeout to 60 seconds or five
times the value of MessageTimeout, whichever is greater.
-- slurmctld - Check if --deadline has been reached and not satisfied on held
jobs, otherwise they could remain without automatic cancellation until after
the job is released.
-- scrun/slurmrestd/sackd - Avoid closing all listening sockets when
interrupted from signal such as SIGALRM. Normal shutdown remains unaffected.
-- Remove systemd AbandonScope() logic for scope units as it is not needed.
-- Fix GresUsed output from `scontrol show nodes --details` showing GRES types
that are not configured on a node.
-- slurmrestd - Fatal during start up when loading content plugin fails.
-- slurmrestd - Reduce complexity in URL path matching.
-- data_parser/v0.0.41 - Emit a warning instead of an error if a disabled
parser is invoked.
-- Federation - allow client command operation when slurmdbd is unavailable.
-- Enforce mutual exclusivity of --systemd and -D when launching daemons
-- slurmctld - remove -d option
-- burst_buffer/lua - Trigger a burst_buffer event for strigger when the
real_size function fails.
-- burst_buffer/lua - Added two new hooks: slurm_bb_test_data_in and
slurm_bb_test_data_out. The syntax and use of the new hooks are documented
in etc/burst_buffer.lua.example. These are required to exist. slurmctld now
checks on startup if the burst_buffer.lua script loads and contains all
required hooks; slurmctld will exit with a fatal error if this is not
successful. Added PollInterval to burst_buffer.conf. Removed the arbitrary
limit of 512 copies of the script running simultaneously.
-- sackd/slurmrestd/scrun - Avoid using empty string while logging unix socket
connections from a listening connection.
-- Fix 20 character username limit from 'sacctmgr show events'
-- Log an error if UsePss or NoShare are configured with a plugin other than
jobacct_gather/linux. In such case these parameters are ignored.
-- helpers.conf - Added Flags=rebootless parameter allowing feature changes
without rebooting compute nodes.
-- scontrol - Add new subcommand 'power' for node power control.
-- data_parser/v0.0.41 - Implement parser of distribution for
/slurm/v0.0.41/job/submit.
-- data_parser/v0.0.41 - Change distribution_plane_size field type from UINT16
to UINT16_NO_VAL for /slurm/v0.0.41/job/submit.
-- topology/block - Replaced the BlockLevels with BlockSizes in topology.conf.
-- Fix slurmd cgroup/v2 startup race with systemd and cgroupfs.
-- Add SystemdTimeout= parameter in cgroup.conf.
-- Add QOS limit MaxTRESRunMinsPerAccount.
-- Add QOS limit MaxTRESRunMinsPerUser.
-- jobcomp/{elasticsearch,kafka} - Send priority alongside the rest of fields.
-- Add contain_spank option to SlurmdParameters. When set, spank_user_init(),
spank_task_post_fork(), and spank_task_exit() will execute within the
job_container/tmpfs plugin namespace.
-- Update job reason appropriately when bf_licenses is used.
-- slurmrestd - Tagged `script` field as deprecated in
'POST /slurm/v0.0.41/job/submit' in anticipation of removal in future
OpenAPI plugin versions.
-- Fix salloc/sbatch/srun crashing with certain invalid nodelist requests.
-- Optimize jobacctgather by not iterating every time over pids that have
already finished.
-- Remote SPANK callbacks invoked by srun get called once instead of twice.
-- auth/slurm - Support multiple keys through slurm.jwks.
-- sched/backfill - Fix issue with bf_continue where a job partition request
could be incorrectly reset back to a partition that is no longer specified
after a job partition update processed during a lock yield time window.
-- slurmrestd - Explicitly set process as dumpable (and ptrace-able) at
startup for systems where suid_dumpable is not 2.
-- slurmrestd - Tag all /slurm/v0.0.39/ and /slurmdb/v0.0.39/ endpoints as
deprecated in anticipation of removal in Slurm 24.11.
-- Add ELIGIBLE environment variable to jobcomp/script plugin.
-- slurmrestd,sackd,scrun - Improve outgoing data efficiency using
non-contiguous write support in kernel.
-- sackd - Add support for SACKD_DEBUG, SACKD_STDERR_DEBUG, and
SACKD_SYSLOG_DEBUG environment variables to control logging.
-- mpi/pmi2 - PMI_process_mapping values have been adapted for executions
where arbitrary distribution/SLURM_HOSTFILE is used. Now it can take into
account multiple instances of the same node inside SLURM_HOSTFILE.
-- Avoid wrong limit oriented (i.e. QosMaxGresPer*) job pending reason for jobs
actually pending on Resources when GPUs are requested per job.
-- Fix --ntasks-per-node not being treated as a max count of tasks per node
when used in combination with --ntasks. --ntasks option will now take
precedence as it is documented.
-- Accept X11 cookies that do not have a display number associated with it.
-- Always use the QOS name for SLURM_JOB_QOS environment variables.
Previously the batch environment would use the description field,
which was usually equivalent to the name.
-- slurmrestd - Add "CRON_JOBS" as possible flag value to the following:
'DELETE /slurm/v0.0.40/jobs' flags field.
'DELETE /slurm/v0.0.41/jobs' flags field.
'DELETE /slurm/v0.0.40/job/{job_id}?flags=' flags query parameter.
'DELETE /slurm/v0.0.41/job/{job_id}?flags=' flags query parameter.
-- Fix ScronParameters=explicit_scancel when using the rest api DELETE jobs
query: if the CRON_JOBS flag is not used then cron jobs will not be
cancelled. The NO_CRON_JOBS flag is ignored in v0.0.40 and removed in
v0.0.41.
-- Pass multi-partition job priorities to job for squeue to display.
-- cgroup/v2 - Require dbus-1 version >= 1.11.16.
-- Add RestrictedCoresPerGPU configuration option.
-- Fix how ntasks is inferred from --cpus-per-task when using --nodes,
--threads-per-core, or --hint=nomultithread.
-- For PreemptMode=CANCEL and PreemptMode=REQUEUE assume that job signalled
for GraceTime was preempted.
-- slurmd - Retry fetching configs indefinitely during startup.
-- Fix SPANK options not bing sent to remote context when --export was used.
-- slurmrestd - Attempt to automatically convert enumerated string arrays with
incoming non-string values into strings. Add warning when incoming value for
enumerated string arrays can not be converted to string and silently ignore
instead of rejecting entire request.
-- slurmrestd - Require `user` and `association_condition` fields to be
populated for requests to 'POST /slurmdb/v0.0.41/users_association'.
-- Allow NodeSet names to be used in SuspendExcNodes.
-- SuspendExcNodes=<nodes>:N now counts allocated nodes in N. The first N
powered up nodes in <nodes> are protected from being suspended.
-- Add SlurmctldParameters=max_powered_nodes=N, which prevents powering up
nodes after the max is reached.
-- Store output, error and input paths in the database and make them available
in accounting tools.
-- slurmrestd - Add 'POST /slurm/v0.0.41/job/allocate' endpoint.
-- Fix issues related to the extern step getting killed before other steps.
This includes the job_containter/tmpfs plugin not cleaning up.
-- Add USER_DELETE reservation flag to allow users with access to a reservation
to delete it.
-- Add CgroupPlugin=disabled to disable any interaction with Cgroups.
-- slurmrestd - Add "STEPMGR_ENABLED" as possible flag value to the following:
'GET /slurm/v0.0.41/jobs' flags field.
'GET /slurm/v0.0.41/job/{job_id}' flags query parameter.
-- scontrol,squeue - Added possible flags "STEPMGR_ENABLED" to '.jobs[].flags' for
'scontrol show jobs --{json|yaml}' and 'squeue --{json|yaml}' responses.
-- Add SlurmctldParameters=enable_stepmgr to enable step management through
the slurmstepd instead of the controller.
-- Avoid slurmstepd infinite loop waiting for tasks termination.
-- Fix logging of JSON/YAML values in some messages where nothing would be
printed as the value instead of the actual JSONified version of the parsed
string.
-- slurmrestd,sackd,scrun - Improve logic around handling kernel provided
buffer size of incoming data in files/sockets/pipes to avoid crashes.
-- Add --segment to job allocation to be used in topology/block.
-- Add --exclusive=topo for use with topology/block.
-- Add ExclusiveTopo to a partition definition in slurm.conf.
-- Add new 'BLOCKED' state to a node.
-- Account coordinators may not increase association job limits above
parent ones
-- Account coordinators can now suspend/resume jobs owned by member users.
-- Add DisableCoordDBD slurmdbd configuration parameter to disable the
coordinator status in all slurmdbd interactions.
-- slurmrestd - Added possible flags "WithAssociations" and "WithCoordinators"
to `.accounts[].flags` for "GET /slurmdb/v0.0.41/accounts/" and
"POST /slurmdb/v0.0.41/accounts/" endpoints.
-- sacctmgr - Added possible flags "WithAssociations" and "WithCoordinators"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- slurmrestd - Rename URL query parameter "with_assocs" to "WithAssociations"
for "GET /slurmdb/v0.0.41/accounts?WithAssociations".
-- slurmrestd - Rename URL query parameter "with_coords" to "WithCoordinators"
for "GET /slurmdb/v0.0.41/accounts?WithCoordinators".
-- slurmrestd - Rename URL query parameter "with_deleted" to "deleted"
for "GET /slurmdb/v0.0.41/accounts?deleted".
-- slurmrestd - Added possible flags "RemoveUsersAreCoords" and
"UsersAreCoords" to `.accounts[].flags` for "GET /slurmdb/v0.0.41/accounts/"
and "POST /slurmdb/v0.0.41/accounts/" endpoints.
-- sacctmgr - Added possible flags "RemoveUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- slurmrestd - Add URL query parameter "UsersAreCoords" and
"RemoveUsersAreCoords" for
"GET /slurmdb/v0.0.41/accounts?UsersAreCoords&RemoveUsersAreCoords".
-- sacctmgr - Add new possible new flags "NoUpdate" and "Exact" to
'.associations[].flags' response from 'sacctmgr show assocs --{json|yaml}'.
-- slurmrestd - Added possible flags "NoUpdate" and "Exact" to
`.associations[].flags` for "GET /slurmdb/v0.0.41/associations/"
and "POST /slurmdb/v0.0.41/associations/" endpoints.
-- Fix false success of REQUEST_FORWARD_DATA RPC that made pmix to get out
of sync during initialization.
-- slurmrestd - Allow startup when slurmdbd is not configured and avoid loading
slurmdbd specific plugins.
-- Added PrologFlags=RunInJob to make prolog and epilog run inside the job
extern step to include it in the job's cgroup.
-- Return '*' for the password field for nss_slurm instead of "x".
-- slurmrestd - Add "topo" as possible value to the following:
'GET /slurm/v0.0.41/jobs' in '.jobs[].shared' field
'GET /slurm/v0.0.41/job/{job_id}' in '.jobs[].shared' field
'POST /slurm/v0.0.41/job/submit' in '.job.shared' and '.jobs[].shared'
'POST /slurm/v0.0.41/job/allocate' in '.job.shared' and '.jobs[].shared'
-- sacctmgr - Added possible flags "NoUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- sacct - Add "topo" as possible value to output of 'sacct --{json|yaml}' to
'.jobs[].shared' field.
-- squeue - Add "topo" as possible value to output of 'squeue --{json|yaml}' to
'.jobs[].shared' field.
-- scontrol - Add "topo" as possible value to output of
'scontrol show jobs --{json|yaml}' to '.jobs[].shared' field.
-- slurmrestd - Add "topo" as possible value to the following:
'GET /slurm/v0.0.41/jobs' in '.jobs[].exclusive' field
'GET /slurm/v0.0.41/job/{job_id}' in '.jobs[].exclusive' field
'POST /slurm/v0.0.41/job/submit' in '.job.exclusive' and
'.jobs[].exclusive'
'POST /slurm/v0.0.41/job/allocate' in '.job.exclusive' and
'.jobs[].exclusive'
-- sacctmgr - Added possible flags "RemoveUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- sacct - Add "topo" as possible value to output of 'sacct --{json|yaml}' to
'.jobs[].exclusive' field.
-- squeue - Add "topo" as possible value to output of 'squeue --{json|yaml}' to
'.jobs[].exclusive' field.
-- scontrol - Add "topo" as possible value to output of
'scontrol show jobs --{json|yaml}' to '.jobs[].exclusive' field.
-- slurmrestd - Add fields '.job.segment_size' and '.jobs[].segment_size' to
'POST /slurm/v0.0.41/job/submit' and 'POST /slurm/v0.0.41/job/allocate'.
-- sacctmgr - Added possible flags "NoUsersAreCoords" and "UsersAreCoords"
to `.associations[].flags` for `sacctmgr show assocs --{json|yaml}`
response.
-- slurmrestd - Added possible flags "NoUsersAreCoords" and "UsersAreCoords" to
`.associations[].flags` for "GET /slurmdb/v0.0.41/associations/"
and "POST /slurmdb/v0.0.41/associations/" endpoints.
-- Add ability to reserve MPI ports at the job level for stepmgr jobs and
subdivide them at the step level.
-- slurmrestd - Fix possible memory leak from failed job submissions to
'POST /slurm/v0.0.{39,40,41}/job/submit'.
-- slurmrestd - Fix possible memory leak from failed job allocation requests to
'POST /slurm/v0.0.{39,40,41}/job/allocate'.
* Changes in Slurm 23.11.11
===========================
* Changes in Slurm 23.11.10
===========================
-- switch/hpe_slingshot - Fix issue that could result in a 0 length state file.
-- Fix unnecessary message protocol downgrade for unregistered nodes.
-- Fix unnecessarily packing alias addrs when terminating jobs with a mix of
non-cloud/dynamic nodes and powered down cloud/dynamic nodes.
-- Fix allowing access to reservations without MaxStartDelay set.
-- Fix scontrol allowing updating job with bad cpus-per-task value.
-- sattach - Fix regression from 23.11.9 security fix leading to crash.
* Changes in Slurm 23.11.9
==========================
-- Fix many commands possibly reporting an "Unexpected Message Received" when
in reality the connection timed out.
-- Fix heterogeneous job components not being signaled with scancel --ctld and
'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given,
the heterogeneous job components match the given filters, and the
heterogeneous job leader does not match the given filters.
-- Fix regression from 23.02 impeding job licenses from being cleared.
-- Move error to log_flag which made _get_joules_task error to be logged to the
user when too many rpcs were queued in slurmd for gathering energy.
-- slurmrestd - Prevent a slurmrestd segfault when modifying an association
without specifying max TRES limits in the request if those TRES
limits are currently defined in the association. This affects the following
fields of endpoint 'POST /slurmdb/v0.0.38/associations/':
'associations/max/tres/per/job'
'associations/max/tres/per/node'
'associations/max/tres/total'
'associations/max/tres/minutes/per/job'
'associations/max/tres/minutes/total'
-- Fix power_save operation after recovering from a failed reconfigure.
-- scrun - Delay shutdown until after start requested. This caused scrun
to never start or shutdown and hung forever when using --tty.
-- Fix backup slurmctld potentially not running the agent when taking over as
the primary controller.
-- Fix primary controller not running the agent when a reconfigure of the
slurmctld fails.
-- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time.
-- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when
more than 10 threads try to get energy at the same time. This prevented
the possibility to get energy from slurmd by any step until slurmd was
restarted, so losing energy accounting metrics in the node.
-- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the
query parameters specified account, association, cluster, constraints,
format, groups, job_name, partition, qos, reason, reservation, state, users,
or wckey. This affects the following endpoints:
'GET /slurmdb/v0.0.39/jobs'
-- switch/hpe_slingshot - Fix security issue around managing VNI access.
CVE-2024-42511.
* Changes in Slurm 23.11.8
==========================
-- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running.
-- Fix slurmctld crash after a job has been resized.
-- Fix slurmctld and slurmdbd potentially stopping instead of performing a
logrotate when recieving SIGUSR2 when using auth/slurm.
-- Fix not having a disabled value for keepalive CommunicationParameters in
slurm.conf when these parameters are not set. This can log an error when
setting a socket, for example during slurmdbd registration with ctld.
-- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02.
-- Fix "Could not find group" errors from validate_group() when using
AllowGroups with large /etc/group files.
-- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field,
which was never usable. Now it explicitly ignores the value and emits a
warning if it is used for the following endpoints:
'POST /slurm/v0.0.39/job/{job_id}'
'POST /slurm/v0.0.39/job/submit'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.40/job/submit'
-- Fix getting user environment when using sbatch with "--get-user-env" or
"--export=" when there is a user profile script that reads /proc.
-- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but
GresTypes is not configured.
-- Do not log the following errors when AcctGatherEnergyType plugins are used
but a node does not have or cannot find sensors:
"error: _get_joules_task: can't get info from slurmd"
"error: slurm_get_node_energy: Zero Bytes were transmitted or received"
However, the following error will continue to be logged:
"error: Can't get energy data. No power sensors are available. Try later"
-- Fix cloud nodes not being able to forward to nodes that restarted with new
IP addresses.
-- sacct - Fix printing of job group for job steps.
-- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1.
-- Fix slurmctld crash on a batch job submission with "--nodes 0,...".
-- Fix dynamic IP address fanout forwarding when using auth/slurm.
* Changes in Slurm 23.11.7
==========================
-- slurmrestd - Correct OpenAPI specification for
'GET /slurm/v0.0.40/jobs/state' having response as null.
-- Allow running jobs on overlapping partitions if jobs don't specify -s.
-- Fix segfault when requesting a shared gres along with an exclusive
allocation.
-- Fix regression in 23.02 where afternotok and afterok dependencies were
rejected for federated jobs not running on the origin cluster of the
submitting job.
-- slurmctld - Disable job table locking while job state cache is active when
replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`.
-- Fix sanity check when setting tres-per-task on the job allocation as well as
the step.
-- slurmrestd - Fix compatiblity with auth/slurm.
-- Fix issue where TRESRunMins gets off correct value if using
QOS UsageFactor != 1.
-- slurmrestd - Require `user` and `association_condition` fields to be
populated for requests to 'POST /slurmdb/v0.0.40/users_association'.
-- Avoid a slurmctld crash with extra_constraints enabled when a job requests
certain invalid --extra values.
-- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job
array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array
task (e.g. 1_10), which previously signaled the whole array job instead.
-- Fix a possible slurmctld segfault when at some point we failed to create an
external launcher step.
-- Allow the slurmctld to open a connection to the slurmdbd if the first
attempt fails due to a protocol error.
-- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob.
-- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units.
-- Fix a rare deadlock on slurmctld shutdown or reconfigure.
-- Fix issue that only left one thread on each core available when "CPUs=" is
configured to total thread count on multi-threaded hardware and no other
topology info ("Sockets=", "CoresPerSocket", etc.) is configured.
-- Fix the external launcher step not being allocated a VNI when requested.
-- jobcomp/kafka - Fix payload length when producing and sending a message.
-- scrun - Avoid a crash if RunTimeDelete is called before the container
finishes.
-- Save the slurmd's cred_state while reconfiguring to prevent the loss job
credentials.
* Changes in Slurm 23.11.6
==========================
-- Avoid limiting sockets per node to one when using gres enforce-binding.
-- slurmrestd - Avoid permission denied errors when attempting to listen on
the same port multiple times.
-- Fix GRES reservations where the GRES has no topology
(no cores= in gres.conf).
-- Ensure that thread_id_rpc is gone before priority_g_fini().
-- Fix scontrol reboot timeout removing drain state from nodes.
-- squeue - Print header on empty reponse to `--only-job-state`.
-- Fix slurmrestd not ending job properly when xauth is not present and a x11
job is sent.
-- Add experimental job state caching with
SchedulerParameters=enable_job_state_cache to speed up querying job states
with squeue --only-job-state.
-- slurmrestd - Correct dumping of invalid ArrayJobIds returned from
'GET /slurm/v0.0.40/jobs/state'.
-- squeue - Correct dumping of invalid ArrayJobIds returned from
`squeue --only-job-state --{json|yaml}`.
-- If scancel --ctld is not used with --interactive, --sibling, or specific
step ids, then this option issues a single request to the slurmctld to
signal all jobs matching the specified filters. This greatly improves
the performance of slurmctld and scancel. The updated --ctld option also
fixes issues with the --partition or --reservation scancel options for jobs
that requested multiple partitions or reservations.
-- slurmrestd - Give EINVAL error when failing to parse signal name to numeric
signal.
-- slurmrestd - Allow ContentBody for all methods per RFC7230 even if ignored.
-- slurmrestd - Add 'DELETE /slurm/v0.0.40/jobs' endpoint to allow bulk job
signaling via slurmctld.
-- Fix combination of --nodelist and --exclude not always respecting the
excluded node list.
-- Fix jobs incorrectly allocating nodes exclusively when started on a
partition that doesn't enforce it. This could happen if a multi-partition
job doesn't specify --exclusive and is evaluated first on a partition
configured with OverSubscribe=EXCLUSIVE but ends up starting in a partition
configured with OverSubscribe!=EXCLUSIVE evaluated afterwards.
-- Setting GLOB_SILENCE flag no longer exposes old bugged behavior.
-- Fix associations AssocGrpCPURunMinutes being incorrectly computed for
running jobs after a controller reconfiguration/restart.
-- Fix scheduling jobs that request --gpus and nodes have different node
weights and different numbers of gpus.
-- slurmrestd - Add "NO_CRON_JOBS" as possible flag value to the following:
'DELETE /slurm/v0.0.40/jobs' flags field.
'DELETE /slurm/v0.0.40/job/{job_id}?flags=' flags query parameter.
-- Fix scontrol segfault/assert failure if the TRESPerNode parameter is used
when creating reservations.
-- Avoid checking for wsrep_on when restoring streaming replication settings.
-- Clarify in the logs that error "1193 Unknown system variable 'wsrep_on'" is
innocuous.
-- accounting_storage/mysql - Fix problem when loading reservations from an
archive dump.
-- slurmdbd - Fix minor race condition when sending updates to a shutdown
slurmctld.
-- slurmctld - Fix invalid refusal of a reservation update.
-- openapi - Fix memory leak of /meta/slurm/cluster response field.
-- Fix memory leak when using auth/slurm and AuthInfo=use_client_ids.
* Changes in Slurm 23.11.5
==========================
-- Fix Debian package build on systems that are not able to query the systemd
package.
-- data_parser/v0.0.40 - Emit a warning instead of an error if a disabled
parser is invoked.
-- slurmrestd - Improve handling when content plugins rely on parsers
that haven't been loaded.
-- Fix old pending jobs dying (Slurm version 21.08.x and older) when upgrading
Slurm due to "Invalid message version" errors.
-- Have client commands sleep for progressively longer periods when backed off
by the RPC rate limiting system.
-- slurmctld - Ensure agent queue is flushed correctly at shutdown time.
-- slurmdbd - correct lineage construction during assoc table conversion for
partition based associations.
-- Add new RPCs and API call for faster querying of job states from slurmctld.
-- slurmrestd - Add endpoint '/slurm/{data_parser}/jobs/state'.
-- squeue - Add `--only-job-state` argument to use faster query of job states.
-- Make a job requesting --no-requeue, or JobRequeue=0 in the slurm.conf,
supersede RequeueExit[Hold].
-- Add sackd man page to the Debian package.
-- Fix issues with tasks when a job was shrinked more than once.
-- Fix reservation update validation that resulted in reject of correct
updates of reservation when the reservation was running jobs.
-- Fix possible segfault when the backup slurmctld is asserting control.
-- Fix regression introduced in 23.02.4 where slurmctld was not properly
tracking the total GRES selected for exclusive multi-node jobs, potentially
and incorrectly bypassing limits.
-- Fix tracking of jobs typeless GRES count when multiple typed GRES with the
same name are also present in the job allocation. Otherwise, the job could
bypass limits configured for the typeless GRES.
-- Fix tracking of jobs typeless GRES count when request specification has a
typeless GRES name first and then typed GRES of different names (i.e.
--gres=gpu:1,tmpfs:foo:2,tmpfs:bar:7). Otherwise, the job could bypass
limits configured for the generic of the typed one (tmpfs in the example).
-- Fix batch step not having SLURM_CLUSTER_NAME filled in.
-- slurmstepd - Avoid error during `--container` job cleanup about
RunTimeQuery never being configured. Results in cleanup where job steps not
fully started.
-- Fix nodes not being rebooted when using salloc/sbatch/srun "--reboot" flag.
-- Send scrun.lua in configless mode.
-- Fix rejecting an interactive job whose extra constraint request cannot
immediately be satisfied.
-- Fix regression in 23.11.0 when parsing LogTimeFormat=iso8601_ms that
prevented milliseconds from being printed.
-- Fix issue where you could have a gpu allocated as well as a shard on that
gpu allocated at the same time.
-- Fix slurmctld crashes when using extra constraints with job arrays.
-- sackd/slurmrestd/scrun - Avoid memory leak on new unix socket connection.
-- The failed node field is filled when a node fails but does not time out.
-- topology/block - Implement topology_p_generate_node_ranking().
-- slurmrestd - Remove requiring job script field and job component script
fields to both be populated in the `POST /slurm/v0.0.40/job/submit`
endpoint as there can only be one batch step script for a job.
-- slurmrestd - When job script is provided in '.jobs[].script' and '.script'
fields, the '.script' field's value will be used in the
`POST /slurm/v0.0.40/job/submit` endpoint.
-- slurmrestd - Reject HetJob submission missing or empty batch script for
first Het component in the `POST /slurm/v0.0.40/job/submit` endpoint.
-- slurmrestd - Reject job when empty batch script submitted to the
POST /slurm/v0.0.40/job/submit` endpoint.
-- Fix pam_slurm and pam_slurm_adopt when using auth/slurm.
-- slurmrestd - Add 'cores_per_socket' field to
`POST /slurm/v0.0.40/job/submit` endpoint.
-- Fix srun and other Slurm commands running within a "configless" salloc when
salloc itself fetched the config.
-- Enforce binding with shared gres selection if requested.
-- Fix job allocation failures when the requested tres type or name ends in
"gres" or "license".
-- accounting_storage/mysql - Fix lineage string construction when adding a
user association with a partition.
-- Fix sattach command.
-- Fix ReconfigFlags. Due how reconfig was changed in 23.11, they will also
be used to influence the slurmctld startup as well.
-- Fix starting slurmd in configless mode if MUNGE support was disabled.
* Changes in Slurm 23.11.4
==========================
-- Fix a memory leak when updating partition nodes.
-- Don't leave a partition around if it fails to create with scontrol.
-- Fix segfault when creating partition with bad node list from scontrol.
-- Fix preserving partition nodes on bad node list update from scontrol.
-- Fix assertion in developer mode on a failed message unpack.
-- Fix repeat POWER_DOWN requests making the nodes available for ping.
-- Fix rebuilding job alias_list on restart when nodes are still powering up.
-- Fix INVALID nodes running health check.
-- Fix cloud/future nodes not setting addresses on invalid registration.
-- scrun - Remove the requirement to set the SCRUN_WORKING_DIR environment
variable. This was a regression in 23.11.
-- Add warning for using select/linear with topology/tree.
This combination will not be supported in the next major version.
-- Fix health check program not being run after first pass of all nodes when
using MaxNodeCount.
-- sacct - Set process exit code to one for all errors.
-- Add SlurmctldParameters=disable_triggers option.
-- Fix issue running steps when the allocation requested an exclusive
allocation shards along with shards.
-- Fix cleaning up the sleep process and the cgroup of the extern step if
slurm_spank_task_post_fork returns an error.
-- slurm_completion - Add missing --gres-flags= options
multiple-tasks-per-sharing and one-task-per-sharing.
-- scrun - Avoid race condition that could cause outbound network
communications to incorrectly rejected with an incomplete packet error.
-- scrun - Gracefully handle kernel giving invalid expected number of incoming
bytes for a connection causing incoming packet corruption resulting in
connection getting closed.
-- srun - return 1 when a step lauch fails
-- scrun - Avoid race condition that could cause deadlock during shutdown.
-- Fix scontrol listpids to work under dynamic node scenarios.
-- Add --tres-bind to --help and --usage output.
-- Add --gres-flags=allow-task-sharing to allow GPUs to still be accessible
among all tasks when binding GPUs to specific tasks.
-- Fix issue with CUDA_VISIBLE_DEVICES showing the same MIG device for all
tasks when using MIGs with --tres-per-task or --gpus-per-task.
-- slurmctld - Prevent a potential hang during shutdown/reconfigure if the
association cache thread was previously shut down.
-- scrun - Avoid race condition that could cause scrun to hang during
shutdown when connections have pending events.
-- scrun - Avoid excessive polling of connections during shutdown that could
needlessly cause 100% CPU usage on a thread.
-- sbcast - Use user identity from broadcast credential instead of looking it
up locally on the node.
-- scontrol - Remove "abort" option handling.
-- Fix an error message referring to the wrong RPC.
-- Fix memory leak on error when creating dynamic nodes.