Skip to content
GitLab
  1. 07 Aug, 2018 2 commits
    • Morris Jette's avatar
      Avoid node config_list entry fragmentation. · be0abe8a
      Morris Jette authored 
      Only split nodes here if a node_features plugin is in use.

Otherwise node fragmentation will occur if the node config has
CPUs specified but not CoresPerSocket and Sockets.

This could be avoided by filling out the node definition, but
adding this workaround for backwards compatiblity.

Bug 5039.
      be0abe8a
    • Marshall Garey's avatar
      Create TMPDIR after task prologs have run. · 0c560d67
      Marshall Garey authored 
      Task prologs could set or modify this, so wait to create the
directory until after they've finished.

Bug 5367.
      0c560d67
  3. 06 Aug, 2018 3 commits
    • Tim Wickberg's avatar
      Change debug messages() in _launch_handler(). · c3c78acd
      Tim Wickberg authored 
      After changes to slurm_send_only_node_msg(), this message is much
more likely to appear on systems with overloaded interconnects since
that connection handling code may end up retransmitting messages
that were actually received (but that the transmit side could not
verify were delivered successfully).

As the error() message stated, this isn't actually an error, and
the code will proceed happily past this point. So drop the debug
level, and remove the surrealist "this is not an error" part.

Bug 5164.
      c3c78acd
    • Tim Wickberg's avatar
      Modify slurm_send_only_node_msg() to catch issues with socket. · 06582da8
      Tim Wickberg authored 
      There are subtle issues involved in treating a TCP transmission
as a unidirectional message delivery layer.

The original code path looks like: connect(), write(), close().
But Linux handles the write() and close() asynchronously behind the
scenes, and does not block until that write() has been ACK'd by the
remote end. So the write() and close() may succeed, even with data
still in flight. A communication error - and message loss - would
have been silently ignored, leading to unreliable message transmission.

Worse yet, one side of the connection would believe it sent the message,
while the receive side swears it never saw the packets. This leads to
infrequent and yet seemingly impossible data loss, and a very tough
bug to chase down.

This teardown code tries to force the connection to shut down in an
orderly manner, giving Slurm a chance to catch a connection problem
and the upstream calling path an opportunity to retransmit.

This teardown code is based on an approach described in Section 7.5
of "UNIX Network Programming" Volume 1 (Third Edition), specifically
the subsection regarding SO_LINGER. (And also covers why SO_LINGER is
not sufficent to prevent this issue.)

Bug 5164.
      06582da8
    • Tim Wickberg's avatar
      Retransmit on all errors. · a572d5d6
      Tim Wickberg authored 
      Bug 5164.
      a572d5d6
  5. 04 Aug, 2018 1 commit
  7. 31 Jul, 2018 1 commit
  9. 27 Jul, 2018 2 commits
  11. 24 Jul, 2018 1 commit
  13. 19 Jul, 2018 7 commits
  15. 18 Jul, 2018 5 commits
  17. 17 Jul, 2018 5 commits
  19. 16 Jul, 2018 2 commits
  21. 13 Jul, 2018 1 commit
  23. 12 Jul, 2018 4 commits
  25. 09 Jul, 2018 1 commit
  27. 06 Jul, 2018 5 commits