- 23 Mar, 2016 39 commits
-
-
Danny Auble authored
-
Danny Auble authored
-
Danny Auble authored
-
Tim Wickberg authored
With bcast split into its own directory -lz should not be required throughout. This reverts commit e7981406.
-
Tim Wickberg authored
Remove unused struct and macro from file_bcast.h. Free file_bcast_info_t to prevent leak.
-
Danny Auble authored
-
Danny Auble authored
-
Tim Wickberg authored
1) Add a new global file_bcast_list to store info on in-progress file transfers, cache FD there rather than reopening the file for every block. 2) Restructure security mechanisms. First block will fork() and open the file, and pass the FD back to the thread. Thread then registers this file transfer in the file_bcast_list. Split fork() stuff into _file_bcast_register_file to keep _rpc_file_bcast readable. 3) Successive blocks are handled within the thread. Security is handled by matching uid and file name to existing file transfer. TODO: 1) Write transfer cleanup function to remove stalled transfers. 2) Use mmap for file output. 3) Allow for parallel block transfer. Current code assumes blocks will always arrive in order. Out of order blocks will result in corrupted output. (sbcast currently prevents this by requiring each message to be ack'd before continuing, but at a likely severe performance penalty.) 4) Add stats on receive side.
-
Tim Wickberg authored
This reverts commit 8c8c3407488fe3f0a552d2359ef5b487330ee8ba. Thread-only isn't portable, need to use fork() on first block to ensure file security and containers are handled correctly.
-
Tim Wickberg authored
-
Tim Wickberg authored
Fix bad cast in 3a604563, and update pct to 64-bits to prevent truncation of intermediate value (pct * 100).
-
Danny Auble authored
-
Morris Jette authored
Update srun and sbcast man pages and help messages for compression library argument.
-
Tim Wickberg authored
Relies on Linux-specific behavior of setfsuid/gid, so disable on other platforms for now.
-
Danny Auble authored
Fix linking issues to only have the compression libs link to file_bcast which intern get pulled in my anyone linking to the .la.
-
Danny Auble authored
Move compression stuff over to the file_bcast lib, so we only have to link directly with that and not pull libs all over the place.
-
Tim Wickberg authored
- Fix issue where max_out calculation for zlib was incorrect - use deflateBound to properly calculate required buffer size, 1024 is not sufficient padding for uncompressable input. - Work towards sending file offsets across the wire in preparation for mmap'd output for lz4.
-
Tim Wickberg authored
-
Tim Wickberg authored
Rather than compressing at most block_len into a message, compress up to (10 * block_len) into a single message. 10x arbitrarily chosen to mitigate buffer issues when uncompresssing in slurmd. Testing against a 100MB zero file, this reduces the messages required from 13 to 2.
-
Tim Wickberg authored
-
Tim Wickberg authored
Regression in last commit required zlib to send each chunk as a separate block, rather than packing multiple chunks per block.
-
Tim Wickberg authored
-
Tim Wickberg authored
1) Use mmap'd I/O. 2) Move buffer handling down to the individual compression routines. This will eventually allow them to read more than buffer_len at a time in order to emit a buffer_len message, and thus minimize the number of messages send. (Current logic sends the same number of messages, but the payload for each can be compressed. I expect more performance gains are available by limit the message count.) 3) Move buffer_len to a global, and change function signatures.
-
Danny Auble authored
-
Danny Auble authored
Move files from common to new dir location for bcast. This also sets up the 3 locations that need to be linked to compression libs.
-
Danny Auble authored
-
Danny Auble authored
-
Danny Auble authored
-
Tim Wickberg authored
-
Tim Wickberg authored
-
Tim Wickberg authored
And switch CHUNK macro -> chunk within the function.
-
Tim Wickberg authored
Add orig_len to data structure to avoid guessing uncompressed size. Have zlib use this as well to avoid xrealloc() calls. (Future improvement to zlib would avoid use of the temporary buffer + memcpy calls.) Note that this is handling each block independently. Stream mode would be better, switch to that in the future for additional performance gains.
-
Tim Wickberg authored
Also change sbcast info line to print type as int rather than as bool.
-
Tim Wickberg authored
Note one quirk - SLURM_COMPRESS env var must be set to a type. Setting to 'true' or '1' will not work, when it would have enabled zlib before.
-
Tim Wickberg authored
If no type given current default is zlib. 'lz4' and 'zlib' are the only two types supported currently. Print an error if unknown type given, but continue with compression disabled.
-
Tim Wickberg authored
Stub out lz4 functions, will need to be filled in later. sbcast/srun also need work to allow them to specify compression routine.
-
Danny Auble authored
-
Brian Christiansen authored
-
Morris Jette authored
When a node's MCDRAM configuration changes, the amount of High Bandwidth Memory (HBM) changes also. HBM is treated like a GRES that can be allocated to jobs, so we update each node's GRES value when its MCDRAM configuration changes.
-
- 22 Mar, 2016 1 commit
-
-
Morris Jette authored
-