- 01 Apr, 2015 17 commits
-
-
Matthieu Hautreux authored
-
Matthieu Hautreux authored
Having MAX_PREFIX_CNT to 1k limits the number of elements that can be expressed using expressions like node[1-1025].ib[0] to 1024. As 64k is the max number of nodes that can be handled by Slurm, increase that value to 64k in order to be able to express at least that number of elements using a[x]b[y] when y is the same value for all the elements. Note: also required by the layouts parsing logic when parsing float values in expressions like : 'Entity=node[1-64000] FloatKey=[1-64000].[0]'
-
Matthieu Hautreux authored
layouts plugins inheritances among entities KVs like KEYSPEC_UPDATE_CHILDREN_SUM (see layouts_mgr.h) are now fully managed by the layouts manager. An extra third stage is added while loading the configuration and state informations during startup in order to automatically update the layouts entities KVs based on the relationships defined in the plugin among the keys. The third stage is executed right after the second one that is in charge of establishing the relations among the entities. When an update is performed using the API laouts_update_layout call, the autoupdate logic is also applied. Note that a layout plugin must explicitely ask for autoupdate support by the layouts manager using the autoupdate field of its layouts_plugin_spec_t internal structure 'plugin_spec'. layouts API calls using the push/pull logic also leverage the autoupdate mechanism. Note that for now, the autoupdate is "full" whenever it is called. Improvements should be considered in the future to improve the performance of the logic when possible, especially in push/pull contexts.
-
Matthieu Hautreux authored
> layouts_entity_{set,get,setpush,pullget}_kv[_ref] calls These new calls enable to perform the get/set operations on the key/value pairs of a particular entity in a defined layout. The push/pull operations enable to call for the automatic update of the linked entities in the layouts (i.e. parent) for push and for the automatic update of the requested entity using the linked entities information for pull. Note that 2 variations are proposed for each call, differentiated by the _ref suffix. When _ref is suffixed, then the calls only deal with references and no copy of the data is performed, only data references (pointers) are set/get. The _ref suffixed calls should then be considered as more efficient but can lead to issues if the pointers are modified after the get operations occur but before their real use. For that reason, layouts_{lock,unlock} calls would certainly appear in a next commit as well as _nolock suffixed calls in order to better handle those scenarios. Note that the ...pull/push... calls are just stubbed in that commit. The logic will be added later. > layouts_entity_get_mkv[_ref] calls These new calls enable to perform the get operations on a set of key/value pairs of a particular entity in a defined layout. (mkv stands for multiple k/v pairs) When _ref is suffixed, then the calls only deal with references and no copy of the data is performed, only data references (pointers) are got. The _ref suffixed calls should then be considered as more efficient but can lead to issues if the pointers are modified after the get operations occur but before their real use. For that reason, layouts_{lock,unlock} calls would certainly appear in a next commit as well as _nolock suffixed calls in order to better handle these scenarios.
-
Matthieu Hautreux authored
The previous logic directly associated xtree_node_t elements with entities. The list of nodes associated to an entity was thus directly made of xtree_node_t elements each one pointing directly to the entity. An entity being layout agnostic, that is to say not bound to a particular layout it was not possible to look for the xtree_node_t element associated with a particular layout in an entity. This was due to the fact that the entity_node element was not used as an intermediate level to link the entities and their different xtree_node_t elements. Indeed, this is the role of the entity_node to associate a position in a particular layout with an entity. The new logic ensures that when building a layout relational structure : - (1) an entity node associated with the targeted layout is created for the entity and references both the layout and the entity it is associated too. - (2) the entity node is the element that is added to the (tree) relational structure of the layout resulting in the creation of a relational node (xtree_node_t) that will reference it directly. - (3) the created relational node (xtree_node_t) is referenced in the associated entity node in order to provide a way to go back to the position in the relational structure from the entity node, thus enabling to search the neighborhood of an entity for a particular layout only knowing the entity and the layout to look at. The packing logic was modified to take into account the new level of indirection (the entity_node element) when walking the relational structure of a particular layout. The debug dump logic was modified too to reflect the changes.
-
Matthieu Hautreux authored
-
Matthieu Hautreux authored
Layout entities Key/value pairs were automatically merged during updates with the previous logic. It is still the case by default but keyspec can now also be declared as RDONLY in layouts plugins in order to avoid that. Note that RDONLY keys can still be written by the plugins implementations when necessary. It only affects the automerge logic avoiding to let administrators update them with 'scontrol layouts update ...' commands directly.
-
Matthieu Hautreux authored
it does not provide any information and make things longer than necessary.
-
Matthieu Hautreux authored
layouts_update_layout(...) now enables to process an incoming buffer containing a contiguous set of packed strings and push that information into a particular layout. Only the entities key/value pairs are updated for now. layouts_p_update_done is a new callback that will be triggered with an array of entities pointers when an update is performed on a particular layout. This callback will help layout plugins to perform various consistency checks and/or update any internal states that should be updated based on the modified entities information. No information concerning the udated key/value pairs are provided for now. For finer-grained controls over what has changed, the plugins will have to use both the parse_entity callback and the update_done callback to handle what has changed and what should be rebuilt after the update for now.
-
Matthieu Hautreux authored
-
Matthieu Hautreux authored
layouts_update_layout(...) now enables to process an incoming buffer containing a contiguous set of packed strings and push that information into a particular layout. Only the entities key/value pairs are updated for now. layouts_p_update_done is a new callback that will be triggered with an array of entities pointers when an update is performed on a particular layout. This callback will help layout plugins to perform various consistency checks and/or update any internal states that should be updated based on the modified entities information. No information concerning the udated key/value pairs are provided for now. For finer-grained controls over what has changed, the plugins will have to use both the parse_entity callback and the update_done callback to handle what has changed and what should be rebuilt after the update for now.
-
Matthieu Hautreux authored
see 024c9fab036c8e6110b9276367dcf875a81876ef
-
Matthieu Hautreux authored
> add a new 'operator' field in s_p_values_t Provide the capability to manage key[+-*/]=val entries in the parsing logic. A new operator field now enables to know which operation is associated to a pair of key/value instead of the classical set. Introduced operators are provided by the slurm_parser_operator_t type with the following value : * S_P_OPERATOR_ADD * S_P_OPERATOR_SUB * S_P_OPERATOR_MUL * S_P_OPERATOR_DIV * S_P_OPERATOR_MIN * S_P_OPERATOR_MAX * S_P_OPERATOR_AVG It is the caller of the parsing logic that is responsible of using that information in order to act as requested. The parsing logic do not do any computation based on the extracted operator. S_P_OPERATOR_{MIN,MAX,AVG} are internal operators not yet bound to particular char in the parsing logic of 'key[operator]=val' entries. They are only used for layouts internal logic. Why: Introducing the layouts framework, the parsing logic of SLURM was extended to manage efficiently conf elements expansions. Layouts heavily rely on the SLURM parser to tansform textual configurations into internal representations. Looking at a way to provide live update of Layouts information, it was interesting to reuse the parsing logic of SLURM instead of trying to do the same thing somewhere else. Introducing a "key[+-*/]=val" feature was thus mandatory in order to apply numerical modifications of the layouts internal states on the controller side instead of trying to compute that on the client side and then send the updated information, loosing the capability to perform operations atomicallly. > add S_P_{FLOAT,DOUBLE,LONG_DOUBLE} and related s_p_get_* functions > enable true/false string parameter with S_P_BOOLEAN Layouts logic was using true/false instead of yes/no|0/1|up/down that were supported with S_P_BOOLEAN. Add true/false instead of reverting to one of the already available parameters as it was strange to not have them. > add "_." as valid characters in key names > add s_p_parse_buffer to parse configuration lines from a Slurm Buffer s_p_parse_file is a convenient way to handle the parsing of Slurm configuration file. It however only works with traditional file and can not be used to process a bunch of configuration strings. s_p_parse_buffer provides the capability to parse a bunch of contiguous strings packed into a Buf type, stopping the processing at the first unsuccessfull string extraction from the buffer.
-
Matthieu Hautreux authored
Add a stage 1.1 in the loading of the layouts in order to recover the updated entities prior to building the relationships among them (stage 2) using the states saved in Slurm state save location for the layouts to load. Information concerning the relations among entities provided by the 'Enclosed' conf pragma are not taken into account for now when reading from layouts state files. No layout plugins callbacks are called when doing that for now.
-
Matthieu Hautreux authored
Add a function to help packing a particular layout into a Buffer as a set of strings terminated by a "\0" string and representing the current expanded form of the layout configuration as it could be described in a standard conf file. Use that function to provide a state save logic generating expanded conf files in the state save location, one file per generated layout. Entities not linked to a particular layout at save state time will not be saved as considered as no longer necessary. Note that the tree structure of the layouts are also saved in the packed information as well as in the generated expanded file. Note: fix a bug when freeing layouts in slurm_layouts_fini along the way.
-
Matthieu Hautreux authored
The layouts_keydef struct only stored the associated key name prefixed by the type of layouts (or mgr) inorder to have a unique identification of all the keydefs in the central hash table. To help accessing the original key name as specified in the layout conf file, the shortkey field now stores a copy of the original key. This is a short increase in the memory footprint of the layouts_keydef struct but as this structure is only allocated once per key in a layout, the global overhead is very small but provides an efficient way to retrieve the key name.
-
Morris Jette authored
-
- 31 Mar, 2015 10 commits
-
-
Morris Jette authored
-
Morris Jette authored
-
Morris Jette authored
-
Morris Jette authored
SPANK naming changes: For environment variables set using the spank_job_control_setenv() function, the values were available in the slurm_spank_job_prolog() and slurm_spank_job_epilog() functions using getenv where the name was given a prefix of "SPANK_". That prefix has been removed for consistency with the environment variables available in the Prolog and Epilog scripts. bug 1570
-
Morris Jette authored
No change in logic
-
Morris Jette authored
Requests by normal user to reset a job priority (even to lower it) will result in an error saying to change the job's nice value instead.
-
Morris Jette authored
-
Morris Jette authored
Conflicts: src/slurmctld/job_mgr.c src/slurmctld/job_scheduler.c
-
Morris Jette authored
A non-administrator change to job priority will not be persistent except for holding the job. User's wanting to change a job priority on a persistent basis should reset it's "nice" value. Without this change a user might lower a job's priority, but Slurm will not lower it more based upon fair-share changes.
-
Morris Jette authored
Increase the MAX_PACK_MEM_LEN define to avoid PMI2 failure when fencing with large amount of ranks (to 1GB).
-
- 30 Mar, 2015 2 commits
-
-
David Bigagli authored
-
Morris Jette authored
-
- 27 Mar, 2015 3 commits
-
-
Morris Jette authored
Verify that all plugin version numbers are identical to the component attempting to load them. Without this verification, the plugin can reference Slurm functions in the caller which differ (e.g. the underlying function's arguments could have changed between Slurm versions). NOTE: All plugins (except SPANK) must be built against the identical version of Slurm in order to be used by any Slurm command or daemon. This should eliminate some very difficult to diagnose problems due to use of old plugins.
-
Brian Christiansen authored
Bug 1469 Return values from void functions are unknown and were causing list_for_each to short ciruit processing of the job list.
-
David Bigagli authored
-
- 26 Mar, 2015 8 commits
-
-
Morris Jette authored
-
David Bigagli authored
-
Nathan Yee authored
This validates that sbcast works on a job step allocation, which is a subset of a job allocation.
-
Morris Jette authored
-
Morris Jette authored
This is a correction to patch 5a9277b1
-
Morris Jette authored
This is a correction to patch 5a9277b1
-
Morris Jette authored
Conflicts: doc/html/documentation.shtml
-
Morris Jette authored
Fix for misleading job submit failure errors sent to users. Previous error could indicate why specific nodes could not be used (e.g. too small memory) when other nodes could be used, but were not for another reason. bug 1537
-