1. 02 Apr, 2015 5 commits
  2. 01 Apr, 2015 20 commits
    • Morris Jette's avatar
      Address errors reported by CLANG · bceaa099
      Morris Jette authored
      _entity_add_data() in src/common/entity.c defined as "static" to avoid
          confusion with same function name in src/common/layouts_mgr.c
      _entity_get_node_walkfunc() in src/common/entity.c defined as "static"
      Removed redundant set of "rc" variable _layouts_autoupdate_layout_if_allowed(),
          src/common/layouts_mgr.c
      Initialized variable "rc" in _layouts_read_state(), src/common/layouts_mgr.c
          in case the named file does not exist (stat() returns an error)
      The CLANG tool reported a possible error in _autoupdate_layout_tree(),
          src/common/layouts_mgr.c in the event that xtree_node_get_data()
          returns a NULL value and that pointer is used later in the function for
          "cnode->entity". I added "return 1" if xtree_node_get_data() returns NULL,
          but am not yet sure if that is the correct action.
      The CLANG tool reported possible errors related to a NULL value pointer in
          s_p_parse_pair_with_op(), src/common/parse_config.c which have been fixed.
      bceaa099
    • Matthieu Hautreux's avatar
    • Matthieu Hautreux's avatar
    • Matthieu Hautreux's avatar
    • Matthieu Hautreux's avatar
      [PATCH 15/17] hostlist: set MAX_PREFIX_CNT to 64k instead of 1k · 0def0d25
      Matthieu Hautreux authored
      Having MAX_PREFIX_CNT to 1k limits the number of elements that can be
      expressed using expressions like node[1-1025].ib[0] to 1024.
      
      As 64k is the max number of nodes that can be handled by Slurm, increase
      that value to 64k in order to be able to express at least that number of
      elements using a[x]b[y] when y is the same value for all the elements.
      
      Note: also required by the layouts parsing logic when parsing float values
      in expressions like : 'Entity=node[1-64000] FloatKey=[1-64000].[0]'
      0def0d25
    • Matthieu Hautreux's avatar
      [PATCH 14/17] layouts: add autoupdate of entities KVs based on inheritance relationships · 0df634ae
      Matthieu Hautreux authored
      layouts plugins inheritances among entities KVs like KEYSPEC_UPDATE_CHILDREN_SUM
      (see layouts_mgr.h) are now fully managed by the layouts manager.
      
      An extra third stage is added while loading the configuration and state
      informations during startup in order to automatically update the layouts
      entities KVs based on the relationships defined in the plugin among the keys.
      The third stage is executed right after the second one that is in charge of
      establishing the relations among the entities.
      
      When an update is performed using the API laouts_update_layout call, the
      autoupdate logic is also applied.
      
      Note that a layout plugin must explicitely ask for autoupdate support by the
      layouts manager using the autoupdate field of its layouts_plugin_spec_t internal
      structure 'plugin_spec'.
      
      layouts API calls using the push/pull logic also leverage the autoupdate
      mechanism.
      
      Note that for now, the autoupdate is "full" whenever it is called.
      Improvements should be considered in the future to improve the performance
      of the logic when possible, especially in push/pull contexts.
      0df634ae
    • Matthieu Hautreux's avatar
      [PATCH 13/17] layouts: extend layouts entity management API · 18b56355
      Matthieu Hautreux authored
      > layouts_entity_{set,get,setpush,pullget}_kv[_ref] calls
      
      These new calls enable to perform the get/set operations on the key/value pairs
      of a particular entity in a defined layout. The push/pull operations enable to
      call for the automatic update of the linked entities in the layouts (i.e. parent)
      for push and for the automatic update of the requested entity using the linked
      entities information for pull.
      
      Note that 2 variations are proposed for each call, differentiated by the _ref
      suffix. When _ref is suffixed, then the calls only deal with references and no
      copy of the data is performed, only data references (pointers) are set/get. The
      _ref suffixed calls should then be considered as more efficient but can lead to
      issues if the pointers are modified after the get operations occur but before
      their real use. For that reason, layouts_{lock,unlock} calls would certainly
      appear in a next commit as well as _nolock suffixed calls in order to better
      handle those scenarios.
      
      Note that the ...pull/push... calls are just stubbed in that commit. The logic
      will be added later.
      
      > layouts_entity_get_mkv[_ref] calls
      
      These new calls enable to perform the get operations on a set of key/value pairs
      of a particular entity in a defined layout. (mkv stands for multiple k/v pairs)
      
      When _ref is suffixed, then the calls only deal with references and no copy of
      the data is performed, only data references (pointers) are got. The _ref suffixed
      calls should then be considered as more efficient but can lead to issues if the
      pointers are modified after the get operations occur but before their real use.
      
      For that reason, layouts_{lock,unlock} calls would certainly appear in a next
      commit as well as _nolock suffixed calls in order to better handle these
      scenarios.
      18b56355
    • Matthieu Hautreux's avatar
      [PATCH 12/17] layouts: refactor the entity_node and tree layout logic · f3b7e017
      Matthieu Hautreux authored
      The previous logic directly associated xtree_node_t elements with entities.
      The list of nodes associated to an entity was thus directly made of xtree_node_t
      elements each one pointing directly to the entity.
      
      An entity being layout agnostic, that is to say not bound to a particular layout
      it was not possible to look for the xtree_node_t element associated with a
      particular layout in an entity. This was due to the fact that the entity_node
      element was not used as an intermediate level to link the entities and their
      different xtree_node_t elements. Indeed, this is the role of the entity_node to
      associate a position in a particular layout with an entity.
      
      The new logic ensures that when building a layout relational structure :
      
      - (1) an entity node associated with the targeted layout is created for the
        entity and references both the layout and the entity it is associated too.
      - (2) the entity node is the element that is added to the (tree) relational
        structure of the layout resulting in the creation of a relational node
        (xtree_node_t) that will reference it directly.
      - (3) the created relational node (xtree_node_t) is referenced in the associated
        entity node in order to provide a way to go back to the position in the
        relational structure from the entity node, thus enabling to search the
        neighborhood of an entity for a particular layout only knowing the entity
        and the layout to look at.
      
      The packing logic was modified to take into account the new level of indirection
      (the entity_node element) when walking the relational structure of a particular
      layout. The debug dump logic was modified too to reflect the changes.
      f3b7e017
    • Matthieu Hautreux's avatar
    • Matthieu Hautreux's avatar
      [PATCH 10/17] layouts: add read-only keyspec support · 0cccbefa
      Matthieu Hautreux authored
      Layout entities Key/value pairs were automatically merged during updates with
      the previous logic. It is still the case by default but keyspec can now
      also be declared as RDONLY in layouts plugins in order to avoid that.
      
      Note that RDONLY keys can still be written by the plugins implementations when
      necessary. It only affects the automerge logic avoiding to let administrators
      update them with 'scontrol layouts update ...' commands directly.
      0cccbefa
    • Matthieu Hautreux's avatar
      [PATCH 09/17] layouts: remove the slurm_ prefix from layouts API functions · a302f763
      Matthieu Hautreux authored
      it does not provide any information and make things longer than necessary.
      a302f763
    • Matthieu Hautreux's avatar
      [PATCH 06/17] layouts: add layouts_update_layout() logic and factorize conf parsing calls · 694d7840
      Matthieu Hautreux authored
      layouts_update_layout(...) now enables to process an incoming buffer containing
      a contiguous set of packed strings and push that information into a particular
      layout. Only the entities key/value pairs are updated for now.
      
      layouts_p_update_done is a new callback that will be triggered with an array of
      entities pointers when an update is performed on a particular layout. This
      callback will help layout plugins to perform various consistency checks and/or
      update any internal states that should be updated based on the modified entities
      information.
      
      No information concerning the udated key/value pairs are provided for now.
      For finer-grained controls over what has changed, the plugins will have to use
      both the parse_entity callback and the update_done callback to handle what has
      changed and what should be rebuilt after the update for now.
      694d7840
    • Matthieu Hautreux's avatar
    • Matthieu Hautreux's avatar
      [PATCH 06/17] layouts: add layouts_update_layout() logic and factorize conf parsing calls · 5ff4f187
      Matthieu Hautreux authored
      layouts_update_layout(...) now enables to process an incoming buffer containing
      a contiguous set of packed strings and push that information into a particular
      layout. Only the entities key/value pairs are updated for now.
      
      layouts_p_update_done is a new callback that will be triggered with an array of
      entities pointers when an update is performed on a particular layout. This
      callback will help layout plugins to perform various consistency checks and/or
      update any internal states that should be updated based on the modified entities
      information.
      
      No information concerning the udated key/value pairs are provided for now.
      For finer-grained controls over what has changed, the plugins will have to use
      both the parse_entity callback and the update_done callback to handle what has
      changed and what should be rebuilt after the update for now.
      5ff4f187
    • Matthieu Hautreux's avatar
      [PATCH 05/17] layouts: add support for key[+-*/]=value when loading configurations · 89d54f02
      Matthieu Hautreux authored
      see 024c9fab036c8e6110b9276367dcf875a81876ef
      89d54f02
    • Matthieu Hautreux's avatar
      [PATCH 04/17] config parser: add several modifications required by layouts management · e1d2a0c1
      Matthieu Hautreux authored
      > add a new 'operator' field in s_p_values_t
      
      Provide the capability to manage key[+-*/]=val entries in the parsing logic.
      A new operator field now enables to know which operation is associated to
      a pair of key/value instead of the classical set. Introduced operators are
      provided by the slurm_parser_operator_t type with the following value :
      
       * S_P_OPERATOR_ADD
       * S_P_OPERATOR_SUB
       * S_P_OPERATOR_MUL
       * S_P_OPERATOR_DIV
       * S_P_OPERATOR_MIN
       * S_P_OPERATOR_MAX
       * S_P_OPERATOR_AVG
      
      It is the caller of the parsing logic that is responsible of using that
      information in order to act as requested. The parsing logic do not do any
      computation based on the extracted operator.
      
      S_P_OPERATOR_{MIN,MAX,AVG} are internal operators not yet bound to particular
      char in the parsing logic of 'key[operator]=val' entries. They are only used
      for layouts internal logic.
      
      Why:
      
      Introducing the layouts framework, the parsing logic of SLURM was extended to
      manage efficiently conf elements expansions. Layouts heavily rely on the SLURM
      parser to tansform textual configurations into internal representations.
      
      Looking at a way to provide live update of Layouts information, it was
      interesting to reuse the parsing logic of SLURM instead of trying to do the
      same thing somewhere else.
      
      Introducing a "key[+-*/]=val" feature was thus mandatory in order to apply
      numerical modifications of the layouts internal states on the controller side
      instead of trying to compute that on the client side and then send the updated
      information, loosing the capability to perform operations atomicallly.
      
      > add S_P_{FLOAT,DOUBLE,LONG_DOUBLE} and related s_p_get_* functions
      
      > enable true/false string parameter with S_P_BOOLEAN
      
      Layouts logic was using true/false instead of yes/no|0/1|up/down
      that were supported with S_P_BOOLEAN. Add true/false instead of reverting
      to one of the already available parameters as it was strange to not have them.
      
      > add "_." as valid characters in key names
      
      > add s_p_parse_buffer to parse configuration lines from a Slurm Buffer
      
      s_p_parse_file is a convenient way to handle the parsing of Slurm configuration
      file. It however only works with traditional file and can not be used to process
      a bunch of configuration strings.
      
      s_p_parse_buffer provides the capability to parse a bunch of contiguous strings
      packed into a Buf type, stopping the processing at the first unsuccessfull
      string extraction from the buffer.
      e1d2a0c1
    • Matthieu Hautreux's avatar
      [PATCH 03/17] layouts: initial layouts state recovery logic in slurmctld (entities data only) · d3529c4f
      Matthieu Hautreux authored
      Add a stage 1.1 in the loading of the layouts in order to recover the updated
      entities prior to building the relationships among them (stage 2) using the
      states saved in Slurm state save location for the layouts to load.
      
      Information concerning the relations among entities provided by the 'Enclosed'
      conf pragma are not taken into account for now when reading from layouts state
      files. No layout plugins callbacks are called when doing that for now.
      d3529c4f
    • Matthieu Hautreux's avatar
      [PATCH 02/17] layouts: initial layout packing/state_save logic addition · a3dc3818
      Matthieu Hautreux authored
      Add a function to help packing a particular layout into a Buffer as a set of
      strings terminated by a "\0" string and representing the current expanded
      form of the layout configuration as it could be described in a standard conf
      file.
      
      Use that function to provide a state save logic generating expanded conf files
      in the state save location, one file per generated layout. Entities not linked
      to a particular layout at save state time will not be saved as considered as
      no longer necessary.
      
      Note that the tree structure of the layouts are also saved in the packed
      information as well as in the generated expanded file.
      
      Note: fix a bug when freeing layouts in slurm_layouts_fini along the way.
      a3dc3818
    • Matthieu Hautreux's avatar
      [PATCH 01/17] layouts_mgr: add shortkey field into layouts_keydef struct · 48893f8f
      Matthieu Hautreux authored
      The layouts_keydef struct only stored the associated key name prefixed by
      the type of layouts (or mgr) inorder to have a unique identification of
      all the keydefs in the central hash table.
      
      To help accessing the original key name as specified in the layout conf
      file, the shortkey field now stores a copy of the original key. This is
      a short increase in the memory footprint of the layouts_keydef struct but
      as this structure is only allocated once per key in a layout, the global
      overhead is very small but provides an efficient way to retrieve the key
      name.
      48893f8f
    • Morris Jette's avatar
      Code clean up to eliminate CLANG warnings · 5fc68cae
      Morris Jette authored
      5fc68cae
  3. 31 Mar, 2015 10 commits
  4. 30 Mar, 2015 2 commits
  5. 27 Mar, 2015 3 commits
    • Morris Jette's avatar
      Verify plugin versions at load time · 45f671a8
      Morris Jette authored
      Verify that all plugin version numbers are identical to the component
      attempting to load them. Without this verification, the plugin can reference
      Slurm functions in the caller which differ (e.g. the underlying function's
      arguments could have changed between Slurm versions).
      NOTE: All plugins (except SPANK) must be built against the identical
      version of Slurm in order to be used by any Slurm command or daemon. This
      should eliminate some very difficult to diagnose problems due to use of old
      plugins.
      45f671a8
    • Brian Christiansen's avatar
      Change list_for_each priority functions to always return SUCCESS. · b2be6159
      Brian Christiansen authored
      Bug 1469
      Return values from void functions are unknown and were causing
      list_for_each to short ciruit processing of the job list.
      b2be6159
    • David Bigagli's avatar
      Increase the size of MinJobAge. · 711d815e
      David Bigagli authored
      711d815e