Parameter Framework (PF) test plan

Notes

The unit test code MAY be able to list implemented scenarios in this same BDD format in order to check scenario coverage.
All scenarios implicitly start by “GIVEN a parameter framework”.
If a scenario does not specify a config files, the following minimal files SHOULD be used:

Top level file (replace {structurePath} by its value):

<?xml version='1.0' encoding='UTF-8'?>
<ParameterFrameworkConfiguration SystemClassName='test'
                                 TuningAllowed='false' ServerPort='1'>
    <SubsystemPlugins/>
    <StructureDescriptionFileLocation Path='{structurePath}'/>
</ParameterFrameworkConfiguration>

Structure file:

<?xml version='1.0' encoding='UTF-8'?>
<SystemClass Name='test'>
    <Subsystem Name='test' Type='Virtual'>
        <ComponentLibrary/>
        <InstanceDefinition>
            <BooleanParameter Name="placeholder"/>
        </InstanceDefinition>
    </Subsystem>
</SystemClass>

Starting the Parameter Framework

set a logger
- [X] Scenario: Default logger
  - WHEN no logger is set
  - THEN start should succeed
- [X] Scenario: No logger
  - WHEN a nullptr logger is set
  - THEN start should succeed
- [X] Scenario: Logger should receive info and warnings
  - GIVEN config files that emit warnings
  - GIVEN a logger that store logs
  - WHEN the record logger is set
  - THEN start should succeed
  - AND_THEN the logger should have stored info and warning log
- [X] Scenario: Unset logger
  - GIVEN config files that emit warnings
  - GIVEN a logger that store logs
  - WHEN the record logger is set
  - AND_WHEN a nullptr logger is set
  - THEN start should succeed
  - AND_THEN the record logger should NOT have stored any info or warning log
set/get various properties before startup:
- “force no remote interface”
  - [ ] Scenario: Tuning OK
    - GIVEN config files with tuning enabled on a valid port
    - WHEN the remote interface is NOT forbiden
    - THEN start should succeed
  - [ ] Scenario: Invalid tunning
    - GIVEN config files with tuning enabled on an invalid port (0?)
    - WHEN the remote interface is NOT forbiden
    - THEN start should fail
  - [ ] Scenario: Forbiden invalid tunning
    - GIVEN config files with tuning enabled on an invalid port (0?)
    - WHEN the remote interface is forbiden
    - THEN start should succeed
- “failure on missing subsystem”
  - [ ] Scenario: Plugin OK
    - GIVEN config files with a valid plugin
    - WHEN the missing subsystem policy is set to failure
    - THEN start should succeed
  - [X] Scenario: Wrong plugin
    - GIVEN config files with a invalid plugin
    - WHEN the missing subsystem policy is set to failure
    - THEN start should fail
  - [ ] Scenario: Wrong plugin but fallback
    - GIVEN config files with a invalid plugin
    - WHEN the missing subsystem policy is set to success
    - THEN start should succeed
- “failure on failed settings load”
  - [X] Scenario: Settings OK
    - GIVEN config files with a valid settings file
    - WHEN settings load failure is requested to abort start
    - THEN start should succeed
  - [X] Scenario: Wrong settings
    - GIVEN config files with a invalid (non existant?) settings file
    - WHEN settings load failure is requested to abort start
    - THEN start should fail
  - [X] Scenario: Wrong settings but ignore
    - GIVEN config files with a invalid (non existant?) settings file
    - WHEN settings load failure is requested to abort start
    - THEN start should succeed
- “schema folder location” and “validate schema on start”
  - [ ] Scenario: Schema OK
    - GIVEN config files with correct default schema location
    - WHEN schema folder location is left to default
    - WHEN schema validation is enabled
    - THEN start should succeed
  - [ ] Scenario: Inexisting schemas
    - GIVEN config files with correct default schema location
    - WHEN schema folder location is set to an invalid location (/doesNotExist ?)
    - WHEN schema validation is enabled
    - THEN start should fail
  - [ ] Scenario: Inexisting ignored schemas
    - GIVEN config files with correct default schema location
    - WHEN schema folder location is set to an invalid location (/doesNotExist ?)
    - WHEN schema validation is disabled
    - THEN start should succeed
  - [ ] Scenario: Inexisting used schemas
    - GIVEN config files with custom schema location (./shemaCustomFolderName)
    - WHEN schema folder location is set to the custom shema location
    - WHEN schema validation is enabled
    - THEN start should succeed
  - [ ] Scenario: Schema OK, config KO
    - GIVEN incorect config files with correct default schema location
    - WHEN schema validation is enabled
    - THEN start should fail

Error cases/special cases

invalid configuration or a file can't be read:
- [X] Scenario: invalid top level config file
  - GIVEN an incorect (empty ? wrong balisa ?) config top level file.
  - THEN start should fail
- [X] Scenario: invalid structure
  - GIVEN an incorect (empty ? wrong balisa ?) structure file.
  - THEN start should fail
- [X] Scenario: invalid settings
  - GIVEN an incorect (empty ? wrong balisa ?) settings file.
  - THEN start should fail
plugins can't be found (unless silenced by the user)
- Done in “failure on missing subsystem”
usage of <xi:include .../>
- [ ] Scenario: Arbitrary xincluded node
  - GIVEN a top level file xincluding a file with the “SettingsConfiguration” balisa
  - THEN start should succeed
plugins:
- non-empty “Location” attribute
  - [ ] Scenario: Plugin location OK
    - GIVEN a top level file with a plugin folder location attribute
    - THEN start should succeed
- empty “Location” attribute
  - relative path
    - [ ] Scenario: Empty Location, standard plugin install
      - GIVEN config files with an empty location attribute
      - GIVEN a plugin installed in a dlopen standard path
      - GIVEN that “Location/Plugin[name]” is the plugin name
      - THEN start should succeed
    - [ ] Scenario: Empty Location, non standard plugin install
      - GIVEN config files with an empty location attribute
      - GIVEN a plugin NOT installed in a dlopen standard path
      - GIVEN that “Location/Plugin[name]” is the plugin name
      - THEN start should fail
  - absolute path
    - [ ] Scenario: Empty Location, plugin absolute path
      - GIVEN config files with an empty location attribute
      - GIVEN a plugin
      - GIVEN that “Location/Plugin[name]” is the plugin absolute path
      - THEN start should succeed
    - [ ] Scenario: Empty Location, plugin non existing path
      - GIVEN config files with an empty location attribute
      - GIVEN that “Location/Plugin[name]” is an non existing path (/libDoesNotExist.so?)
      - THEN start should fail

Logging

info and warning
- Partly covered by the “set a logger” section
- [ ] Scenario: Plugin info and warning log
  - GIVEN a plugin that log a unique string as info and warning
  - GIVEN a logger that store logs
  - WHEN the record logger is set
  - THEN start should succeed
  - AND_THEN the logger should have stored the plugin specific info and warning log

Basic Domains & Configurations management

[ ] Scenario: create and delete an empty domain
- GIVEN a started parameter framework without domains
- THEN creating a domain with an arbitrary name (“domain1”?) should succeed
- (AND_THEN the domain list should contain the domain name)
- THEN deleting the domain should succeed
[ ] Scenario: create and delete a configuration
- GIVEN a started parameter framework with a domain
- THEN creating a configuration in this domain with an arbitrary name (“config1”) should succeed
- (AND_THEN the domain configuration list should contain the configuration name)
- THEN deleting the configuration should succeed
- (AND_THEN the domain configuration list should not contain the configuration name)
[ ] Scenario: delete a domain with configurations
- GIVEN a started parameter framework with a domain and several configuration (3?)
- THEN deleting the domain should succeed
- (AND_THEN the domain list should not contain the domain name)
[ ] Scenario: delete all domains
- GIVEN a started parameter framework with several domains
- THEN deleting all domains should succeed
- (AND_THEN the domain list should be empty)
[ ] Scenario: rename a domain
- GIVEN a started parameter framework with a domain and several configuration (3?)
- THEN renaming the domain to an arbitrary name should succeed

Error cases/special cases

[ ] Scenario: create a domain with an already-existing name
- GIVEN a started parameter framework with a domain (arbitrary name)
- THEN creating a second domain with the same name than the first should fail
[ ] Scenario: create a configuration with an already-existing name
- GIVEN a started parameter framework with a domain containing one configuration (arbitrary name)
- THEN creating a second configuration in this domain with the same name than the fist should fail
[ ] delete a non-existent domain
- GIVEN a started parameter framework without domain
- THEN deleting an arbitrary named domain should fail
[ ] delete a non-existent configuration
- GIVEN a started parameter framework one domain
- THEN deleting in this domain a arbitrary named configuration should fail
[ ] list configuration of a unknown domain
- GIVEN a started parameter framework without domain
- THEN listing configuration of an arbitrary named domain should fail
[ ] Scenario: rename a domain to an already used name
- GIVEN a started parameter framework with two domains with arbitrary different names (“domain1” and “domain2”)
- THEN renaming first domain (“domain1”) to the name of the second one (“domain2”) should fail
[ ] Scenario: rename a configuration to an already used name
- GIVEN a started parameter framework with one domain containing two configuration with arbitrary different names (“config1” and “config2”)
- THEN renaming first configuration (“config1”) to the name of the second one (“config2”) should fail
[ ] Scenario: rename a domain to its own name
- GIVEN a started parameter framework one arbitrary named domain
- THEN renaming the domain to its own name should succeed
[ ] rename a configuration to its own name
- GIVEN a started parameter framework a domain and one configuration
- THEN renaming the configuration to its own name should succeed

Domain modification

set/get sequence awareness
add/list/remove elements to a domain
(split domain ?)
(listBelongingDomains ?)
(listAssociatedDomains ?)
(listAssociatedElements ?)
(listConflictingElements ?)
(listRogueElements ?)

Error cases/special cases

add (or remove) a non-existent element
add (or remove) a parameter to (or from) a non-existent domain

Configurations modification

set a configuration rule; get it back
- trivial rules, complex rules (fuzzing ?)
- clear a rule
save/restore a configuration
set/get element sequence
set/get a parameter for a given configuration
set a parameter belonging to a domain and save a configuration

Error cases

set a wrongly-formatted rule
set a rule using an unknown criterion (or unknown criterion value)
set/get a rule to a non-existent domain (or configuration)

Criteria

create a criterion
- exclusive
- inclusive
(list criteria)
set a criterion; get a criterion value
- exclusive criterion
- inclusive criterion
  - special case: “none” value

Error cases/special cases

duplicated criterion name (with the same type/with a different type)
duplicated criterion value
max supported number of criterion values

Settings Import/Export

import xml domains
- w/ or w/o settings
- from a file/from a string
- all/single
export xml domains
- w/ or w/o settings
- to a file/from a string
- all/single

Error case/special cases

invalid settings
- invalid xml (should we use a fuzzing framework ?)
- valid xml but not matching the XSD (ditto)
- valid but unusable data:
  - unknown criterion/criterion values
  - unknown parameters
unreadable/unwritable file
single domain import overwrite
- fail if overwrite not requested
- succeed if overwrite requested

Parameters

structures are correctly instantiated
- parameters in the “instance definition”
- component types instantiation
- parameter blocks
- bit blocks
- arrays
check parameter mappings are correctly applied
- at all levels possible (subsystem, component type, component instance, block, parameter...)
- context mappings, amends, instantiation mappings
set/get a parameter
parameter handles
test on parameter values
- booleans
- bit blocks
- numeric types ranges
- fixed-point tests
- string length

Parameter Framework (PF) test plan

Notes

Starting the Parameter Framework

Error cases/special cases

Logging

Basic Domains & Configurations management

Error cases/special cases

Domain modification

Error cases/special cases

Configurations modification

Error cases

Criteria

Error cases/special cases

Settings Import/Export

Error case/special cases

Parameters

Configuration application

Misc/to be classified

Error cases