Skip to content

Task for OSC 2.1+: How and where should the expected SUT behavior be specified?

Originates from

Here are my notes and some further thoughts from the Simulation group meeting on 2021-07-01 concerning the question (A) How and where should the expected ego behavior be specified?

In general I see three different alternatives to specify the expected SUT behavior (but maybe there are more):

1) Specify also the expected SUT behavior within the scenario itself

  • This would mean to have a complete specification of what should happen in the scenario
  • This was done in an example that I showed in the meeting where I refined and added actions for the SUT to the scenario.
  • But if doing so it is important to assure that after a scenario observation one is able to differentiate between the question
    • (a) whether the environmental objects (other road users, etc.) behaved according to the scenario specification and the question
    • (b) whether the SUT behaved correctly regarding its expected behavior.
  • Or in other words: We need to be able to distinguish between cases where the scenario did not happen and cases where the SUT has failed (or both).
  • Currently, to the best of my knowledge, such a differentiation cannot be explicitly done in the OSC2 language.
  • One could think that it might be sufficient to consider each constraint to the ego behavior as expected ego behavior and each other constraint as the actual scenario specification. However, for constraints where multiple actors are involved (e.g. distances or speed differences) it is not implicitly clear whether this is a requirement to the SUT or part of the scenario.

1b) Define a scenario without ego expectations to keep this re-usable and define a test scenario that extends the original scenario by SUT expectations

  • This is somehow just a variant or extension of 1) which allows to define different test scenarios for the same core scenario and hence for better re-usability.

2) Put everything regarding the expected SUT behavior into coverage statements

  • I made an example for that which I showed yesterday. Slides are available in Teams.
  • One feedback from the discussion was that this feature may not be expressive enough to also express more complex requirements than the example that I showed.
  • In particular when thinking about not just measuring some value in the scenario execution and comparing it with some threshold but cases where the expected SUT behavior depends on the constellation and behavior of the other objects in the scenario.
    • For example, imagine an ALKS+ system that is also able to change lanes in the same scenario as specified here.
    • Then it might depend on the occupancy of the lane beside the SUT whether it should perform a lange change as evasive maneuver or an emergency brake. On the first glance, this can hardly addressed solely by coverage statements but needs constructs from the actual scenario language to specify when and how a lane change should happen.
    • We should definitely try to make an example for that to make this more tangible. Any volunteers? :-)

3) Mixture of 1) and 2)

  • The third option just says that you might need both options 1) and 2).
  • In cases where you get along with coverage statements you use them
  • For cases where the expressive power of coverage is not sufficient, use elements of the scenario language to specify the expected SUT behavior and introduce some way of marking them as expected ego behavior
Edited by Matthias Bueker