SDV System Test Creation

System tests refer to any SDV test created using the SDV Test Framework.

Test creation

Test location

  • <test_repository_root>/sample_tests
  • <test_repository_root>/e2e_tests
  • <test_repository_root>/long_running_tests
  • <test_repository_root>/performance_tests
  • <test_repository_root>/hardware

Test files

  • README.md: All tests must include a description of the test's purpose and how to run the test.

  • Test file: All tests follow a similar structure:

"""SDV Name Test"""

from sdv_test_fw.test_execution import sdv_base_test, sdv_test_runner

class SdvTypeNameTest(sdv_base_test.SdvBaseTestClass):

    def setup_class(self):
      # Setup code. Executed only once at the beginning of the test.
      super().setup_class()
      self.sdv_device1 = self.get_device('device1')
      self.sdv_device2 = self.get_device('device2')
      ...

    # Remove if not needed.
    def setup_test(self):
      super().setup_test()
      # Setup code. Executed before every test case.
      # Remove override if not needed.

    # Remove if not needed.
    def teardown_test(self):
      # Cleanup code. Executed after every test case.
      super().teardown_test()

    # Remove if not needed.
    def teardown_class(self):
      # Cleanup code. Executed once at the end of the test.
      super().teardown_class()

    def test_name_case1(self):
      # Test case step
      # Test case verification

    def test_name_case2(self):
      # Test case step
      # Test case verification

if __name__ == '__main__':
    # Start Test Execution Using SDV Test Framework
    sdv_test_runner.run()
  • Build file: Android.bp. The file structure is as follows:
python_test_host {
    name: "SdvTypeNameTest", // Should match the name of the test class.
    main: "sdv_type_name_test.py",
    srcs: [
        "sdv_type_name_test.py",
    ],
    data: [
        ":sdv_test_fw_device_configs",
    ],
    test_options: {
        unit_test: false,
    },
    defaults: [
        "sdv_test_fw_defaults",
    ],
    test_config_template: ":<DEFAULT_TEMPLATE_NAME>",
}

Naming Convention

To identify and find the different types of tests, tests must be created following a specific naming convention.

Sample Test

  • Filename: sdv_sample_<NAME>_test.py

  • Class name: SdvSampleNameTest

E2E Test

  • Filename: sdv_e2e_<NAME>_test.py

  • Class name: SdvE2ENameTest

Long-Running Test

  • Filename: sdv_long_running_<NAME>_test.py

  • Class name: SdvLongRunningNameTest

Performance Test

  • Filename: sdv_performance_<NAME>_test.py

  • Class name: SdvPerformanceNameTest

Hardware Test

  • Filename: sdv_hw_<NAME>_test.py

  • Class name: SdvHWNameTest

Code Guidelines

This section provides guidelines and best practices for writing SDV System Tests.

Python and Mobly

Familiarize yourself with Python style guide and Mobly best practices, and consider the following SDV-specific recommendations:

  • Avoid using Mobly imports directly except for assertions. The SDV Test Framework builds on it with a focus on SDV.

  • Assertions: Use Mobly assertions directly.

SDV Tests

The following sections outline specific guidelines and best practices for developing tests within the SDV Test Framework.

Setup and Cleanup

Setup and cleanup code must be outside the test cases. Teardown methods are called even if the test fails, to make proper device cleanup.

The location for setup and teardown code depends on the test's specific needs, even if the test is interrupted:

  1. To run only once at the beginning and end of the entire test, use setup_class and teardown_class. For example, get devices, set variable values or state that does not change between test cases, configure common device properties, or set flags.
def setup_class(self):
  super().setup_class()
  # setup code

def teardown_class(self):
  # teardown code
  super().teardown_class()
  1. To run between test cases, before and after each of them. For example, an interactive session or common service execution.
def setup_test(self):
  super().setup_test()
  # setup code

def teardown_test(self):
  # teardown code
  super().teardown_test()

Parameterized test cases

Use parameterized test cases when the steps are common across different test cases to avoid code repetition.

from absl.testing import parameterized

@parameterized.named_parameters(
  {
      'testcase_name': 'ab',
      'input1': 'a',
      'input2': 'b',
  },
  {
      'testcase_name': 'cd',
      'input1': 'c',
      'input2': 'd',
  },
  )
  def test_name(self, input1, input2):
    # test

The example creates two test cases test_name_ab and test_name_cd.

One test case for behavior verification

Test cases should be compact and focus on one specific behavior. If multiple behaviors share common preconditions or steps, consider splitting them. You can use setup_test or parameterization to minimize the amount of repetitive code.

Following this approach makes tests easier to read and debug because it clearly indicates which steps and conditions failed.

Example
test_verify_process():
  device.start_process()

  # precondition 1
  device.send_signal1()
  # verification signal1 received
  ...

  # precondition 2
  device.send_signal2()
  # verification signal2 received
  ...

  # precondition 3
  device.start_agent()
  # verification behavior
  ...

  device.kill_process()

test_setup_test():
  super().setup_test()
  device.start_process()

test_signal1():
  # precondition
  device.send_signal1()
  # verification signal1 received
  ...

test_signal2():
  # precondition
  device.send_signal2()
  # verification signal2 received
  ...

test_agent():
  # precondition
  device.start_agent()
  # verification behavior
  ...

teardown_test():
  device.kill_process()
  super().teardown_test()

Deterministic test behavior

Avoid adding conditionals in the test that branch its behavior. If a verification needs to be split, use two different test cases instead.

Don't use Exceptions

Tests and common helpers must use assertions instead of exceptions. This facilitates debugging and follows testing patterns.

Example
result = self.some_calculations()
if result is None:
  raise Exception("No result")

result = self.some_calculations()
self.get_test_validator().assert_is_not_none(result)
Example
if not self.device.is_subprocess_running(
  self.EXPECTED_PROCESS
):
  raise Exception("Process is not running")

self.get_test_validator().assert_true(
  self.device.is_subprocess_running(self.EXPECTED_PROCESS),
  "Process is not running"
)

Don't use sleep()

Avoid using sleep() because it increases test execution time and causes flakiness.

When the test requires waiting for an event or verification to continue, use the Waiting Methods provided in the framework instead.

Use waiting methods carefully because test execution remains blocked until the condition matches or the timeout is reached.

When you need to wait for a condition to complete in a test, ask the following questions:

  1. What is a reasonable timeout?

    If an event is expected within a specific timeframe, the timeout should match that expectation to make sure the test fails quickly. Reduce the timeout if necessary (the default is 30s).

  2. How expensive is the operation performed by the waiting method?

    Avoid calling expensive operations frequently. Increase the poll interval if necessary (the default is 0.5s).

Test cases with requirements

If a test has test cases with explicit requirements to work (for example, the target where it should run), you can skip them if they don't match the requirements:

def test_with_requirement():
  self.get_test_validator().skip_if(expr, reason)
  # Test case