Commit Graph

124 Commits

Author SHA1 Message Date
James Prestwood 41b8a32dfc test-runner: have start_iwd take a storage dir
Allow the storage directory (default /tmp/iwd) to be configured
just like the state directory. This is in order to support multiple
IWD instances which require separate storage directories for network
provisioning files.
2020-12-08 15:15:33 -06:00
James Prestwood e1e1c4edd1 test-runner: introduce network namespaces
Our simulated environment was really only meant to test air-to-air
communication by using mac80211_hwsim. Protocols like DHCP use IP
communication which starts to fall apart when using hwsim radios.
Mainly unicast sockets do not work since there is no underlying
network infrastructure.

In order to simulate a more realistic environment network namespaces
are introduced in this patch. This allows wireless phy's to be added
to a network namespace and unique IWD instances manage those phys.
This is done automatically when 'NameSpaces' entries are configured
in hw.conf:

[SETUP]
num_radios=2

[NameSpaces]
ns0=rad1,...

This will create a namespace named ns0, and add rad1 to that
namespace. rad1 will not appear as a phy in what's being called the
'root' namespace (the default namespace).

As far as a test is concerned you can create a new IWD() class and
pass the namespace in. This will start a new IWD instance in that
namespace:

ns0 = ctx.get_namespace('ns0')
wd_ns0 = IWD(start_iwd=True, namespace=ns0)

'wd_ns0' can now be used to interact with IWD in that namespace, just
like any other IWD class object.
2020-11-18 11:01:11 -06:00
James Prestwood a6808aa55a auto-t: introduce new autotest syntax
Sometimes improperly written tests can end up causing future tests
to fail. For faster debugging you can now add a '+' after a given
autotest which will start that test and run all tests which come
alphabetically after it (as if you are running a full autotest suite).

Example:

./test-runner -A testWPA+

This will run testWPA, testWPA2, testWPA2-no-CCMP, testWPA2-SHA256,
and testWPA2withMFP.
2020-11-16 17:05:45 -06:00
James Prestwood 423f9a4e60 auto-t: do away with -1 test results
This can result in strange test results since there was no less
than zero checks before subtracting the total tests from failed
tests. In case of an internal exception we can just set all values
to zero. This will be handled specially as we do for timeout
errors.
2020-11-16 17:05:45 -06:00
James Prestwood 1eb82d661f auto-t: catch exceptions in post_test 2020-11-16 17:05:45 -06:00
James Prestwood f88d45c9d3 test-runner: delay starting the shell until after pre_test
This makes --shell somewhat more useful by copying all the test
files and starting test processes before dumping the user into
a shell.
2020-11-16 17:05:45 -06:00
James Prestwood 6bf514c4c3 test-runner: store process stdout
If a blocking process is started store the output in case
the caller needs it. Output will be stored in Process.out.
2020-11-16 17:05:45 -06:00
James Prestwood 4bf8bf2396 test-runner: add option to specify subtests to run
You can now specify a limited list of subtests to run out of a
full auto-test using --sub-tests,-S. This option is limited in
that it is only meant to be used with a single autotest (since
it doesn't make much sense otherwise).

The subtest can be specified both with or without the file
extension.

Example usage:

./test-runner -A testAP -S failure_test,dhcp_test.py

This will only run the two subtests and exclude any other *.py
tests present in the test directory.
2020-11-16 17:05:45 -06:00
James Prestwood c098db1837 test-runner: fix Radio/Interface circular reference
Code was added with commit 04487f575b which passes a radio object
to the Interface class constructor and stores it in the Interface
object. The radio class also stores each Interface object which
creates a circular reference and causes the Radio to stick around
long after the tests finishes.

I cannot see why the Interface needs to keep track of the Radio
object. None of the wpa_supplicant utilities use this so it has
been removed.
2020-11-03 13:57:37 -06:00
James Prestwood 26c4b8b4fe test-runner: add switch for genl debugging
Providing 'iwd-genl' in the list of verbose options will enable
IWD_GENL_DEBUG.
2020-10-22 09:24:55 -05:00
James Prestwood 334b03478b test-runner: enable valgrind track origins
This gives a more exact location to where a memory problem
is happening. It also uses more RAM so the VM was upgraded to
256MB from 192MB.
2020-10-22 09:24:55 -05:00
Andrew Zaborowski 04487f575b test-runner: Reserve radios for wpa_supplicant
Add support for a WPA_SUPPLICANT section in  hw.conf where
'radN=<config_path>' lines will only reserve radios and create
interfaces for the autotest to be able to start wpa_supplicant on them,
i.e. this prevents iwd or hostapd from being started on them but doesn't
start a wpa_supplicant instance by itself.
2020-10-19 17:20:21 -05:00
James Prestwood 313d8dbbed auto-t: remove requirement for /var/lib/{ead,iwd}
The host systems configuration directories for IWD/EAD were
being mounted in the virtual machine. This required that the
host create these directories before hand. Instead we can
just set up the system and IWD/EAD to use directories in /tmp
that we create when we start the VM. This avoids the need for
any host configuration.
2020-10-14 13:03:17 -05:00
Andrew Zaborowski db038022e8 test-runner: Make hwsim medium optional
Allow the "hwsim_medium=no" setting in hw.conf's SETUP section to
disable starting hwsim.  It looks like the packets going through
userspace add enough latency that active scans don't work, probe
responses don't arrive within the "dwell time" or probe requests are not
ACKed on time.  I've tried modifying tools/hwsim.c to respond with the
HWSIM_CMD_TX_INFO_FRAME cmd as the first thing after receiving a
HWSIM_CMD_FRAME and even skipping the queue in ell/genl.c by writing the
command synchronously, but neither helped enough to make the scans work.
This does not rule out that hwsim or the way our scans are done can be
fixed and that would obviously be better than what I did in this patch.
2020-09-29 13:20:29 -05:00
Andrew Zaborowski b321db20ad test-runner: Add flags for DHCP and TLS verbose output
Add "iwd-tls" and "iwd-dhcp" flags for -v.  These should automatically
enable "-v iwd" output because of the substring matching.
2020-09-29 13:20:26 -05:00
Andrew Zaborowski 15c04bb395 test-runner: Enable --p2p when creating interfaces
Enable p2p for the hwsim radios so that we can test P2P.  This can be
made optional for the tests but it seems safe to enable always.
2020-09-29 13:20:24 -05:00
James Prestwood 1a02fdcefd test-runner: add iwmon options
This extends test-runner to also use iwmon if --log is enabled.
For this case the iwmon log will be found inside each test
log directory.

A new option, --monitor <file> was added in case full logging isn't
desired (potentially for timing issues) but a iwmon log is needed.
Be aware that when --monitor is used test-runner will mount the
entire parent directory. test-runner itself will only write to the
file specified, but just know that the parent directory is available
as read-write inside the VM.

--log takes precedence over --monitor, meaning the iwmon log will
be written to <logdir>/<test>/iwmon instead of the file specified
with --monitor if both options are provided.
2020-09-29 09:10:07 -05:00
James Prestwood 9897efe75c test-runner: allow EAD-based autotests
The virtual environment changed slightly adding two network adatpers
which are connected to the same backend so they can communicate with
each other (basically connected to a switch). The hostapd command
line was modified to allow no interfaces to be passed in which lets
us create zero radios but still specify a radius_config file.
2020-09-17 16:19:39 -05:00
James Prestwood 6b3d6f9d60 test-runner: open up dbus config for dbus-monitor
dbus-monitor was not able to eavesdrop on method calls without
some changed to the dbus config.
2020-09-16 14:35:38 -05:00
James Prestwood f3e2c4e5c3 test-runner: remove ofono/hardcoded IWD options 2020-09-16 14:30:58 -05:00
James Prestwood 2e7fb5ee50 test-runner: use **kwargs for start_process
This is just a more concise/pythonic way of doing function arguments.
Since Process/start_process have basically the same argument names
we can simplify and use **kwargs which will pass the named arguments
directly to Process(). This also allows us to add arguments to Process
without touching start_process if we need.
2020-09-15 16:45:52 -05:00
James Prestwood 561c64e7f8 test-runner: print results even if test run is canceled 2020-09-11 13:42:13 -05:00
James Prestwood 6e2aba3907 auto-t: Increase all test timeouts
Slower systems may not be able to make some timeouts that tests
mandated. All timeouts were increased significantly to allow tests
to pass on slow systems.
2020-09-10 17:59:49 -05:00
James Prestwood 17955fcf5a tools: post test-runner rewrite cleanup
Removed test-runner.c, and renamed py_runner to test-runner. Removed
tools/test-runner from .gitignore.

This was done as a separate commit to avoid a nasty diff between the
existing test runner, and the new python version
2020-09-10 17:59:49 -05:00