This can result in strange test results since there was no less
than zero checks before subtracting the total tests from failed
tests. In case of an internal exception we can just set all values
to zero. This will be handled specially as we do for timeout
errors.
You can now specify a limited list of subtests to run out of a
full auto-test using --sub-tests,-S. This option is limited in
that it is only meant to be used with a single autotest (since
it doesn't make much sense otherwise).
The subtest can be specified both with or without the file
extension.
Example usage:
./test-runner -A testAP -S failure_test,dhcp_test.py
This will only run the two subtests and exclude any other *.py
tests present in the test directory.
Code was added with commit 04487f575b which passes a radio object
to the Interface class constructor and stores it in the Interface
object. The radio class also stores each Interface object which
creates a circular reference and causes the Radio to stick around
long after the tests finishes.
I cannot see why the Interface needs to keep track of the Radio
object. None of the wpa_supplicant utilities use this so it has
been removed.
Add support for a WPA_SUPPLICANT section in hw.conf where
'radN=<config_path>' lines will only reserve radios and create
interfaces for the autotest to be able to start wpa_supplicant on them,
i.e. this prevents iwd or hostapd from being started on them but doesn't
start a wpa_supplicant instance by itself.
The host systems configuration directories for IWD/EAD were
being mounted in the virtual machine. This required that the
host create these directories before hand. Instead we can
just set up the system and IWD/EAD to use directories in /tmp
that we create when we start the VM. This avoids the need for
any host configuration.
Allow the "hwsim_medium=no" setting in hw.conf's SETUP section to
disable starting hwsim. It looks like the packets going through
userspace add enough latency that active scans don't work, probe
responses don't arrive within the "dwell time" or probe requests are not
ACKed on time. I've tried modifying tools/hwsim.c to respond with the
HWSIM_CMD_TX_INFO_FRAME cmd as the first thing after receiving a
HWSIM_CMD_FRAME and even skipping the queue in ell/genl.c by writing the
command synchronously, but neither helped enough to make the scans work.
This does not rule out that hwsim or the way our scans are done can be
fixed and that would obviously be better than what I did in this patch.
This extends test-runner to also use iwmon if --log is enabled.
For this case the iwmon log will be found inside each test
log directory.
A new option, --monitor <file> was added in case full logging isn't
desired (potentially for timing issues) but a iwmon log is needed.
Be aware that when --monitor is used test-runner will mount the
entire parent directory. test-runner itself will only write to the
file specified, but just know that the parent directory is available
as read-write inside the VM.
--log takes precedence over --monitor, meaning the iwmon log will
be written to <logdir>/<test>/iwmon instead of the file specified
with --monitor if both options are provided.
The virtual environment changed slightly adding two network adatpers
which are connected to the same backend so they can communicate with
each other (basically connected to a switch). The hostapd command
line was modified to allow no interfaces to be passed in which lets
us create zero radios but still specify a radius_config file.
This is just a more concise/pythonic way of doing function arguments.
Since Process/start_process have basically the same argument names
we can simplify and use **kwargs which will pass the named arguments
directly to Process(). This also allows us to add arguments to Process
without touching start_process if we need.
Slower systems may not be able to make some timeouts that tests
mandated. All timeouts were increased significantly to allow tests
to pass on slow systems.
Removed test-runner.c, and renamed py_runner to test-runner. Removed
tools/test-runner from .gitignore.
This was done as a separate commit to avoid a nasty diff between the
existing test runner, and the new python version
This patch completely re-writes test-runner in Python. This was done
because the existing C test-runner had some clunky work arounds and
maintaining or adding new features was starting to become a huge pain.
There were a few aspects of test-runner which continually had to
be dealt with when adding any new functionality:
* Argument parsing: Adding new arguments to test-runner wasn't so
bad, but if you wanted those arguments passed into the VM it
became a huge pain. Arguments needed to be parsed, then re-formatted
into the qemu command line, then re-parsed in a special order
(backwards) once in the VM. The burden for adding new arguments was
quite high so it was avoided (at least by me) at all costs.
* The separation between C and Python: The tests are all written in
python, but the executables, radios, and interfaces were all created
from C. The way we solved this was by encoding the require info as
environment variables, then parsing those from Python. It worked,
but it was, again, a huge pain.
* Process management: It started with all processes being launched
from C, but eventually tests required the ability to start IWD, or
kill hostapd ungracefully in order to test certain functionality.
Since the processes were tracked in C, Python had no way of
signalling that it killed a process and when it started one C had
no idea. This was mitigated (basically by killall), but it was
no where close to an elegant solution.
Re-writing test-runner in python solves all these problems and will
be much easier to maintain.
* Argument parsing: Now all arguments are forwarded automatically
to the VM. The ArgParse library takes care of parsing and each
argument is stored in a dictionary.
* Separation between C and Python: No more C, so no more separation.
* Process management: Python will now manage all processes. This
allows a test to kill, restart, or start a new process and not
have to remember the PID or to kill it after the test.
There are a few more important aspects of the python implementation
that should now be considered when writing new tests:
* The IWD constructor now has different default arugments. IWD
will always be started unless specified and the configuration
directory will always be /tmp
* Any non *.py file in the test directory will be copied to /tmp.
This avoids the need for 'tmpfs_extra_stuff' completely.
* ctrl_interface will automatically be appended to every hostapd
config. There is no need to include this in a config file from
now on.
* Test cleanup is extremely important. All tests get run in the
same interpreter now and the tests themselves are actually loaded
as python modules. This means e.g. if you somehow kept a reference
to IWD() any subsequent tests would not start since IWD is still
running.
* For debugging, the test context can be printed which shows running
processes, radios, and interfaces.
Three non-native python modules were used: PrettyTable, colored, and
pyroute2
$ pip3 install prettytable
$ pip3 install termcolor
$ pip3 install pyroute2
Besides being undefined behaviour, signed integer overflow can cause
unexpected comparison results. In the case of network_rank_compare(),
a connected network with rank INT_MAX would cause newly inserted
networks with negative rank to be inserted earlier in the ordered
network list. This is reflected in the GetOrderedMethods() DBus method
as can be seen in the following iwctl output:
[iwd]# station wlan0 get-networks
Network name Security Signal
----------------------------------------------------
BEOLAN 8021x **** }
BeoBlue psk *** } all unknown,
UI_Test_Network psk *** } hence assigned
deneb_2G psk *** } negative rank
BEOGUEST open **** }
> titan psk ****
Linksys05274_5GHz_dmt psk ****
Lyngby-4G-4 5GHz psk ****
If an application has a bug and hangs on SIGTERM this causes
test-runner to hang as well. This is obviously an issue with
the application in question, but test-runner should have a way
of continuing onto the next test rather than hanging.
Instead we can use WNOHANG and a sleep to allow applications
some amount of time to exit, and if they haven't use SIGKILL
instead as well as print an error. Similar to how
wait_for_socket works. The timeout is hard coded to 2 seconds
(100ms sleep + 20 iterations).
Previously iwmon was running per-test, which would jumble any subtests
together into the same log file making it hard to parse. Now create
a separate directory for each subtest and put the monitor log and
pcap there.
Using mac80211_hwsim can sometimes result in out of order messages
coming from the kernel. Since mac80211_hwsim immediately sends out
frames and the kernel keeps command responses in a separate queue,
bad scheduling can result in these messages being out of order.
In some cases we receive Auth/Assoc frames before the response to
our original CMD_CONNECT. This causes autotests to fail randomly,
some more often than others.
To fix this we can introduce a small delay into hwsim. Just a 1ms
delay makes the random failures disappear in the tests. This delay
is also makes hwsim more realistic since actual hardware will always
introduce some kind of delay when sending or receiving frames.
When running test-runner as non-root the environment variables
SUDO_GID/SUDO_UID were unset, causing atoi to segfault. This replaces
atoi with strtol, and checks the existance of SUDO_GID/SUDO_UID
before trying to turn it into an integer. This patch also allows
the uid/gid to be read from the user if running as non-root.
Note: running as non-root does require the users permissions to be
setup properly. Directories and files are created when running with
logging, so if the user running test-runner does not have these
permissions the creation of these files will fail.
The configuration value of iwd_config_dir was defaulting to /etc/iwd
which, in the context of test-runner, is probably not the best idea.
The system may have a main.conf file in /etc/iwd which could cause
tests to fail or behave unexpectedly.
In addition all tests which use iwd_config_dir set it to /tmp anyways.
Because of this, the new default value will be /tmp and no tests will
even need to bother setting this.
The configuration value itself is not being removed because it may be
useful to set arbitrary paths (e.g. /etc/iwd) for example when using
the shell functionality.
This key is special in hostapd, and was being treated as a normal hostapd
config file. This special radius config file needs to be kept unpaired from
any interfaces so now its passed in as a separate argument and appended to
the end of the hostapd execute command.
Tests which use a standalone RADIUS server may crash due to
the wiphy array not taking into account the 'radius_server'
key which is skipped during setup.
The goto was jumping to a label which freed the wiphy list which
had not yet been initialized. This also fixes another similar issue
if chdir fails (in this case tmpfs_extra_stuff would get freed
before being allocated).
Some test cases require (at least with recent hostapd versions) a
stand alone radius server. This is done using driver=none in the
hostapd config file. For this use case hostapd does not need any
radio since its not doing anything wireless related.
Now inside the hw.conf file, under the HOSTAPD group, you can
specify a config file as the value to 'radius_server' key. This
config file will be used without any associated radio when hostapd
is started.
These two processes are executed per-test so they can be passed the
test name and have the logs stored only in tests that actually need
them rather than at the top level.
After changing the valgrind log to --log-fd, it appears that
the log file just gets appended, where before it was overwritten.
This makes test print out all previous tests valgrind results.
Now after printing out the valgrind info we can erase the file
for the next test.
This was a mistake in the original implementation. Since test-runner
is always run from the tools directory using --log with no directory
specified would result in many log files being put into the iwd tree.
Instead we can make --log take a required argument so its more
obvious where all the logs are going to go.
localtime indexes month starting at zero so adding 1 gives us a folder
name with the correct month.
The year is also set as 'years since 1900', so we need to add 1900 to
the year to get the actual year.
Historically if you wanted to see output from a python test you needed
to specify -v pytests. This was also the case if IWD was started from
python.
Nearly every time I run test-runner I would specify "-v iwd,pytests"
only to get the IWD output on these specific tests.
Instead we can special case 'python3' (previously 'pytests') inside
execute_program so that turning on verbosity for 'iwd' also turns it
on for the python tests.
After the logging changes verbose IWD with valgrind did not show any IWD
output. This commit fixes this by checking the verbosity against log_name
rather than argv[0] since log_name has a special case for valgrind/iwd.
The valgrind logic in start_iwd was refactored to only use the --log-fd
option rather than using --log-file in addition to --log-fd.
All the processes verbose output is just written to stdout, and very
hard to parse after the test runs. Having test-runner write the
processes output to separate log files is much nicer to view after
the test runs.
A read/write file system is created which is separate and isolated
from the current FS mount (which mounts the whole host file system).
Doing this requires the user to create a folder somewhere to be used
as the mount point. This folder is where all the log files will end
up after test-runner runs.
Since test-runner must be run as root, care was taken to keep the
log files owned by the user which runs test-runner. The UID/GID
are passed to the VM, and any log files created are chown'ed back
to the user who ran test-runner.
execute_program was refactored, and both verbose and log arguments
were removed. The verbose argument was not really required since
we can do check_verbosity(argv[0]) internally. The 'log' flag was
only used along with --shell, and now the user can simply use
--log instead of dumping /tmp/iwd.log.
You can use this feature by specifying --log[=path] to test-runner.
If no path is provided the current directory will be used. Using
the --log flag will result in all processes being run with the
--verbose flag.
A new folder will be created under the --log path. The folder will
be of the name "run-<year>-<month>-<dat>-<PID>". Under this folder
you will find any global process logs. These are processes that
are only run once for ALL tests (hwsim, ifconfig, dbus etc.). There
will also be folders for specific tests that were run. Inside these
test folders will be logs of processes that are run per-test (iwd,
hostapd, python etc.).
