Commit Graph

460 Commits

Author SHA1 Message Date
James Prestwood 9a15a46df5 test-runner: don't fatally exit on bad test configuration
If a test has no hw.conf file test-runner was fully exiting and not
running any additional tests. This shouldn't happen in practice
since all upstreamed tests should run, but if any locally created
tests existed like this, it would cause the entire test run to exit
early.

Instead raise an exception which bails out of only that test, and
allows the rest to continue.
2021-08-12 15:14:07 -05:00
James Prestwood 8203b1d542 test-runner: start IWD in developer mode
This is being done to utilize the Roam() developer method
2021-08-12 15:14:07 -05:00
James Prestwood e2e625fa18 test-runner: refactor process output code
The Process class requires the ability to write out any processes
output to stdout, logging, or an explicit file, as well as store
it inside python for processing by test utilities. To accomplish
this each process was given a temporary file to write to, and that
file had an IO watch set on it. Any data that was written was then
read, and re-written out to where it needed to go. This ended up
being very buggy and quite complex due to needing to mess with
read/write pointers inside the file.

Popen already creates pipes to stdout if told, and they are accessable
via the p.stdout. Its then as simple as setting an IO watch on that
pipe and keeping the same code for reading out new data and writing
it to any files we want. This greatly reduces the complexity.
2021-07-29 17:51:25 -05:00
James Prestwood 52189d99f6 test-runner: don't reset dbus object on reset()
Doing this is redundant since the dbus object is initialized
anyways when a new test starts.
2021-07-29 17:51:25 -05:00
James Prestwood b5c3fc1222 test-runner: increase RAM to 384MB
Occationally python will fatally terminate trying to load a test
using importlib with an out of memory exception. Increasing RAM
allows reliable exection of all tests.
2021-07-29 17:51:25 -05:00
James Prestwood 75b20ff1d0 test-runner: set msize for --monitor
This was added for logging but --monitor suffers the same warning
without setting msize as well.
2021-07-29 17:51:25 -05:00
James Prestwood ba5bc1f5b3 test-runner: remove TLS debugging artifact
When logging is enabled TLS debugging is turned on which creates
a PEM file during runtime. There is no way for IWD itself to clean
this up since its meant to be there for debugging.
2021-07-29 17:51:25 -05:00
James Prestwood 8d40c37469 test-runner: set msize to remove runtime warning
Newer QEMU version warn that msize is set too low and may result
in poor IO performance. The default is 8KiB which QEMU claims is
too low. Explicitly setting to 10KiB removes the warning:

qemu-system-x86_64: warning: 9p: degraded performance: a
reasonable high msize should be chosen on client/guest side
(chosen msize is <= 8192).
See https://wiki.qemu.org/Documentation/9psetup#msize for details.
2021-07-28 09:32:42 -05:00
Denis Kenzior 074ed35b3d test-runner: Quiet a warning about msize
We seem to be not specifying the msize for the root filesystem, which
results in this warning being printed:

emu-system-x86_64: warning: 9p: degraded performance: a reasonable high msize should be chosen on client/guest side (chosen msize is <= 8192). See https://wiki.qemu.org/Documentation/9psetup#msize for details.

There doesn't seem to be much performance difference in the end since
iwd does not process large files.
2021-05-25 18:42:23 -05:00
Denis Kenzior 8e68e73c43 auto-t: Do not remove valgrind.log
Right now the --valgrind option logs to a static file named
'valgrind.log'.  This means that for any test that run multiple
instances of iwd, output is lost for all invocations except the last.
Fix that by using a per-process log file and making sure that all log
files are printed to stdout when the test ends.

This approach isn't perfect since it is possible for the pid to be
reused, but better than the current behavior.
2021-05-24 14:31:04 -05:00
James Prestwood f9c2fa7bb2 test-runner: remove stale file after test
test-runner will print out if files were left behind after a
test which lets the developer know something was not cleaned
up. But in this case test-runner should also remove these files
so they are not left, and printed, for each subsequent test.
2021-05-10 10:12:48 -05:00
James Prestwood 751db56045 test-runner: increase RAM for valgrind
Certain tests like testAP spawn two IWD process in separate
namespaces. When --valrind is used this eats up quite a bit
of RAM and causes the VM to run out of memory and start
killing off processes.
2021-05-04 10:30:55 -05:00
James Prestwood 4085aceb4a test-runner: fix verbose output with --valgrind
Since using --valgrind actually runs IWD using the valgrind
process the --verbose flag would only work if 'valgrind' was
also specified. This was taken into account with is_verbose
but the actual logic enabling stdout did not use that helper.
This was due, in part, to logging since is_verbose will always
return true if --log is used. To fix this a new flag was added
to is_verbose which omits the --log check to handle this
specific case.
2021-04-29 12:56:51 -05:00
James Prestwood 8aac527e29 test-runner: fix process output truncation
There was a bug with process output where the last bit of data would
never make it into stdout or log files. This was due to the IO watch
being cleaned up when the process was killed and never allowing it
to finish writing any pending data.

Now the IO watch implementation has been moved out into its own
function (io_process) which is now used to write the final bits of
data out on process exit.
2021-04-22 13:25:23 -05:00
James Prestwood d11974c717 test-runner: add timeout for waiting for process to finish
If a process hangs on exit test-runner would sit indefinitely
waiting.
2021-04-22 13:25:23 -05:00
James Prestwood 709c86086f test-runner: fix process cleanup loop
The processes in the list ultimately get removed for each
kill() call. This causes strange behavior since the list is
being iterated and each iteration is removing items. Instead
iterate over a new temporary list so the actual process list
can be cleaned up.
2021-04-22 13:25:23 -05:00
Denis Kenzior 38a7c4a8dc test-runner: Don't bother enabling IWD_GENL_DEBUG
IWD_GENL_DEBUG is not generally useful anymore as it just prints a
hexdump of the raw data on the socket.  The messages are quite verbose
and spam test-runner logs for little utility.
2021-04-20 10:48:50 -05:00
James Prestwood 0756c301f3 hwsim: add Prefix match rule support
Hwsim rules now have a 'Prefix' property which will allow
matching frames based on their payload data.
2021-04-05 17:47:42 -05:00
James Prestwood 66b4723216 test-runner: fix duplicate process output
Process output was being duplicated when -v was used. This was
due to both stderr and stdout being appended to the write_fd list
as well as stderr being set to stdout in the Popen call.

To fix this only stdout should be appended to the write_fd list,
but then there comes a problem with closing the streams. stdout
cannot be closed, so instead it is special cased. A new
verbose boolean was added to Process which, if True, will
cause any output to be written to stdout explicitly.
2021-03-02 10:37:22 -06:00
James Prestwood c660feadf1 test-runner: stop/warn on processes running after test
This gives the dev some inidication their test did not clean up
correctly and running processes remain.
2021-02-26 10:09:56 -06:00
James Prestwood 670f063a30 test-runner: warn on files left after test completes 2021-02-26 10:09:56 -06:00
James Prestwood 2e55e00662 test-runner: start haveged in foreground
haveged was forking itself which then prevented test-runner from
cleaning it up.
2021-02-26 10:09:56 -06:00
James Prestwood 73b89555fa test-runner: fix Namespace class ref problems
The Namespace class was never being removed when tests finished.
This is fixed by unreffing the hwsim internal _radio object which
both cleans up the radio and allows the Namespace to be removed.
2021-02-26 10:09:56 -06:00
James Prestwood d8abc88e4c test-runner: move process cleanup into kill()
This moves all the de-init code into kill(), which fixes a few
reference issues causing processes to hang around longer than
desired. If the process terminates on its own and/or the last
reference is lost __del__ will kill the process and clean up.
2021-02-26 10:09:56 -06:00
James Prestwood 48d27c982d test-runner: reset namespaces on test completion 2021-02-26 10:09:56 -06:00
James Prestwood 8f33926a75 test-runner: use cleanup() for start_dbus
Use a cleanup routine to remove the dbus_address/dbus_cfg files
2021-02-26 10:09:56 -06:00
James Prestwood 11f0abebe6 test-runner: add better Hostapd cleanup
There were a few issues with the cleanup of Hostapd. First the
process was only being killed, which did not actually remove
the process from the list.

In addition, with EAP-SIM/AKA tests, hostapd created sim_db
unix socket files which it does not clean up.
2021-02-26 10:09:36 -06:00
James Prestwood bf9d2b6c52 test-runner: allow radio remove failure
This is somewhat of an open issue/TODO but for now this avoids
the exception caused by trying to remove a radio that has been
moved to a namespace. Once a radio is moved hwsim loses that phy
and can no longer interact with it. This causes the Destroy()
method call to fail.
2021-02-26 10:09:36 -06:00
James Prestwood 5e79cda7c6 test-runner: cleanup output files/others in Process
A cleanup parameter was added to __init__ which can be used
by processes which create any additional files or require more
a custom cleanup routine. Some additional house keeping was
done to make Process cleanup more robust.
2021-02-26 10:09:36 -06:00
James Prestwood 697b4ce82f test-runner: remove concept of 'multi_test'
Though multi-test processes seemed like a good idea in terms of
efficiency, the additional code/special cases was not worth it
for the only two multi-test processes (dbus/haveged). Intead this
concept was removed completely and TestContext/Namespaces will
now start all processes for each individual test. This also is
fair to all tests as a previous failed test could end up bleeding
into future tests.
2021-02-26 10:09:36 -06:00
James Prestwood 90b695cfdf test-runner: fix start_dbus and clean up config
start_dbus was not returning the Process class which was
expected by the caller. This resulted in the config not
being cleaned up.
2021-02-19 10:16:37 -06:00
James Prestwood 4682c22ece test-runner: add __str__ method to Process
Printing out processes was done manually but instead we can
make Process printing extendable by adding its own __str__
method. This now will print if the process is a multi-test
process as well.
2021-02-19 10:16:37 -06:00
James Prestwood c10bd14cde test-runner: use unique name for namespace output files
Output files in namespaces were not handled differently and would
end up overwriting/duplicating files from the root namespace. These
are now named /tmp/<process>-<namespace>-out.
2021-02-19 10:16:37 -06:00
James Prestwood 1791ef1dc7 test-runner: refactor Process class
The process class was quite hard to understand, and somewhat
fragile when multiple output options were needed like verbose
and logging, and in some cases even an additional output file.

To make things simpler we can have all processes output to a
temporary file (/tmp/<name>-out) and set a GLib IO watch on
that file. When the IO watch callback fires any additional
files (stdout, log files, output files) can be written to.

For wait=True processes we do not use an IO watch, but do
the same thing once the process exits, write to any additional
output files using the process output we already have.
2021-02-19 10:16:37 -06:00
James Prestwood 272a60a478 test-runner: require root user to run with --monitor
Since the monitor file requires touching the host file
system, root access is required.
2021-02-19 10:16:37 -06:00
James Prestwood c2d71fdaa5 test-runner: clear log dir before new log test run
The log dir was never being cleaned out prior to a new logging
test run. This could leave old stale files around. Note that this
will remove any past log files so if you need them, you want to
make a copy before running test-runner with --log again.
2021-02-19 10:16:37 -06:00
James Prestwood a0865c9010 test-runner: start dmesg process with start_process
This avoids the need to pass in the context explicitly.
2021-02-19 10:16:37 -06:00
Denis Kenzior 897ef661fe hwsim: Don't use l_genl_msg_new_sized with a conditional
l_genl API will now automatically grow message buffers as needed, so
there's no need to make the logic over-complicated
2021-02-08 15:37:21 -06:00
James Prestwood af8438878f test-runner: clean up dbus path file properly
Dbus should be started as a multi-test process from the
TestContext, which leaves the dbus address file around for
the full test run. For Namespaces dbus-daemon should be
closed when the Namespace closes.
2021-02-08 11:56:42 -06:00
James Prestwood a8768e354d test-runner: print error if kernel/qemu path is not found 2021-02-05 18:01:49 -06:00
James Prestwood f21e79c81c test-runner: add monitor to path 2021-02-05 18:01:49 -06:00
James Prestwood ed80dc68aa test-runner: clean up temporary files 2021-02-05 18:01:43 -06:00
James Prestwood b60f564bed test-runner: print Radio class path 2021-02-05 14:43:17 -06:00
James Prestwood b0e970ae38 test-runner: fix logging, verbose, and process output
There were some major problems related to logging and process
output. Tests which required output from start_process would
break if used with '--log/--verbose'. This is because we relied
on 'communicate' to retrieve the process output, but Popen does
not store process output when stdout/stderr are anything other
than PIPE.

Intead, in the case of logging or outfiles, we can simply read
from the file we just wrote to.

For an explicit --verbose application we must handle things
slightly different. A keyword argument was added to Process,
'need_out' which will ensure the process output is kept
regardless of --log or --verbose.

Now a user should be able to use --log/--verbose without any
tests failing.
2021-01-26 13:41:23 -06:00
James Prestwood cc345582ad test-runner: don't always print "verbose on for ..."
This shouldn't have been a dbg print, but rather a normal print
which will only be printed when '-d' is used.
2021-01-26 13:41:01 -06:00
James Prestwood 296f1d0d1c test-runner: fix verbose arguments as single string
The verbose arguments come in from the QEMU command line as a
single string. This should have been split into an array immediately
but was not. This led to issues like hostapd debug being enabled
when "-v hostapd_cli" was passed in.
2021-01-26 13:40:49 -06:00
Andrew Zaborowski a55f8864d2 testrunner: Fix /tmp files cleanup on error
Since the list of files copied to /tmp was part of the return value from
pre_test(), if an exception occurred inside pre_test(), "copied" would
be undefined and the post_test(ctx, copied) call in the finally clause
cause another exception:

raceback (most recent call last):
  File "/home/balrog/repos/iwd/tools/test-runner", line 1508, in <module>
    run_tests()
  File "/home/balrog/repos/iwd/tools/test-runner", line 1242, in run_tests
    run_auto_tests(config.ctx, args)
  File "/home/balrog/repos/iwd/tools/test-runner", line 1166, in run_auto_tests
    post_test(ctx, copied)
UnboundLocalError: local variable 'copied' referenced before assignment

(apart from not being able to clean up the files).  Pass "copied" as a
paremeter to pre_test instead.
2021-01-25 14:01:58 -06:00
James Prestwood 01c108938e test-runner: Use DBus for hwsim radios
Use the hwsim DBus API rather than command line. This both is
faster and more dynamic than doing so with the command line.
This also avoids tracking the radio ID since we can just hang
on to the radio Dbus object directly.
2020-12-17 20:13:56 -06:00
James Prestwood 5567caf64a hwsim: add iftype/cipher disabling through DBus
Update the Dbus API to allow disabling iftypes and ciphers
just as you can with the command line.
2020-12-17 20:11:25 -06:00
James Prestwood fe0879f7d1 test-runner: start hwsim always (optionally --no-register)
Always start hwsim but if tests do not need radio rules start
with the --no-register option.
2020-12-17 20:11:13 -06:00
James Prestwood e8031bab2a test-runner: add wait_for_dbus_service
Common API to wait for a DBus service to appear on the bus
2020-12-17 20:10:57 -06:00
James Prestwood 6a1853b01a hwsim: change radio Create() to take a dictionary
The Create() API was limited to only taking a Name and boolean
(for p2p enabling). The actual hwsim nl80211 API can take more
attributes than this (which are actually utilized when creating
from the command line). To get the DBus API up to the same
functionality the two arguments in Create were replaced with
a single dictionary. This allows for extending later if more
arguments are needed.
2020-12-17 20:10:33 -06:00
James Prestwood 799ab03f59 hwsim: check pending_create_msg before replying
In the NEW_RADIO callback hwsim was assuming that DBus had no
yet replied to the Create() method. In some cases the NEW_RADIO
event fires before the actual callback which will respond to
DBus. This causes a crash in the create callback.
2020-12-17 20:08:54 -06:00
James Prestwood 554a78fbf0 hwsim: add --no-register option
Starts hwsim but does not register to mac80211_hwsim. This is to
allow autotests to disable hwsim, while still having the ability
to create/destroy radios over DBus.
2020-12-17 20:08:49 -06:00
James Prestwood 4e4ba6769d test-runner: add iwd-acd to verbose options
This will turn on IWD_ACD_DEBUG
2020-12-08 15:16:28 -06:00
James Prestwood ada46fb2db test-runner: set processor count for VM
For better reliability the processor count is now set to qemu.
In cases of low CPU count (< 2) hosts the processor count is
limited to 1. Otherwise half of the host cores will be used for
the VM.
2020-12-08 15:15:51 -06:00
James Prestwood 41b8a32dfc test-runner: have start_iwd take a storage dir
Allow the storage directory (default /tmp/iwd) to be configured
just like the state directory. This is in order to support multiple
IWD instances which require separate storage directories for network
provisioning files.
2020-12-08 15:15:33 -06:00
James Prestwood e1e1c4edd1 test-runner: introduce network namespaces
Our simulated environment was really only meant to test air-to-air
communication by using mac80211_hwsim. Protocols like DHCP use IP
communication which starts to fall apart when using hwsim radios.
Mainly unicast sockets do not work since there is no underlying
network infrastructure.

In order to simulate a more realistic environment network namespaces
are introduced in this patch. This allows wireless phy's to be added
to a network namespace and unique IWD instances manage those phys.
This is done automatically when 'NameSpaces' entries are configured
in hw.conf:

[SETUP]
num_radios=2

[NameSpaces]
ns0=rad1,...

This will create a namespace named ns0, and add rad1 to that
namespace. rad1 will not appear as a phy in what's being called the
'root' namespace (the default namespace).

As far as a test is concerned you can create a new IWD() class and
pass the namespace in. This will start a new IWD instance in that
namespace:

ns0 = ctx.get_namespace('ns0')
wd_ns0 = IWD(start_iwd=True, namespace=ns0)

'wd_ns0' can now be used to interact with IWD in that namespace, just
like any other IWD class object.
2020-11-18 11:01:11 -06:00
James Prestwood a6808aa55a auto-t: introduce new autotest syntax
Sometimes improperly written tests can end up causing future tests
to fail. For faster debugging you can now add a '+' after a given
autotest which will start that test and run all tests which come
alphabetically after it (as if you are running a full autotest suite).

Example:

./test-runner -A testWPA+

This will run testWPA, testWPA2, testWPA2-no-CCMP, testWPA2-SHA256,
and testWPA2withMFP.
2020-11-16 17:05:45 -06:00
James Prestwood 423f9a4e60 auto-t: do away with -1 test results
This can result in strange test results since there was no less
than zero checks before subtracting the total tests from failed
tests. In case of an internal exception we can just set all values
to zero. This will be handled specially as we do for timeout
errors.
2020-11-16 17:05:45 -06:00
James Prestwood 1eb82d661f auto-t: catch exceptions in post_test 2020-11-16 17:05:45 -06:00
James Prestwood f88d45c9d3 test-runner: delay starting the shell until after pre_test
This makes --shell somewhat more useful by copying all the test
files and starting test processes before dumping the user into
a shell.
2020-11-16 17:05:45 -06:00
James Prestwood 6bf514c4c3 test-runner: store process stdout
If a blocking process is started store the output in case
the caller needs it. Output will be stored in Process.out.
2020-11-16 17:05:45 -06:00
James Prestwood 4bf8bf2396 test-runner: add option to specify subtests to run
You can now specify a limited list of subtests to run out of a
full auto-test using --sub-tests,-S. This option is limited in
that it is only meant to be used with a single autotest (since
it doesn't make much sense otherwise).

The subtest can be specified both with or without the file
extension.

Example usage:

./test-runner -A testAP -S failure_test,dhcp_test.py

This will only run the two subtests and exclude any other *.py
tests present in the test directory.
2020-11-16 17:05:45 -06:00
James Prestwood c098db1837 test-runner: fix Radio/Interface circular reference
Code was added with commit 04487f575b which passes a radio object
to the Interface class constructor and stores it in the Interface
object. The radio class also stores each Interface object which
creates a circular reference and causes the Radio to stick around
long after the tests finishes.

I cannot see why the Interface needs to keep track of the Radio
object. None of the wpa_supplicant utilities use this so it has
been removed.
2020-11-03 13:57:37 -06:00
James Prestwood 26c4b8b4fe test-runner: add switch for genl debugging
Providing 'iwd-genl' in the list of verbose options will enable
IWD_GENL_DEBUG.
2020-10-22 09:24:55 -05:00
James Prestwood 334b03478b test-runner: enable valgrind track origins
This gives a more exact location to where a memory problem
is happening. It also uses more RAM so the VM was upgraded to
256MB from 192MB.
2020-10-22 09:24:55 -05:00
Andrew Zaborowski 04487f575b test-runner: Reserve radios for wpa_supplicant
Add support for a WPA_SUPPLICANT section in  hw.conf where
'radN=<config_path>' lines will only reserve radios and create
interfaces for the autotest to be able to start wpa_supplicant on them,
i.e. this prevents iwd or hostapd from being started on them but doesn't
start a wpa_supplicant instance by itself.
2020-10-19 17:20:21 -05:00
James Prestwood 313d8dbbed auto-t: remove requirement for /var/lib/{ead,iwd}
The host systems configuration directories for IWD/EAD were
being mounted in the virtual machine. This required that the
host create these directories before hand. Instead we can
just set up the system and IWD/EAD to use directories in /tmp
that we create when we start the VM. This avoids the need for
any host configuration.
2020-10-14 13:03:17 -05:00
Andrew Zaborowski db038022e8 test-runner: Make hwsim medium optional
Allow the "hwsim_medium=no" setting in hw.conf's SETUP section to
disable starting hwsim.  It looks like the packets going through
userspace add enough latency that active scans don't work, probe
responses don't arrive within the "dwell time" or probe requests are not
ACKed on time.  I've tried modifying tools/hwsim.c to respond with the
HWSIM_CMD_TX_INFO_FRAME cmd as the first thing after receiving a
HWSIM_CMD_FRAME and even skipping the queue in ell/genl.c by writing the
command synchronously, but neither helped enough to make the scans work.
This does not rule out that hwsim or the way our scans are done can be
fixed and that would obviously be better than what I did in this patch.
2020-09-29 13:20:29 -05:00
Andrew Zaborowski b321db20ad test-runner: Add flags for DHCP and TLS verbose output
Add "iwd-tls" and "iwd-dhcp" flags for -v.  These should automatically
enable "-v iwd" output because of the substring matching.
2020-09-29 13:20:26 -05:00
Andrew Zaborowski 15c04bb395 test-runner: Enable --p2p when creating interfaces
Enable p2p for the hwsim radios so that we can test P2P.  This can be
made optional for the tests but it seems safe to enable always.
2020-09-29 13:20:24 -05:00
James Prestwood 1a02fdcefd test-runner: add iwmon options
This extends test-runner to also use iwmon if --log is enabled.
For this case the iwmon log will be found inside each test
log directory.

A new option, --monitor <file> was added in case full logging isn't
desired (potentially for timing issues) but a iwmon log is needed.
Be aware that when --monitor is used test-runner will mount the
entire parent directory. test-runner itself will only write to the
file specified, but just know that the parent directory is available
as read-write inside the VM.

--log takes precedence over --monitor, meaning the iwmon log will
be written to <logdir>/<test>/iwmon instead of the file specified
with --monitor if both options are provided.
2020-09-29 09:10:07 -05:00
James Prestwood 9897efe75c test-runner: allow EAD-based autotests
The virtual environment changed slightly adding two network adatpers
which are connected to the same backend so they can communicate with
each other (basically connected to a switch). The hostapd command
line was modified to allow no interfaces to be passed in which lets
us create zero radios but still specify a radius_config file.
2020-09-17 16:19:39 -05:00
James Prestwood 6b3d6f9d60 test-runner: open up dbus config for dbus-monitor
dbus-monitor was not able to eavesdrop on method calls without
some changed to the dbus config.
2020-09-16 14:35:38 -05:00
James Prestwood f3e2c4e5c3 test-runner: remove ofono/hardcoded IWD options 2020-09-16 14:30:58 -05:00
James Prestwood 2e7fb5ee50 test-runner: use **kwargs for start_process
This is just a more concise/pythonic way of doing function arguments.
Since Process/start_process have basically the same argument names
we can simplify and use **kwargs which will pass the named arguments
directly to Process(). This also allows us to add arguments to Process
without touching start_process if we need.
2020-09-15 16:45:52 -05:00
James Prestwood 9acb89d6f4 tools: remove old test-runner.c 2020-09-14 16:46:20 -05:00
James Prestwood 561c64e7f8 test-runner: print results even if test run is canceled 2020-09-11 13:42:13 -05:00
James Prestwood 6e2aba3907 auto-t: Increase all test timeouts
Slower systems may not be able to make some timeouts that tests
mandated. All timeouts were increased significantly to allow tests
to pass on slow systems.
2020-09-10 17:59:49 -05:00
James Prestwood 17955fcf5a tools: post test-runner rewrite cleanup
Removed test-runner.c, and renamed py_runner to test-runner. Removed
tools/test-runner from .gitignore.

This was done as a separate commit to avoid a nasty diff between the
existing test runner, and the new python version
2020-09-10 17:59:49 -05:00
James Prestwood 0772d4b61a auto-t: introduce pure python test-runner re-write
This patch completely re-writes test-runner in Python. This was done
because the existing C test-runner had some clunky work arounds and
maintaining or adding new features was starting to become a huge pain.

There were a few aspects of test-runner which continually had to
be dealt with when adding any new functionality:

 * Argument parsing: Adding new arguments to test-runner wasn't so
   bad, but if you wanted those arguments passed into the VM it
   became a huge pain. Arguments needed to be parsed, then re-formatted
   into the qemu command line, then re-parsed in a special order
   (backwards) once in the VM. The burden for adding new arguments was
   quite high so it was avoided (at least by me) at all costs.
 * The separation between C and Python: The tests are all written in
   python, but the executables, radios, and interfaces were all created
   from C. The way we solved this was by encoding the require info as
   environment variables, then parsing those from Python. It worked,
   but it was, again, a huge pain.
 * Process management: It started with all processes being launched
   from C, but eventually tests required the ability to start IWD, or
   kill hostapd ungracefully in order to test certain functionality.
   Since the processes were tracked in C, Python had no way of
   signalling that it killed a process and when it started one C had
   no idea. This was mitigated (basically by killall), but it was
   no where close to an elegant solution.

Re-writing test-runner in python solves all these problems and will
be much easier to maintain.

 * Argument parsing: Now all arguments are forwarded automatically
   to the VM. The ArgParse library takes care of parsing and each
   argument is stored in a dictionary.
 * Separation between C and Python: No more C, so no more separation.
 * Process management: Python will now manage all processes. This
   allows a test to kill, restart, or start a new process and not
   have to remember the PID or to kill it after the test.

There are a few more important aspects of the python implementation
that should now be considered when writing new tests:

 * The IWD constructor now has different default arugments. IWD
   will always be started unless specified and the configuration
   directory will always be /tmp
 * Any non *.py file in the test directory will be copied to /tmp.
   This avoids the need for 'tmpfs_extra_stuff' completely.
 * ctrl_interface will automatically be appended to every hostapd
   config. There is no need to include this in a config file from
   now on.
 * Test cleanup is extremely important. All tests get run in the
   same interpreter now and the tests themselves are actually loaded
   as python modules. This means e.g. if you somehow kept a reference
   to IWD() any subsequent tests would not start since IWD is still
   running.
 * For debugging, the test context can be printed which shows running
   processes, radios, and interfaces.

Three non-native python modules were used: PrettyTable, colored, and
pyroute2

$ pip3 install prettytable
$ pip3 install termcolor
$ pip3 install pyroute2
2020-09-10 17:59:49 -05:00
Alvin Šipraga bfd8cead95 treewide: guard compare functions against signed integer overflow
Besides being undefined behaviour, signed integer overflow can cause
unexpected comparison results. In the case of network_rank_compare(),
a connected network with rank INT_MAX would cause newly inserted
networks with negative rank to be inserted earlier in the ordered
network list. This is reflected in the GetOrderedMethods() DBus method
as can be seen in the following iwctl output:

  [iwd]# station wlan0 get-networks
    Network name                    Security  Signal
  ----------------------------------------------------
    BEOLAN                          8021x     **** }
    BeoBlue                         psk       ***  } all unknown,
    UI_Test_Network                 psk       ***  } hence assigned
    deneb_2G                        psk       ***  } negative rank
    BEOGUEST                        open      **** }
  > titan                           psk       ****
    Linksys05274_5GHz_dmt           psk       ****
    Lyngby-4G-4 5GHz                psk       ****
2020-08-14 10:55:30 -05:00
James Prestwood f7c036a801 test-runner: use WNOHANG option waiting for PID
If an application has a bug and hangs on SIGTERM this causes
test-runner to hang as well. This is obviously an issue with
the application in question, but test-runner should have a way
of continuing onto the next test rather than hanging.

Instead we can use WNOHANG and a sleep to allow applications
some amount of time to exit, and if they haven't use SIGKILL
instead as well as print an error. Similar to how
wait_for_socket works. The timeout is hard coded to 2 seconds
(100ms sleep + 20 iterations).
2020-06-22 10:52:32 -05:00
James Prestwood 32a55fb75c hwsim: add new 'Delay' property to Rules
This allows a matching frame to be delayed by some number of
milliseconds.
2020-06-08 15:13:02 -05:00
James Prestwood a0029f8b6e test-runner: run iwmon per-subtest
Previously iwmon was running per-test, which would jumble any subtests
together into the same log file making it hard to parse. Now create
a separate directory for each subtest and put the monitor log and
pcap there.
2020-05-01 23:01:26 -05:00
James Prestwood 7641782924 tools: delay hwsim sending frame
Using mac80211_hwsim can sometimes result in out of order messages
coming from the kernel. Since mac80211_hwsim immediately sends out
frames and the kernel keeps command responses in a separate queue,
bad scheduling can result in these messages being out of order.
In some cases we receive Auth/Assoc frames before the response to
our original CMD_CONNECT. This causes autotests to fail randomly,
some more often than others.

To fix this we can introduce a small delay into hwsim. Just a 1ms
delay makes the random failures disappear in the tests. This delay
is also makes hwsim more realistic since actual hardware will always
introduce some kind of delay when sending or receiving frames.
2020-05-01 20:38:15 -05:00
James Prestwood f37966a24d test-runner: create pcap when logging 2020-04-02 00:32:38 -05:00
James Prestwood 689a1ed858 test-runner: fix segfault running with --log non-root
When running test-runner as non-root the environment variables
SUDO_GID/SUDO_UID were unset, causing atoi to segfault. This replaces
atoi with strtol, and checks the existance of SUDO_GID/SUDO_UID
before trying to turn it into an integer. This patch also allows
the uid/gid to be read from the user if running as non-root.

Note: running as non-root does require the users permissions to be
setup properly. Directories and files are created when running with
logging, so if the user running test-runner does not have these
permissions the creation of these files will fail.
2020-04-02 00:31:36 -05:00
James Prestwood 7d5ce16f1a test-runner: make default iwd_config_dir=/tmp
The configuration value of iwd_config_dir was defaulting to /etc/iwd
which, in the context of test-runner, is probably not the best idea.
The system may have a main.conf file in /etc/iwd which could cause
tests to fail or behave unexpectedly.

In addition all tests which use iwd_config_dir set it to /tmp anyways.
Because of this, the new default value will be /tmp and no tests will
even need to bother setting this.

The configuration value itself is not being removed because it may be
useful to set arbitrary paths (e.g. /etc/iwd) for example when using
the shell functionality.
2020-03-21 00:10:32 -05:00
Andrew Zaborowski c41eb6b2b0 tools: Add utility to tx Probe Requests 2020-03-20 10:18:04 -05:00
James Prestwood 24235a2af8 test-runner: fix improper loading of radius_config
This key is special in hostapd, and was being treated as a normal hostapd
config file. This special radius config file needs to be kept unpaired from
any interfaces so now its passed in as a separate argument and appended to
the end of the hostapd execute command.
2020-03-13 23:52:26 -05:00
James Prestwood af5c77aeba test-runner: fix crash for tests using radius_server
Tests which use a standalone RADIUS server may crash due to
the wiphy array not taking into account the 'radius_server'
key which is skipped during setup.
2020-03-13 23:52:26 -05:00
James Prestwood 37886066d8 test-runner: fix crash if ofono is not installed
The goto was jumping to a label which freed the wiphy list which
had not yet been initialized. This also fixes another similar issue
if chdir fails (in this case tmpfs_extra_stuff would get freed
before being allocated).
2020-03-13 23:52:26 -05:00
Daniel Wagner 83ddd88e61 test-runner: Add missing 's' option to command line parser 2020-03-09 14:10:20 -05:00
Daniel Wagner 3195485e80 test-runner: Add missing space between qemus arguments 2020-03-09 14:10:00 -05:00
Denis Kenzior 0508879e2a hwsim: Use L_DBUS_PROPERTY_FLAG_AUTO_EMIT 2020-01-29 10:51:40 -06:00
Torstein Husebø 759dbdd37f treewide: fix typos 2020-01-21 16:03:28 -06:00
James Prestwood c93a4d71d6 test-runner: add special radius_server key
Some test cases require (at least with recent hostapd versions) a
stand alone radius server. This is done using driver=none in the
hostapd config file. For this use case hostapd does not need any
radio since its not doing anything wireless related.

Now inside the hw.conf file, under the HOSTAPD group, you can
specify a config file as the value to 'radius_server' key. This
config file will be used without any associated radio when hostapd
is started.
2020-01-06 11:18:48 -06:00
James Prestwood ede2c11d66 test-runner: print which file failed to fchown 2019-11-21 13:59:44 -06:00
James Prestwood 73f42c2db4 test-runner: move ofono/phonesim logs into test dir
These two processes are executed per-test so they can be passed the
test name and have the logs stored only in tests that actually need
them rather than at the top level.
2019-11-14 15:23:03 -06:00
James Prestwood 186a5195b5 test-runner: erase valgrind log after each test
After changing the valgrind log to --log-fd, it appears that
the log file just gets appended, where before it was overwritten.
This makes test print out all previous tests valgrind results.

Now after printing out the valgrind info we can erase the file
for the next test.
2019-11-13 16:35:41 -06:00
James Prestwood cb00ae1481 test-runner: make --log take a required argument
This was a mistake in the original implementation. Since test-runner
is always run from the tools directory using --log with no directory
specified would result in many log files being put into the iwd tree.

Instead we can make --log take a required argument so its more
obvious where all the logs are going to go.
2019-11-13 12:36:50 -06:00
James Prestwood 506f502900 test-runner: fix logging directory name
localtime indexes month starting at zero so adding 1 gives us a folder
name with the correct month.

The year is also set as 'years since 1900', so we need to add 1900 to
the year to get the actual year.
2019-11-06 17:04:00 -06:00
James Prestwood 2a2f60d86b test-runner: run iwmon if --log is used
Now that execute_program handles logging automatically its trivial
to add iwmon to the test and get monitor logs as well as normal
process output.
2019-11-06 17:04:00 -06:00
James Prestwood 5709a4afcf test-runner: special case python3 verbose option
Historically if you wanted to see output from a python test you needed
to specify -v pytests. This was also the case if IWD was started from
python.

Nearly every time I run test-runner I would specify "-v iwd,pytests"
only to get the IWD output on these specific tests.

Instead we can special case 'python3' (previously 'pytests') inside
execute_program so that turning on verbosity for 'iwd' also turns it
on for the python tests.
2019-11-06 15:16:40 -06:00
James Prestwood 8ec4a53adb test-runner: refactor/fix valgrind logging issue
After the logging changes verbose IWD with valgrind did not show any IWD
output. This commit fixes this by checking the verbosity against log_name
rather than argv[0] since log_name has a special case for valgrind/iwd.

The valgrind logic in start_iwd was refactored to only use the --log-fd
option rather than using --log-file in addition to --log-fd.
2019-11-06 15:15:20 -06:00
James Prestwood 6bdb154da5 test-runner: log process output to host system
All the processes verbose output is just written to stdout, and very
hard to parse after the test runs. Having test-runner write the
processes output to separate log files is much nicer to view after
the test runs.

A read/write file system is created which is separate and isolated
from the current FS mount (which mounts the whole host file system).
Doing this requires the user to create a folder somewhere to be used
as the mount point. This folder is where all the log files will end
up after test-runner runs.

Since test-runner must be run as root, care was taken to keep the
log files owned by the user which runs test-runner. The UID/GID
are passed to the VM, and any log files created are chown'ed back
to the user who ran test-runner.

execute_program was refactored, and both verbose and log arguments
were removed. The verbose argument was not really required since
we can do check_verbosity(argv[0]) internally. The 'log' flag was
only used along with --shell, and now the user can simply use
--log instead of dumping /tmp/iwd.log.

You can use this feature by specifying --log[=path] to test-runner.
If no path is provided the current directory will be used. Using
the --log flag will result in all processes being run with the
--verbose flag.

A new folder will be created under the --log path. The folder will
be of the name "run-<year>-<month>-<dat>-<PID>". Under this folder
you will find any global process logs. These are processes that
are only run once for ALL tests (hwsim, ifconfig, dbus etc.). There
will also be folders for specific tests that were run. Inside these
test folders will be logs of processes that are run per-test (iwd,
hostapd, python etc.).
2019-11-04 14:41:18 -06:00
Denis Kenzior fb76ca1f1c hwsim: Update to the new ell API 2019-10-28 15:02:30 -05:00
Marcel Holtmann 152b56a12a treewide: Move the Intel copyright forward to 2019 2019-10-25 00:43:08 +02:00
Tim Kourt 25b3cb77e1 hwsim: Fix potential memory leak
If msg has multiple RADIO NAME attributes, memory leak occurs.

Note that this doesn't happen in practice.
2019-10-23 17:58:51 -05:00
James Prestwood bf3b403e90 test-runner: fix double free issue on subtest queue
Coverity reported this as a leak, but the test queue is actually
getting freed later and does not need to be freed locally in add_path

This basically reverts c0863e5bc6
2019-10-23 12:28:51 -05:00
Tim Kourt 6fea4e6f4f t-runner: Check for errors after opening file 2019-10-22 21:57:31 -05:00
Tim Kourt e3551ab56b t-runner: Check for NULL before using strcmp 2019-10-22 21:57:31 -05:00
Tim Kourt e859e98869 t-runner: Fix possible assignment of NULL pointer 2019-10-22 21:57:31 -05:00
Tim Kourt 43efadcf92 t-runner: Fix out-of-bounds write 2019-10-22 21:57:31 -05:00
Tim Kourt c0863e5bc6 t-runner: Fix mem leak caused by python test names 2019-10-22 21:57:31 -05:00
Tim Kourt 269bf497c7 t-runner: Fix mem leak caused by unclosed dir 2019-10-22 21:57:31 -05:00
Tim Kourt 7dd32ac093 t-runner: Fix mem leak with sim keys
Use l_settings_get_value instead of l_settings_get_string to
prevent unnecessary memory duplication.
2019-10-22 21:57:31 -05:00
Denis Kenzior b6554ee41c hwsim: Invoke l_queue_remove prior to object deletion
While the current code is quite safe, the new ordering makes more
logical sense and doesn't confuse static analysis tools.
2019-10-17 13:03:08 -05:00
James Prestwood e39bb6bfa0 test-runner: allow all unit tests to be run in VM
The -U parameter only allowed for a list of unit tests to be run.
Most of the time for sanity checking you want to run all the unit
tests so this has been changed to take an optional argument.

Now, the -U flag (by itself) will run all unit tests. Running a
single or list of unit tests can still be achieved by:

--unit-tests=test-eapol,test-crypto
2019-10-09 14:33:08 -05:00
James Prestwood c4ecf2c5c6 tools: ios_convert: embed certs rather than using paths 2019-10-07 23:09:29 -05:00
James Prestwood 64ff5dda77 test-runner: sort test list alphabetically
This makes every full test run consistent. The test list was being
stored as a hashmap, which has no been changed to a queue so we can
insert each test sorted.
2019-09-24 10:07:40 -05:00
Marcel Holtmann 765bb92084 build: Add skeleton manual pages for all installed binaries 2019-09-22 21:57:47 +02:00
James Prestwood d8aadc49b2 tools: enable CONFIG_CRYPTO_ARC4 in kernel config 2019-09-11 11:40:39 -05:00
Marcel Holtmann 1104d69e29 build: Add StateDirectoryMode=0700 option to systemd service 2019-09-09 06:55:22 +02:00
Marcel Holtmann c1a9f71b39 build: Add script to start iwd.service via systemd-run 2019-09-08 22:10:28 +02:00
Marcel Holtmann 7e4a6e8a49 test-runner: Remove pointless NULL check before calling l_free 2019-09-08 18:14:13 +02:00
Marcel Holtmann 9d1f7ad6f5 test-runner: Provide CONFIGURATION_DIRECTORY and STATE_DIRECTORY 2019-09-08 18:12:51 +02:00
Marcel Holtmann 7347bd4098 test-runner: Use execvpe to provide environment context 2019-09-08 18:03:10 +02:00
James Prestwood 7acb910a26 test-runner: better define --shell behavior
Specifically, this defines the behavior when --shell is used when no
specific test is specified. In this case test-runner will assume the
'shell' test/sandbox should be used as the test environment as
running all autotests with --shell is not useful or feasable.
2019-08-28 15:55:57 -05:00
James Prestwood 202f65be69 tools/ios_convert: use join rather than manually looping 2019-08-27 10:14:08 -05:00
James Prestwood de2c808970 tools/ios_convert: add ServerDomainMask
Parses mobileconfig TLSTrustedServerNames
2019-08-27 10:14:08 -05:00
James Prestwood 96ce39c1c2 tools/ios_convert: add raw XML option
This option allows the script to be called with a raw XML file. This
is mostly useful for testing, but since its already implemented we
might as well include it.
2019-08-27 10:14:08 -05:00
James Prestwood ab0c8cae3c tools/ios_convert: parse DisplayedOperatorName
Some hotspot networks do not contain SSID_STR, which was required
for both naming the provisioning file as well as the 'Name' key.
The DisplayedOperatorName is a better option for this 'Name' key
and could also be used for the filename.

Now, DisplayedOperatorName is preferred, and if not found SSID_STR
is used.
2019-08-27 10:14:08 -05:00
James Prestwood b05689fc81 tools/ios_convert: Use tag instead of text for IsHotspot
The IsHotspot key just contains a tag after it (true/false), not
a tag with inner text.
2019-08-22 13:05:16 -05:00
James Prestwood 54a5791195 tools/ios_convert: add 'Name' to hotspot config
Also fixed a indentation issue for NAIRealms
2019-08-22 12:46:16 -05:00
James Prestwood 0e317ed9b1 test-runner: print if haveged fails to start
If haveged is not installed it attempts to start but exits with no
indication of what happened.
2019-08-20 14:23:35 -05:00
Denis Kenzior ed8a4ab931 treewide: Make global variables static 2019-08-15 15:06:59 -05:00
James Prestwood d2639e5d4e test-runner: log iwd output for --shell
This will allow the user to see the iwd output in /tmp/iwd.log.

execute_program was extended to take a 'log' flag. If true, this
will cause the programs output to be stored in /tmp/<name>.log.
This is only useful when using the --shell command as this file
will go away once the VM stops. The verbose flag always overrides
the logging functionality.

For now only iwd output is logged when using --shell.
2019-08-07 16:37:51 -05:00
James Prestwood da65acbd07 test-runner: allow booting directly into a shell
It is sometimes valuable to just boot into a shell in order to manually
test functionality. Since test-runner already is setup to run a minimal
kernel with all the necessary requirements for hostapd/iwd it made
sense to allow the user to do this.

If -s,--shell is passed into test runner, no python tests will be run.
The hw.conf file is still used to setup IWD and hostapd so once booted
into the shell you can still (manually) run the test (e.g. via iwctl).

This also works when using USB/PCI passthrough. This makes testing
out different kernel version with real hardware much quicker than
using the host kernel.
2019-08-07 16:37:51 -05:00
James Prestwood d8dac9a330 tools: add tool for iOS mobileconfig conversion
This tool will convert an iOS 'mobileconfig' file into the IWD
format. The tool only supports PEAP and TTLS networks, including
hotspots.

It will also parse out any certificate chains found in the
mobileconfig file, and verify they lead to a root CA found on the
system. If they do, this root CA will be used as the CACert in
the provisioning file.
2019-07-29 14:04:03 -05:00
James Prestwood edd4f2b2a2 test-runner: allow glob matching of tests
This allows a glob match of tests, e.g.

./test-runner -A testEAP-TTLS*

would run all TTLS based tests
2019-06-05 16:49:45 -05:00
Denis Kenzior 71ebc00fb7 tools: Update test-runner to the new genl api 2019-05-17 17:10:15 -05:00
Denis Kenzior dea23bede7 tools: Update hwsim to the new genl api 2019-05-17 17:10:13 -05:00
James Prestwood f038c11205 test-runner: allow disabling of iftype/cipher
Two new hardware configuration keys were added:

[radX]
iftype_disable=station,ap,adhoc,p2p_client,p2p_go,mesh_point
cipher_disable=wep40,wep104,tkip,ccmp,bip

Any of the above values are supported and can be disabled.
2019-05-13 16:38:23 -05:00
James Prestwood 2d173e5f24 test-runner: add a radio parameters structure for creation
Support is coming to configure radios with a specific set of interface
type and cipher support, so the input to create_hwsim_radio is better
suited to use a parameter structure rather than adding more parameters.
2019-05-13 16:38:23 -05:00
James Prestwood 2d51622c0b test-runner: allow radio list subset to be configured
The radio_confs key was parsed in a way that required all radios
to be specified in the list. This isnt optimal, as you may want to
specially configure a certain radio, while keeping all the others
default.

This change reworks some logic and allows any radio to be specially
configured on its own.
2019-05-13 16:38:23 -05:00
James Prestwood 0e31b33631 hwsim: allow specific radio iftypes/ciphers
mac80211_hwsim now allows setting supported iftypes/ciphers. This patch
enables this support in hwsim. Specific iftypes/ciphers can now be
disabled via the command line when creating a radio:

Disable iftypes:
--iftype-disable station,ap,adhoc,p2p_client,p2p_go,mesh_point

Disable cipher types:
--cipher-disable ccmp,tkip,wep
2019-05-13 16:38:23 -05:00
James Prestwood 79dd04c13a test-runner: in native HW mode, check for AP capabilities
The test should be aborted if there are not enough radios that support
AP mode. The iftype attribute will now be parsed during the wiphy dump
and a flag is set on the wiphy so we know to skip this radio when
creating the hostapd instances. Since hostapd gets started first, it
will automatically choose all the radios it needs which support AP mode.
This leaves the remainder of the radios (potentially STA only) for IWD.
2019-05-13 14:13:34 -05:00