3
0
mirror of https://git.kernel.org/pub/scm/network/wireless/iwd.git synced 2024-12-23 06:02:37 +01:00
Commit Graph

417 Commits

Author SHA1 Message Date
James Prestwood
b0e970ae38 test-runner: fix logging, verbose, and process output
There were some major problems related to logging and process
output. Tests which required output from start_process would
break if used with '--log/--verbose'. This is because we relied
on 'communicate' to retrieve the process output, but Popen does
not store process output when stdout/stderr are anything other
than PIPE.

Intead, in the case of logging or outfiles, we can simply read
from the file we just wrote to.

For an explicit --verbose application we must handle things
slightly different. A keyword argument was added to Process,
'need_out' which will ensure the process output is kept
regardless of --log or --verbose.

Now a user should be able to use --log/--verbose without any
tests failing.
2021-01-26 13:41:23 -06:00
James Prestwood
cc345582ad test-runner: don't always print "verbose on for ..."
This shouldn't have been a dbg print, but rather a normal print
which will only be printed when '-d' is used.
2021-01-26 13:41:01 -06:00
James Prestwood
296f1d0d1c test-runner: fix verbose arguments as single string
The verbose arguments come in from the QEMU command line as a
single string. This should have been split into an array immediately
but was not. This led to issues like hostapd debug being enabled
when "-v hostapd_cli" was passed in.
2021-01-26 13:40:49 -06:00
Andrew Zaborowski
a55f8864d2 testrunner: Fix /tmp files cleanup on error
Since the list of files copied to /tmp was part of the return value from
pre_test(), if an exception occurred inside pre_test(), "copied" would
be undefined and the post_test(ctx, copied) call in the finally clause
cause another exception:

raceback (most recent call last):
  File "/home/balrog/repos/iwd/tools/test-runner", line 1508, in <module>
    run_tests()
  File "/home/balrog/repos/iwd/tools/test-runner", line 1242, in run_tests
    run_auto_tests(config.ctx, args)
  File "/home/balrog/repos/iwd/tools/test-runner", line 1166, in run_auto_tests
    post_test(ctx, copied)
UnboundLocalError: local variable 'copied' referenced before assignment

(apart from not being able to clean up the files).  Pass "copied" as a
paremeter to pre_test instead.
2021-01-25 14:01:58 -06:00
James Prestwood
01c108938e test-runner: Use DBus for hwsim radios
Use the hwsim DBus API rather than command line. This both is
faster and more dynamic than doing so with the command line.
This also avoids tracking the radio ID since we can just hang
on to the radio Dbus object directly.
2020-12-17 20:13:56 -06:00
James Prestwood
5567caf64a hwsim: add iftype/cipher disabling through DBus
Update the Dbus API to allow disabling iftypes and ciphers
just as you can with the command line.
2020-12-17 20:11:25 -06:00
James Prestwood
fe0879f7d1 test-runner: start hwsim always (optionally --no-register)
Always start hwsim but if tests do not need radio rules start
with the --no-register option.
2020-12-17 20:11:13 -06:00
James Prestwood
e8031bab2a test-runner: add wait_for_dbus_service
Common API to wait for a DBus service to appear on the bus
2020-12-17 20:10:57 -06:00
James Prestwood
6a1853b01a hwsim: change radio Create() to take a dictionary
The Create() API was limited to only taking a Name and boolean
(for p2p enabling). The actual hwsim nl80211 API can take more
attributes than this (which are actually utilized when creating
from the command line). To get the DBus API up to the same
functionality the two arguments in Create were replaced with
a single dictionary. This allows for extending later if more
arguments are needed.
2020-12-17 20:10:33 -06:00
James Prestwood
799ab03f59 hwsim: check pending_create_msg before replying
In the NEW_RADIO callback hwsim was assuming that DBus had no
yet replied to the Create() method. In some cases the NEW_RADIO
event fires before the actual callback which will respond to
DBus. This causes a crash in the create callback.
2020-12-17 20:08:54 -06:00
James Prestwood
554a78fbf0 hwsim: add --no-register option
Starts hwsim but does not register to mac80211_hwsim. This is to
allow autotests to disable hwsim, while still having the ability
to create/destroy radios over DBus.
2020-12-17 20:08:49 -06:00
James Prestwood
4e4ba6769d test-runner: add iwd-acd to verbose options
This will turn on IWD_ACD_DEBUG
2020-12-08 15:16:28 -06:00
James Prestwood
ada46fb2db test-runner: set processor count for VM
For better reliability the processor count is now set to qemu.
In cases of low CPU count (< 2) hosts the processor count is
limited to 1. Otherwise half of the host cores will be used for
the VM.
2020-12-08 15:15:51 -06:00
James Prestwood
41b8a32dfc test-runner: have start_iwd take a storage dir
Allow the storage directory (default /tmp/iwd) to be configured
just like the state directory. This is in order to support multiple
IWD instances which require separate storage directories for network
provisioning files.
2020-12-08 15:15:33 -06:00
James Prestwood
e1e1c4edd1 test-runner: introduce network namespaces
Our simulated environment was really only meant to test air-to-air
communication by using mac80211_hwsim. Protocols like DHCP use IP
communication which starts to fall apart when using hwsim radios.
Mainly unicast sockets do not work since there is no underlying
network infrastructure.

In order to simulate a more realistic environment network namespaces
are introduced in this patch. This allows wireless phy's to be added
to a network namespace and unique IWD instances manage those phys.
This is done automatically when 'NameSpaces' entries are configured
in hw.conf:

[SETUP]
num_radios=2

[NameSpaces]
ns0=rad1,...

This will create a namespace named ns0, and add rad1 to that
namespace. rad1 will not appear as a phy in what's being called the
'root' namespace (the default namespace).

As far as a test is concerned you can create a new IWD() class and
pass the namespace in. This will start a new IWD instance in that
namespace:

ns0 = ctx.get_namespace('ns0')
wd_ns0 = IWD(start_iwd=True, namespace=ns0)

'wd_ns0' can now be used to interact with IWD in that namespace, just
like any other IWD class object.
2020-11-18 11:01:11 -06:00
James Prestwood
a6808aa55a auto-t: introduce new autotest syntax
Sometimes improperly written tests can end up causing future tests
to fail. For faster debugging you can now add a '+' after a given
autotest which will start that test and run all tests which come
alphabetically after it (as if you are running a full autotest suite).

Example:

./test-runner -A testWPA+

This will run testWPA, testWPA2, testWPA2-no-CCMP, testWPA2-SHA256,
and testWPA2withMFP.
2020-11-16 17:05:45 -06:00
James Prestwood
423f9a4e60 auto-t: do away with -1 test results
This can result in strange test results since there was no less
than zero checks before subtracting the total tests from failed
tests. In case of an internal exception we can just set all values
to zero. This will be handled specially as we do for timeout
errors.
2020-11-16 17:05:45 -06:00
James Prestwood
1eb82d661f auto-t: catch exceptions in post_test 2020-11-16 17:05:45 -06:00
James Prestwood
f88d45c9d3 test-runner: delay starting the shell until after pre_test
This makes --shell somewhat more useful by copying all the test
files and starting test processes before dumping the user into
a shell.
2020-11-16 17:05:45 -06:00
James Prestwood
6bf514c4c3 test-runner: store process stdout
If a blocking process is started store the output in case
the caller needs it. Output will be stored in Process.out.
2020-11-16 17:05:45 -06:00
James Prestwood
4bf8bf2396 test-runner: add option to specify subtests to run
You can now specify a limited list of subtests to run out of a
full auto-test using --sub-tests,-S. This option is limited in
that it is only meant to be used with a single autotest (since
it doesn't make much sense otherwise).

The subtest can be specified both with or without the file
extension.

Example usage:

./test-runner -A testAP -S failure_test,dhcp_test.py

This will only run the two subtests and exclude any other *.py
tests present in the test directory.
2020-11-16 17:05:45 -06:00
James Prestwood
c098db1837 test-runner: fix Radio/Interface circular reference
Code was added with commit 04487f575b which passes a radio object
to the Interface class constructor and stores it in the Interface
object. The radio class also stores each Interface object which
creates a circular reference and causes the Radio to stick around
long after the tests finishes.

I cannot see why the Interface needs to keep track of the Radio
object. None of the wpa_supplicant utilities use this so it has
been removed.
2020-11-03 13:57:37 -06:00
James Prestwood
26c4b8b4fe test-runner: add switch for genl debugging
Providing 'iwd-genl' in the list of verbose options will enable
IWD_GENL_DEBUG.
2020-10-22 09:24:55 -05:00
James Prestwood
334b03478b test-runner: enable valgrind track origins
This gives a more exact location to where a memory problem
is happening. It also uses more RAM so the VM was upgraded to
256MB from 192MB.
2020-10-22 09:24:55 -05:00
Andrew Zaborowski
04487f575b test-runner: Reserve radios for wpa_supplicant
Add support for a WPA_SUPPLICANT section in  hw.conf where
'radN=<config_path>' lines will only reserve radios and create
interfaces for the autotest to be able to start wpa_supplicant on them,
i.e. this prevents iwd or hostapd from being started on them but doesn't
start a wpa_supplicant instance by itself.
2020-10-19 17:20:21 -05:00
James Prestwood
313d8dbbed auto-t: remove requirement for /var/lib/{ead,iwd}
The host systems configuration directories for IWD/EAD were
being mounted in the virtual machine. This required that the
host create these directories before hand. Instead we can
just set up the system and IWD/EAD to use directories in /tmp
that we create when we start the VM. This avoids the need for
any host configuration.
2020-10-14 13:03:17 -05:00
Andrew Zaborowski
db038022e8 test-runner: Make hwsim medium optional
Allow the "hwsim_medium=no" setting in hw.conf's SETUP section to
disable starting hwsim.  It looks like the packets going through
userspace add enough latency that active scans don't work, probe
responses don't arrive within the "dwell time" or probe requests are not
ACKed on time.  I've tried modifying tools/hwsim.c to respond with the
HWSIM_CMD_TX_INFO_FRAME cmd as the first thing after receiving a
HWSIM_CMD_FRAME and even skipping the queue in ell/genl.c by writing the
command synchronously, but neither helped enough to make the scans work.
This does not rule out that hwsim or the way our scans are done can be
fixed and that would obviously be better than what I did in this patch.
2020-09-29 13:20:29 -05:00
Andrew Zaborowski
b321db20ad test-runner: Add flags for DHCP and TLS verbose output
Add "iwd-tls" and "iwd-dhcp" flags for -v.  These should automatically
enable "-v iwd" output because of the substring matching.
2020-09-29 13:20:26 -05:00
Andrew Zaborowski
15c04bb395 test-runner: Enable --p2p when creating interfaces
Enable p2p for the hwsim radios so that we can test P2P.  This can be
made optional for the tests but it seems safe to enable always.
2020-09-29 13:20:24 -05:00
James Prestwood
1a02fdcefd test-runner: add iwmon options
This extends test-runner to also use iwmon if --log is enabled.
For this case the iwmon log will be found inside each test
log directory.

A new option, --monitor <file> was added in case full logging isn't
desired (potentially for timing issues) but a iwmon log is needed.
Be aware that when --monitor is used test-runner will mount the
entire parent directory. test-runner itself will only write to the
file specified, but just know that the parent directory is available
as read-write inside the VM.

--log takes precedence over --monitor, meaning the iwmon log will
be written to <logdir>/<test>/iwmon instead of the file specified
with --monitor if both options are provided.
2020-09-29 09:10:07 -05:00
James Prestwood
9897efe75c test-runner: allow EAD-based autotests
The virtual environment changed slightly adding two network adatpers
which are connected to the same backend so they can communicate with
each other (basically connected to a switch). The hostapd command
line was modified to allow no interfaces to be passed in which lets
us create zero radios but still specify a radius_config file.
2020-09-17 16:19:39 -05:00
James Prestwood
6b3d6f9d60 test-runner: open up dbus config for dbus-monitor
dbus-monitor was not able to eavesdrop on method calls without
some changed to the dbus config.
2020-09-16 14:35:38 -05:00
James Prestwood
f3e2c4e5c3 test-runner: remove ofono/hardcoded IWD options 2020-09-16 14:30:58 -05:00
James Prestwood
2e7fb5ee50 test-runner: use **kwargs for start_process
This is just a more concise/pythonic way of doing function arguments.
Since Process/start_process have basically the same argument names
we can simplify and use **kwargs which will pass the named arguments
directly to Process(). This also allows us to add arguments to Process
without touching start_process if we need.
2020-09-15 16:45:52 -05:00
James Prestwood
9acb89d6f4 tools: remove old test-runner.c 2020-09-14 16:46:20 -05:00
James Prestwood
561c64e7f8 test-runner: print results even if test run is canceled 2020-09-11 13:42:13 -05:00
James Prestwood
6e2aba3907 auto-t: Increase all test timeouts
Slower systems may not be able to make some timeouts that tests
mandated. All timeouts were increased significantly to allow tests
to pass on slow systems.
2020-09-10 17:59:49 -05:00
James Prestwood
17955fcf5a tools: post test-runner rewrite cleanup
Removed test-runner.c, and renamed py_runner to test-runner. Removed
tools/test-runner from .gitignore.

This was done as a separate commit to avoid a nasty diff between the
existing test runner, and the new python version
2020-09-10 17:59:49 -05:00
James Prestwood
0772d4b61a auto-t: introduce pure python test-runner re-write
This patch completely re-writes test-runner in Python. This was done
because the existing C test-runner had some clunky work arounds and
maintaining or adding new features was starting to become a huge pain.

There were a few aspects of test-runner which continually had to
be dealt with when adding any new functionality:

 * Argument parsing: Adding new arguments to test-runner wasn't so
   bad, but if you wanted those arguments passed into the VM it
   became a huge pain. Arguments needed to be parsed, then re-formatted
   into the qemu command line, then re-parsed in a special order
   (backwards) once in the VM. The burden for adding new arguments was
   quite high so it was avoided (at least by me) at all costs.
 * The separation between C and Python: The tests are all written in
   python, but the executables, radios, and interfaces were all created
   from C. The way we solved this was by encoding the require info as
   environment variables, then parsing those from Python. It worked,
   but it was, again, a huge pain.
 * Process management: It started with all processes being launched
   from C, but eventually tests required the ability to start IWD, or
   kill hostapd ungracefully in order to test certain functionality.
   Since the processes were tracked in C, Python had no way of
   signalling that it killed a process and when it started one C had
   no idea. This was mitigated (basically by killall), but it was
   no where close to an elegant solution.

Re-writing test-runner in python solves all these problems and will
be much easier to maintain.

 * Argument parsing: Now all arguments are forwarded automatically
   to the VM. The ArgParse library takes care of parsing and each
   argument is stored in a dictionary.
 * Separation between C and Python: No more C, so no more separation.
 * Process management: Python will now manage all processes. This
   allows a test to kill, restart, or start a new process and not
   have to remember the PID or to kill it after the test.

There are a few more important aspects of the python implementation
that should now be considered when writing new tests:

 * The IWD constructor now has different default arugments. IWD
   will always be started unless specified and the configuration
   directory will always be /tmp
 * Any non *.py file in the test directory will be copied to /tmp.
   This avoids the need for 'tmpfs_extra_stuff' completely.
 * ctrl_interface will automatically be appended to every hostapd
   config. There is no need to include this in a config file from
   now on.
 * Test cleanup is extremely important. All tests get run in the
   same interpreter now and the tests themselves are actually loaded
   as python modules. This means e.g. if you somehow kept a reference
   to IWD() any subsequent tests would not start since IWD is still
   running.
 * For debugging, the test context can be printed which shows running
   processes, radios, and interfaces.

Three non-native python modules were used: PrettyTable, colored, and
pyroute2

$ pip3 install prettytable
$ pip3 install termcolor
$ pip3 install pyroute2
2020-09-10 17:59:49 -05:00
Alvin Šipraga
bfd8cead95 treewide: guard compare functions against signed integer overflow
Besides being undefined behaviour, signed integer overflow can cause
unexpected comparison results. In the case of network_rank_compare(),
a connected network with rank INT_MAX would cause newly inserted
networks with negative rank to be inserted earlier in the ordered
network list. This is reflected in the GetOrderedMethods() DBus method
as can be seen in the following iwctl output:

  [iwd]# station wlan0 get-networks
    Network name                    Security  Signal
  ----------------------------------------------------
    BEOLAN                          8021x     **** }
    BeoBlue                         psk       ***  } all unknown,
    UI_Test_Network                 psk       ***  } hence assigned
    deneb_2G                        psk       ***  } negative rank
    BEOGUEST                        open      **** }
  > titan                           psk       ****
    Linksys05274_5GHz_dmt           psk       ****
    Lyngby-4G-4 5GHz                psk       ****
2020-08-14 10:55:30 -05:00
James Prestwood
f7c036a801 test-runner: use WNOHANG option waiting for PID
If an application has a bug and hangs on SIGTERM this causes
test-runner to hang as well. This is obviously an issue with
the application in question, but test-runner should have a way
of continuing onto the next test rather than hanging.

Instead we can use WNOHANG and a sleep to allow applications
some amount of time to exit, and if they haven't use SIGKILL
instead as well as print an error. Similar to how
wait_for_socket works. The timeout is hard coded to 2 seconds
(100ms sleep + 20 iterations).
2020-06-22 10:52:32 -05:00
James Prestwood
32a55fb75c hwsim: add new 'Delay' property to Rules
This allows a matching frame to be delayed by some number of
milliseconds.
2020-06-08 15:13:02 -05:00
James Prestwood
a0029f8b6e test-runner: run iwmon per-subtest
Previously iwmon was running per-test, which would jumble any subtests
together into the same log file making it hard to parse. Now create
a separate directory for each subtest and put the monitor log and
pcap there.
2020-05-01 23:01:26 -05:00
James Prestwood
7641782924 tools: delay hwsim sending frame
Using mac80211_hwsim can sometimes result in out of order messages
coming from the kernel. Since mac80211_hwsim immediately sends out
frames and the kernel keeps command responses in a separate queue,
bad scheduling can result in these messages being out of order.
In some cases we receive Auth/Assoc frames before the response to
our original CMD_CONNECT. This causes autotests to fail randomly,
some more often than others.

To fix this we can introduce a small delay into hwsim. Just a 1ms
delay makes the random failures disappear in the tests. This delay
is also makes hwsim more realistic since actual hardware will always
introduce some kind of delay when sending or receiving frames.
2020-05-01 20:38:15 -05:00
James Prestwood
f37966a24d test-runner: create pcap when logging 2020-04-02 00:32:38 -05:00
James Prestwood
689a1ed858 test-runner: fix segfault running with --log non-root
When running test-runner as non-root the environment variables
SUDO_GID/SUDO_UID were unset, causing atoi to segfault. This replaces
atoi with strtol, and checks the existance of SUDO_GID/SUDO_UID
before trying to turn it into an integer. This patch also allows
the uid/gid to be read from the user if running as non-root.

Note: running as non-root does require the users permissions to be
setup properly. Directories and files are created when running with
logging, so if the user running test-runner does not have these
permissions the creation of these files will fail.
2020-04-02 00:31:36 -05:00
James Prestwood
7d5ce16f1a test-runner: make default iwd_config_dir=/tmp
The configuration value of iwd_config_dir was defaulting to /etc/iwd
which, in the context of test-runner, is probably not the best idea.
The system may have a main.conf file in /etc/iwd which could cause
tests to fail or behave unexpectedly.

In addition all tests which use iwd_config_dir set it to /tmp anyways.
Because of this, the new default value will be /tmp and no tests will
even need to bother setting this.

The configuration value itself is not being removed because it may be
useful to set arbitrary paths (e.g. /etc/iwd) for example when using
the shell functionality.
2020-03-21 00:10:32 -05:00
Andrew Zaborowski
c41eb6b2b0 tools: Add utility to tx Probe Requests 2020-03-20 10:18:04 -05:00
James Prestwood
24235a2af8 test-runner: fix improper loading of radius_config
This key is special in hostapd, and was being treated as a normal hostapd
config file. This special radius config file needs to be kept unpaired from
any interfaces so now its passed in as a separate argument and appended to
the end of the hostapd execute command.
2020-03-13 23:52:26 -05:00
James Prestwood
af5c77aeba test-runner: fix crash for tests using radius_server
Tests which use a standalone RADIUS server may crash due to
the wiphy array not taking into account the 'radius_server'
key which is skipped during setup.
2020-03-13 23:52:26 -05:00
James Prestwood
37886066d8 test-runner: fix crash if ofono is not installed
The goto was jumping to a label which freed the wiphy list which
had not yet been initialized. This also fixes another similar issue
if chdir fails (in this case tmpfs_extra_stuff would get freed
before being allocated).
2020-03-13 23:52:26 -05:00
Daniel Wagner
83ddd88e61 test-runner: Add missing 's' option to command line parser 2020-03-09 14:10:20 -05:00
Daniel Wagner
3195485e80 test-runner: Add missing space between qemus arguments 2020-03-09 14:10:00 -05:00
Denis Kenzior
0508879e2a hwsim: Use L_DBUS_PROPERTY_FLAG_AUTO_EMIT 2020-01-29 10:51:40 -06:00
Torstein Husebø
759dbdd37f treewide: fix typos 2020-01-21 16:03:28 -06:00
James Prestwood
c93a4d71d6 test-runner: add special radius_server key
Some test cases require (at least with recent hostapd versions) a
stand alone radius server. This is done using driver=none in the
hostapd config file. For this use case hostapd does not need any
radio since its not doing anything wireless related.

Now inside the hw.conf file, under the HOSTAPD group, you can
specify a config file as the value to 'radius_server' key. This
config file will be used without any associated radio when hostapd
is started.
2020-01-06 11:18:48 -06:00
James Prestwood
ede2c11d66 test-runner: print which file failed to fchown 2019-11-21 13:59:44 -06:00
James Prestwood
73f42c2db4 test-runner: move ofono/phonesim logs into test dir
These two processes are executed per-test so they can be passed the
test name and have the logs stored only in tests that actually need
them rather than at the top level.
2019-11-14 15:23:03 -06:00
James Prestwood
186a5195b5 test-runner: erase valgrind log after each test
After changing the valgrind log to --log-fd, it appears that
the log file just gets appended, where before it was overwritten.
This makes test print out all previous tests valgrind results.

Now after printing out the valgrind info we can erase the file
for the next test.
2019-11-13 16:35:41 -06:00
James Prestwood
cb00ae1481 test-runner: make --log take a required argument
This was a mistake in the original implementation. Since test-runner
is always run from the tools directory using --log with no directory
specified would result in many log files being put into the iwd tree.

Instead we can make --log take a required argument so its more
obvious where all the logs are going to go.
2019-11-13 12:36:50 -06:00
James Prestwood
506f502900 test-runner: fix logging directory name
localtime indexes month starting at zero so adding 1 gives us a folder
name with the correct month.

The year is also set as 'years since 1900', so we need to add 1900 to
the year to get the actual year.
2019-11-06 17:04:00 -06:00
James Prestwood
2a2f60d86b test-runner: run iwmon if --log is used
Now that execute_program handles logging automatically its trivial
to add iwmon to the test and get monitor logs as well as normal
process output.
2019-11-06 17:04:00 -06:00
James Prestwood
5709a4afcf test-runner: special case python3 verbose option
Historically if you wanted to see output from a python test you needed
to specify -v pytests. This was also the case if IWD was started from
python.

Nearly every time I run test-runner I would specify "-v iwd,pytests"
only to get the IWD output on these specific tests.

Instead we can special case 'python3' (previously 'pytests') inside
execute_program so that turning on verbosity for 'iwd' also turns it
on for the python tests.
2019-11-06 15:16:40 -06:00
James Prestwood
8ec4a53adb test-runner: refactor/fix valgrind logging issue
After the logging changes verbose IWD with valgrind did not show any IWD
output. This commit fixes this by checking the verbosity against log_name
rather than argv[0] since log_name has a special case for valgrind/iwd.

The valgrind logic in start_iwd was refactored to only use the --log-fd
option rather than using --log-file in addition to --log-fd.
2019-11-06 15:15:20 -06:00
James Prestwood
6bdb154da5 test-runner: log process output to host system
All the processes verbose output is just written to stdout, and very
hard to parse after the test runs. Having test-runner write the
processes output to separate log files is much nicer to view after
the test runs.

A read/write file system is created which is separate and isolated
from the current FS mount (which mounts the whole host file system).
Doing this requires the user to create a folder somewhere to be used
as the mount point. This folder is where all the log files will end
up after test-runner runs.

Since test-runner must be run as root, care was taken to keep the
log files owned by the user which runs test-runner. The UID/GID
are passed to the VM, and any log files created are chown'ed back
to the user who ran test-runner.

execute_program was refactored, and both verbose and log arguments
were removed. The verbose argument was not really required since
we can do check_verbosity(argv[0]) internally. The 'log' flag was
only used along with --shell, and now the user can simply use
--log instead of dumping /tmp/iwd.log.

You can use this feature by specifying --log[=path] to test-runner.
If no path is provided the current directory will be used. Using
the --log flag will result in all processes being run with the
--verbose flag.

A new folder will be created under the --log path. The folder will
be of the name "run-<year>-<month>-<dat>-<PID>". Under this folder
you will find any global process logs. These are processes that
are only run once for ALL tests (hwsim, ifconfig, dbus etc.). There
will also be folders for specific tests that were run. Inside these
test folders will be logs of processes that are run per-test (iwd,
hostapd, python etc.).
2019-11-04 14:41:18 -06:00
Denis Kenzior
fb76ca1f1c hwsim: Update to the new ell API 2019-10-28 15:02:30 -05:00
Marcel Holtmann
152b56a12a treewide: Move the Intel copyright forward to 2019 2019-10-25 00:43:08 +02:00
Tim Kourt
25b3cb77e1 hwsim: Fix potential memory leak
If msg has multiple RADIO NAME attributes, memory leak occurs.

Note that this doesn't happen in practice.
2019-10-23 17:58:51 -05:00
James Prestwood
bf3b403e90 test-runner: fix double free issue on subtest queue
Coverity reported this as a leak, but the test queue is actually
getting freed later and does not need to be freed locally in add_path

This basically reverts c0863e5bc6
2019-10-23 12:28:51 -05:00
Tim Kourt
6fea4e6f4f t-runner: Check for errors after opening file 2019-10-22 21:57:31 -05:00
Tim Kourt
e3551ab56b t-runner: Check for NULL before using strcmp 2019-10-22 21:57:31 -05:00
Tim Kourt
e859e98869 t-runner: Fix possible assignment of NULL pointer 2019-10-22 21:57:31 -05:00
Tim Kourt
43efadcf92 t-runner: Fix out-of-bounds write 2019-10-22 21:57:31 -05:00
Tim Kourt
c0863e5bc6 t-runner: Fix mem leak caused by python test names 2019-10-22 21:57:31 -05:00
Tim Kourt
269bf497c7 t-runner: Fix mem leak caused by unclosed dir 2019-10-22 21:57:31 -05:00
Tim Kourt
7dd32ac093 t-runner: Fix mem leak with sim keys
Use l_settings_get_value instead of l_settings_get_string to
prevent unnecessary memory duplication.
2019-10-22 21:57:31 -05:00
Denis Kenzior
b6554ee41c hwsim: Invoke l_queue_remove prior to object deletion
While the current code is quite safe, the new ordering makes more
logical sense and doesn't confuse static analysis tools.
2019-10-17 13:03:08 -05:00
James Prestwood
e39bb6bfa0 test-runner: allow all unit tests to be run in VM
The -U parameter only allowed for a list of unit tests to be run.
Most of the time for sanity checking you want to run all the unit
tests so this has been changed to take an optional argument.

Now, the -U flag (by itself) will run all unit tests. Running a
single or list of unit tests can still be achieved by:

--unit-tests=test-eapol,test-crypto
2019-10-09 14:33:08 -05:00
James Prestwood
c4ecf2c5c6 tools: ios_convert: embed certs rather than using paths 2019-10-07 23:09:29 -05:00
James Prestwood
64ff5dda77 test-runner: sort test list alphabetically
This makes every full test run consistent. The test list was being
stored as a hashmap, which has no been changed to a queue so we can
insert each test sorted.
2019-09-24 10:07:40 -05:00
Marcel Holtmann
765bb92084 build: Add skeleton manual pages for all installed binaries 2019-09-22 21:57:47 +02:00
James Prestwood
d8aadc49b2 tools: enable CONFIG_CRYPTO_ARC4 in kernel config 2019-09-11 11:40:39 -05:00
Marcel Holtmann
1104d69e29 build: Add StateDirectoryMode=0700 option to systemd service 2019-09-09 06:55:22 +02:00
Marcel Holtmann
c1a9f71b39 build: Add script to start iwd.service via systemd-run 2019-09-08 22:10:28 +02:00
Marcel Holtmann
7e4a6e8a49 test-runner: Remove pointless NULL check before calling l_free 2019-09-08 18:14:13 +02:00
Marcel Holtmann
9d1f7ad6f5 test-runner: Provide CONFIGURATION_DIRECTORY and STATE_DIRECTORY 2019-09-08 18:12:51 +02:00
Marcel Holtmann
7347bd4098 test-runner: Use execvpe to provide environment context 2019-09-08 18:03:10 +02:00
James Prestwood
7acb910a26 test-runner: better define --shell behavior
Specifically, this defines the behavior when --shell is used when no
specific test is specified. In this case test-runner will assume the
'shell' test/sandbox should be used as the test environment as
running all autotests with --shell is not useful or feasable.
2019-08-28 15:55:57 -05:00
James Prestwood
202f65be69 tools/ios_convert: use join rather than manually looping 2019-08-27 10:14:08 -05:00
James Prestwood
de2c808970 tools/ios_convert: add ServerDomainMask
Parses mobileconfig TLSTrustedServerNames
2019-08-27 10:14:08 -05:00
James Prestwood
96ce39c1c2 tools/ios_convert: add raw XML option
This option allows the script to be called with a raw XML file. This
is mostly useful for testing, but since its already implemented we
might as well include it.
2019-08-27 10:14:08 -05:00
James Prestwood
ab0c8cae3c tools/ios_convert: parse DisplayedOperatorName
Some hotspot networks do not contain SSID_STR, which was required
for both naming the provisioning file as well as the 'Name' key.
The DisplayedOperatorName is a better option for this 'Name' key
and could also be used for the filename.

Now, DisplayedOperatorName is preferred, and if not found SSID_STR
is used.
2019-08-27 10:14:08 -05:00
James Prestwood
b05689fc81 tools/ios_convert: Use tag instead of text for IsHotspot
The IsHotspot key just contains a tag after it (true/false), not
a tag with inner text.
2019-08-22 13:05:16 -05:00
James Prestwood
54a5791195 tools/ios_convert: add 'Name' to hotspot config
Also fixed a indentation issue for NAIRealms
2019-08-22 12:46:16 -05:00
James Prestwood
0e317ed9b1 test-runner: print if haveged fails to start
If haveged is not installed it attempts to start but exits with no
indication of what happened.
2019-08-20 14:23:35 -05:00
Denis Kenzior
ed8a4ab931 treewide: Make global variables static 2019-08-15 15:06:59 -05:00
James Prestwood
d2639e5d4e test-runner: log iwd output for --shell
This will allow the user to see the iwd output in /tmp/iwd.log.

execute_program was extended to take a 'log' flag. If true, this
will cause the programs output to be stored in /tmp/<name>.log.
This is only useful when using the --shell command as this file
will go away once the VM stops. The verbose flag always overrides
the logging functionality.

For now only iwd output is logged when using --shell.
2019-08-07 16:37:51 -05:00
James Prestwood
da65acbd07 test-runner: allow booting directly into a shell
It is sometimes valuable to just boot into a shell in order to manually
test functionality. Since test-runner already is setup to run a minimal
kernel with all the necessary requirements for hostapd/iwd it made
sense to allow the user to do this.

If -s,--shell is passed into test runner, no python tests will be run.
The hw.conf file is still used to setup IWD and hostapd so once booted
into the shell you can still (manually) run the test (e.g. via iwctl).

This also works when using USB/PCI passthrough. This makes testing
out different kernel version with real hardware much quicker than
using the host kernel.
2019-08-07 16:37:51 -05:00
James Prestwood
d8dac9a330 tools: add tool for iOS mobileconfig conversion
This tool will convert an iOS 'mobileconfig' file into the IWD
format. The tool only supports PEAP and TTLS networks, including
hotspots.

It will also parse out any certificate chains found in the
mobileconfig file, and verify they lead to a root CA found on the
system. If they do, this root CA will be used as the CACert in
the provisioning file.
2019-07-29 14:04:03 -05:00
James Prestwood
edd4f2b2a2 test-runner: allow glob matching of tests
This allows a glob match of tests, e.g.

./test-runner -A testEAP-TTLS*

would run all TTLS based tests
2019-06-05 16:49:45 -05:00
Denis Kenzior
71ebc00fb7 tools: Update test-runner to the new genl api 2019-05-17 17:10:15 -05:00
Denis Kenzior
dea23bede7 tools: Update hwsim to the new genl api 2019-05-17 17:10:13 -05:00
James Prestwood
f038c11205 test-runner: allow disabling of iftype/cipher
Two new hardware configuration keys were added:

[radX]
iftype_disable=station,ap,adhoc,p2p_client,p2p_go,mesh_point
cipher_disable=wep40,wep104,tkip,ccmp,bip

Any of the above values are supported and can be disabled.
2019-05-13 16:38:23 -05:00
James Prestwood
2d173e5f24 test-runner: add a radio parameters structure for creation
Support is coming to configure radios with a specific set of interface
type and cipher support, so the input to create_hwsim_radio is better
suited to use a parameter structure rather than adding more parameters.
2019-05-13 16:38:23 -05:00
James Prestwood
2d51622c0b test-runner: allow radio list subset to be configured
The radio_confs key was parsed in a way that required all radios
to be specified in the list. This isnt optimal, as you may want to
specially configure a certain radio, while keeping all the others
default.

This change reworks some logic and allows any radio to be specially
configured on its own.
2019-05-13 16:38:23 -05:00
James Prestwood
0e31b33631 hwsim: allow specific radio iftypes/ciphers
mac80211_hwsim now allows setting supported iftypes/ciphers. This patch
enables this support in hwsim. Specific iftypes/ciphers can now be
disabled via the command line when creating a radio:

Disable iftypes:
--iftype-disable station,ap,adhoc,p2p_client,p2p_go,mesh_point

Disable cipher types:
--cipher-disable ccmp,tkip,wep
2019-05-13 16:38:23 -05:00
James Prestwood
79dd04c13a test-runner: in native HW mode, check for AP capabilities
The test should be aborted if there are not enough radios that support
AP mode. The iftype attribute will now be parsed during the wiphy dump
and a flag is set on the wiphy so we know to skip this radio when
creating the hostapd instances. Since hostapd gets started first, it
will automatically choose all the radios it needs which support AP mode.
This leaves the remainder of the radios (potentially STA only) for IWD.
2019-05-13 14:13:34 -05:00
James Prestwood
7c7aafa812 test-runner: skip hostapd iface creation if native HW
In native hardware mode the interfaces already exist, so there is
no need for hostapd to create new ones.
2019-05-13 14:13:18 -05:00
James Prestwood
97ec50ce28 test-runner: fix radio index 0 deletion
In the PCI/USB passthrough changes the wiphy ID was changed to be an
unsigned integer, where id zero corresponded to an error when in native
hardware mode. Along with this, the radio ID for hwsim was changed to a
pre-increment (only in test-runner), so the radio IDs would start at 1.
The repercussions were not fully investigated, but if they were it would
have been seen that hwsim creates radios IDs starting at zero. This left
test-runner and hwsim with unsynchronized radio IDs, and radio zero
never got deleted after each test causing each successive test to
discover old radio IDs.
2019-05-13 14:11:54 -05:00
Andrew Zaborowski
d256bc91ad test-runner: Drop options no longer supported by qemu
-nodefconfig doesn't exist anymore and according to the docs it either
had the same meaning or was implied by -no-user-config so it wouldn't be
needed anyway.  -balloon doesn't exist anymore and according to
https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06985.html
"-balloon none" was a nop, but I suspect -nodefaults may have already
had the effect of disabling creation of the virtio-balloon device.
2019-05-10 14:15:51 -05:00
James Prestwood
07115e56a1 test-runner: Allow native hardware passthrough
This patch allows the host machine to pass through its PCI/USB network
cards into the test-runner virtual machine. By doing this we can run
nearly all the same autotests using physical/real wireless hardware.

First off, utilizing this feature requires a properly configured host
machine. There are kernel boot parameters and config files that need to
be configured before any of this will work. Unfortunately there is no
way around this, and hence this feature is not particularly aimed for
"the masses", but rather for specially configured test machines.

A new configuration file was introduced (tools/hw.conf) which is just an
example, it should be edited to work with the host machine using it. This
file merely holds the PCI addresses/USB bus of the devices you wish to pass
through to qemu.

Passing in this hardware config file with --hw <file> tells test-runner
that you are attempting to use this new feature. The tests themselves
do not need to change, its the initial test setup that required some
changes.

Since we are no longer creating radios we must discover the radios that
are present (once in the VM). This is done using borrowed code from IWD
to dump wiphys and interfaces. As the wiphys/interfaces are dumped, we
build up the wiphy list. In the hwsim case we still build this list up
when we create the radios, which hasn't changed. This does lead us to
have some special cleaup, where in the native case we just 'reset' the
list into its state pre-test (removing any hostapd flags). And as before
with the hwsim case we fully destroy and free the wiphy list, since a
new list will be created on the next test (along with new radios).

There should not need to be any changes to the tests themselves, but
potentially to some hw.conf files. A new key was introduced, 'needs_hwsim'
which need to be set on any tests that require the hwsim dbus API. This
tells test-runner to skip this test, otherwise it would fail in native
mode.

One last minor detail; the wiphy->id was changed to an unsigned int. This
is to match the type the kernel uses when dumping wiphys. Because of
this '0' is now the error case for both hwsim and native mode rather than
-1. Error checks were updated accordingly.
2019-05-01 17:03:34 -05:00
Andrew Zaborowski
d1c4921b86 test-runner: Only create/remove interfaces for hostapd
Move the interface creation code from configure_hw_radios to
configure_hostapd_instaces so as not to create unneeded interfaces on
the wiphys that IWD is going to manage.  We pass a wiphy whitelist to
IWD later and IWD now creates the interfaces it needs on those managed
wiphys.  Change TEST_WIPHY_LIST format to only include the interface
name for the wiphys used by hostapd.

Note that we still remove interfaces just before removing the hwsim
radios on exit, it seems like there's no point removing the interfaces
in that case.
2019-04-22 16:33:40 -05:00
Andrew Zaborowski
20faa37467 test-runner: Futher fix for the SETUP group support
Drop a pointless asignment of has_hw_conf to false when it's already
false, fix index when accessing radio_conf_list.  Apparently the SETUP
group is not used in any of our test and wasn't tested itself so could
as well have removed the code.
2019-04-22 16:33:40 -05:00
Andrew Zaborowski
1462d68eb6 test-runner: Check radio_conf_list is not null
The option may be present but may not parse correctly as a list in which
case has_hw_conf will be true but radio_conf_list will be NULL and we
might crash.
2019-04-22 16:33:40 -05:00
Andrew Zaborowski
fcd5b8b54f test-runner: Drop unused wiphy.interface_index 2019-04-22 16:33:16 -05:00
James Prestwood
71c7d826e4 tools: fix test_runner_kernel_config for HT/VHT
This should now properly load regulatory.db and disable verification
2019-03-18 12:21:49 -05:00
James Prestwood
7b3831515a test-runner: allow test to set the regulatory domain
HT/VHT require setting the regulatory domain to something other
than 00. This adds an option to the hardware config which allows
the regulatory domain to be set to any country.
2019-02-21 16:45:30 -06:00
James Prestwood
c275581389 tools: add option to disable reg domain verification
HT/VHT require channels who's use is restricted depending on
country. When using these channels, cfg80211 tries to load the
regulatory.db file in /lib/firmware and verify the signature.
This poses a problem as the host machine may not have a signed
regulatory.db, or it may have not been signed with the expected
signature which would cause cfg80211 to fail to load the database.

If cfg80211 fails to load the database the country will be set to
00, which is the most restrictive "world roaming" setting. This
does not allow HT/VHT to work properly.

In the context of test-runner we can simply disable the verification.
Unforunately this is not a very common practice, so CONFIG_EXPERT
must be enabled.

If your system does not have /lib/firmware/regulatory.db you must
get it. More info can be found here:

https://wireless.wiki.kernel.org/en/developers/regulatory/wireless-regdb
2019-02-21 16:42:48 -06:00
Mat Martineau
645b7dda4b tools: Add CONFIG_CRYPTO_USER_API_AEAD for full TLS support 2019-02-17 20:43:38 -06:00
James Prestwood
8956451b26 test-runner: decouple --verbose with --valgrind
When using --valgrind, you must also use --verbose iwd, and, depending
on the tests you may also need to include pytests in the verbose flag.
Since anyone using --valgrind definitely wants to see valgrind info
printed they should not need to enable verbose printing. Also, manually
parsing valgrind prints with IWD prints mixed throughout is a nightmare
even for a single test.

This patch uses valgrind's --log-file flag, which is directed to
/tmp/valgrind.log. After the tests runs we can print out this file.
2019-01-18 08:57:21 -06:00
James Prestwood
1927e40fd0 test-runner: fix garbage output on test start
I missed this on a previous commit fixing the same issue with printing
the test results. This fixes garbage output on lines in between test
runs.
2019-01-17 15:43:12 -06:00
James Prestwood
2ec66c9d3f test-runner: fixed garbage output in test results
When printing test results the line separators were not NULL terminated
causing garbage bytes to be printed at the end.
2018-12-17 11:17:34 -06:00
James Prestwood
dd835aba2e test-runner: fix removal of SIM/AKA/AKA' files
If ofono/phonesim is not found on the system, any test requiring those will
be skipped. In this case we would still try and remove symlinks that were
never created. An error would be printed, but the actual source tree files
were getting removed. This adds a new goto label where we can skip the tmpfs
removal in this error case.
2018-12-05 22:24:08 -06:00
Andrew Zaborowski
862707f943 test-runner: Add dbus verbose output options
-v dbus may help debug find client authentication problems but if
everything is ok it will not print any messages.  -v dbus-monitor starts
the dbus-monitor.
2018-11-30 10:26:30 -06:00
Tim Kourt
2cc4fdbf45 t-runner: Add verbosity option for TLS debug 2018-11-02 15:54:32 -05:00
Denis Kenzior
a4512f3cd6 hwsim: Use l_genl_family_set_unicast_handler 2018-11-02 15:52:56 -05:00
Marcel Holtmann
554e4f55db build: Fix includes for using with -std=c99 compiler option 2018-11-01 22:37:11 +01:00
Marcel Holtmann
72a64fa7fb build: Adjust to the latest ELL signal API changes 2018-11-01 22:09:19 +01:00
Marcel Holtmann
2b36727f3a test-runner: Use l_main_run_with_signal instead of open coding it 2018-11-01 20:46:18 +01:00
Marcel Holtmann
36f5056810 hwsim: Use l_main_run_with_signal instead of open coding it 2018-11-01 19:58:34 +01:00
James Prestwood
a2df686d97 hwsim: limit "Unknown attribute type:" print
Since we don't catch all hwsim attribute types in unicast_handler
you see tons of "Unknown attribute type: X" prints. Since this is
not an error, we should only print if the attribute does not exist
in the attribute list.
2018-10-24 16:23:12 -05:00
James Prestwood
ceca5c42ba hwsim: update hwsim attrs to latest mac80211_hwsim 2018-10-24 16:23:12 -05:00
Marcel Holtmann
2606bc73ad build: Add DAEMON_ prefix to CONFIGDIR and STORAGEDIR 2018-09-14 17:58:44 +02:00
Marcel Holtmann
ea20559599 tools: Change net.connman.iwd.hwsim to net.connman.hwsim 2018-09-14 15:27:57 +02:00
Marcel Holtmann
502aac073f tools: Add configure option and D-Bus policy file for hwsim utility 2018-09-14 15:24:15 +02:00
Marcel Holtmann
b88388b172 hwsim: Fix missing handling of help option -h 2018-09-14 15:17:14 +02:00
Tim Kourt
83e66b5831 tools: add needed CONFIG_SECURITYFS option
Add CONFIG_SECURITYFS as a required configuration option
for the kernels to be used with test-runner.
2018-09-13 16:42:23 -05:00
Mat Martineau
27e9b6c435 tools: Remove unused kernel config
CONFIG_DRIVER_NL80211 isn't an option for the kernel. Maybe it was
mistakenly added based on the hostap configuration instructions in
doc/test-runner.txt

If the intent was to add CONFIG_CFG80211 and CONFIG_MAC80211, they're
already in the relevant kernel defconfigs.
2018-06-19 16:06:08 -05:00
Denis Kenzior
e15caa2288 test-runner: Fix kernel command line parsing
Kernel command line arguments were not being parsed properly, $PATH in
particular was completely screwed up and causing commands in user's
$PATH to fail
2018-06-14 15:19:50 -05:00
James Prestwood
16a2fb2664 test-runner: added CONFIG_NLMON to kernel config 2018-05-24 19:42:19 -05:00
Andrew Zaborowski
465488878e hwsim: Look up radio address in radio_info
In mac80211_hwsim each radio is assigned two addresses, the second (addr
1) being used over nl80211.  In SendFrame we'd hardcode the mapping of
address 0 to address 1 even though we track all the addresses in radio_info,
so instead use that data to find the radio that has given address 0.  Also
if no address 0 matches what was supplied over DBus try to find a matching
address 1.

There are ways userspace can request different addresses when managing
mac80211_hwsim radios and the hardcoded mapping would become wrong.
2018-05-22 11:35:57 -05:00
Andrew Zaborowski
676a1f48c3 hwsim: Don't crash if source radio not found
We already print an error when this happens, no need to crash.
2018-05-22 11:35:05 -05:00
James Prestwood
23b9963b20 test-runner: fix iwd verbosity
Specifying --verbose iwd will now enable the -d flag for IWD
2018-05-21 17:49:01 -05:00
Denis Kenzior
d13f10c111 test-runner: Add hwsim to --gdb support 2018-05-21 17:33:45 -05:00
Tim Kourt
79ce3f645f t-runner: init stat totals
the stat_totals structure was not initialized and produced
a garbage output when printed
2018-05-10 18:58:13 -05:00
Mat Martineau
9252d23358 test-runner: Use l_strdup_printf to combine strings
There's a new "-Werror=format-overflow" warning in gcc 8 that detects
potential overflow or truncation with sprintf/snprintf, so the
test-runner build fails with gcc 8. Using l_strdup_printf allows the
build to succeed, and moves a few large buffers from the stack to the
heap.
2018-05-09 20:41:51 -05:00
James Prestwood
aed98137fb test-runner: set env flag if valgrind is being used
If the hw.conf option 'start_iwd' is false, then iwd is started
inside the python test, which means it will not know if the
--valgrind option was passed to test runner. If this is the case
an environment variable is set so the python test knows the
option is being used.
2018-04-27 11:21:33 -05:00
James Prestwood
eaf93de0fa test-runner: enable verbose output for kernel
This was mistakenly removed. This allows you to use "-v kernel"
2018-04-24 14:55:39 -05:00
James Prestwood
f67505c5df test-runner: fix warning in gdb patch 2018-04-03 15:34:51 -05:00
James Prestwood
461b39f206 test-runner: add feature to debug processes with gdb
Specifying the --gdb <process> option to test runner will run gbd on
the given process. Using this option will turn off all timeouts in
test-runner.
2018-04-02 12:49:09 -05:00