These two processes are executed per-test so they can be passed the
test name and have the logs stored only in tests that actually need
them rather than at the top level.
After changing the valgrind log to --log-fd, it appears that
the log file just gets appended, where before it was overwritten.
This makes test print out all previous tests valgrind results.
Now after printing out the valgrind info we can erase the file
for the next test.
This was a mistake in the original implementation. Since test-runner
is always run from the tools directory using --log with no directory
specified would result in many log files being put into the iwd tree.
Instead we can make --log take a required argument so its more
obvious where all the logs are going to go.
localtime indexes month starting at zero so adding 1 gives us a folder
name with the correct month.
The year is also set as 'years since 1900', so we need to add 1900 to
the year to get the actual year.
Historically if you wanted to see output from a python test you needed
to specify -v pytests. This was also the case if IWD was started from
python.
Nearly every time I run test-runner I would specify "-v iwd,pytests"
only to get the IWD output on these specific tests.
Instead we can special case 'python3' (previously 'pytests') inside
execute_program so that turning on verbosity for 'iwd' also turns it
on for the python tests.
After the logging changes verbose IWD with valgrind did not show any IWD
output. This commit fixes this by checking the verbosity against log_name
rather than argv[0] since log_name has a special case for valgrind/iwd.
The valgrind logic in start_iwd was refactored to only use the --log-fd
option rather than using --log-file in addition to --log-fd.
All the processes verbose output is just written to stdout, and very
hard to parse after the test runs. Having test-runner write the
processes output to separate log files is much nicer to view after
the test runs.
A read/write file system is created which is separate and isolated
from the current FS mount (which mounts the whole host file system).
Doing this requires the user to create a folder somewhere to be used
as the mount point. This folder is where all the log files will end
up after test-runner runs.
Since test-runner must be run as root, care was taken to keep the
log files owned by the user which runs test-runner. The UID/GID
are passed to the VM, and any log files created are chown'ed back
to the user who ran test-runner.
execute_program was refactored, and both verbose and log arguments
were removed. The verbose argument was not really required since
we can do check_verbosity(argv[0]) internally. The 'log' flag was
only used along with --shell, and now the user can simply use
--log instead of dumping /tmp/iwd.log.
You can use this feature by specifying --log[=path] to test-runner.
If no path is provided the current directory will be used. Using
the --log flag will result in all processes being run with the
--verbose flag.
A new folder will be created under the --log path. The folder will
be of the name "run-<year>-<month>-<dat>-<PID>". Under this folder
you will find any global process logs. These are processes that
are only run once for ALL tests (hwsim, ifconfig, dbus etc.). There
will also be folders for specific tests that were run. Inside these
test folders will be logs of processes that are run per-test (iwd,
hostapd, python etc.).
Coverity reported this as a leak, but the test queue is actually
getting freed later and does not need to be freed locally in add_path
This basically reverts c0863e5bc6
The -U parameter only allowed for a list of unit tests to be run.
Most of the time for sanity checking you want to run all the unit
tests so this has been changed to take an optional argument.
Now, the -U flag (by itself) will run all unit tests. Running a
single or list of unit tests can still be achieved by:
--unit-tests=test-eapol,test-crypto
This makes every full test run consistent. The test list was being
stored as a hashmap, which has no been changed to a queue so we can
insert each test sorted.
Specifically, this defines the behavior when --shell is used when no
specific test is specified. In this case test-runner will assume the
'shell' test/sandbox should be used as the test environment as
running all autotests with --shell is not useful or feasable.
This option allows the script to be called with a raw XML file. This
is mostly useful for testing, but since its already implemented we
might as well include it.
Some hotspot networks do not contain SSID_STR, which was required
for both naming the provisioning file as well as the 'Name' key.
The DisplayedOperatorName is a better option for this 'Name' key
and could also be used for the filename.
Now, DisplayedOperatorName is preferred, and if not found SSID_STR
is used.
This will allow the user to see the iwd output in /tmp/iwd.log.
execute_program was extended to take a 'log' flag. If true, this
will cause the programs output to be stored in /tmp/<name>.log.
This is only useful when using the --shell command as this file
will go away once the VM stops. The verbose flag always overrides
the logging functionality.
For now only iwd output is logged when using --shell.
It is sometimes valuable to just boot into a shell in order to manually
test functionality. Since test-runner already is setup to run a minimal
kernel with all the necessary requirements for hostapd/iwd it made
sense to allow the user to do this.
If -s,--shell is passed into test runner, no python tests will be run.
The hw.conf file is still used to setup IWD and hostapd so once booted
into the shell you can still (manually) run the test (e.g. via iwctl).
This also works when using USB/PCI passthrough. This makes testing
out different kernel version with real hardware much quicker than
using the host kernel.
This tool will convert an iOS 'mobileconfig' file into the IWD
format. The tool only supports PEAP and TTLS networks, including
hotspots.
It will also parse out any certificate chains found in the
mobileconfig file, and verify they lead to a root CA found on the
system. If they do, this root CA will be used as the CACert in
the provisioning file.
Two new hardware configuration keys were added:
[radX]
iftype_disable=station,ap,adhoc,p2p_client,p2p_go,mesh_point
cipher_disable=wep40,wep104,tkip,ccmp,bip
Any of the above values are supported and can be disabled.
Support is coming to configure radios with a specific set of interface
type and cipher support, so the input to create_hwsim_radio is better
suited to use a parameter structure rather than adding more parameters.
The radio_confs key was parsed in a way that required all radios
to be specified in the list. This isnt optimal, as you may want to
specially configure a certain radio, while keeping all the others
default.
This change reworks some logic and allows any radio to be specially
configured on its own.
mac80211_hwsim now allows setting supported iftypes/ciphers. This patch
enables this support in hwsim. Specific iftypes/ciphers can now be
disabled via the command line when creating a radio:
Disable iftypes:
--iftype-disable station,ap,adhoc,p2p_client,p2p_go,mesh_point
Disable cipher types:
--cipher-disable ccmp,tkip,wep
The test should be aborted if there are not enough radios that support
AP mode. The iftype attribute will now be parsed during the wiphy dump
and a flag is set on the wiphy so we know to skip this radio when
creating the hostapd instances. Since hostapd gets started first, it
will automatically choose all the radios it needs which support AP mode.
This leaves the remainder of the radios (potentially STA only) for IWD.
In the PCI/USB passthrough changes the wiphy ID was changed to be an
unsigned integer, where id zero corresponded to an error when in native
hardware mode. Along with this, the radio ID for hwsim was changed to a
pre-increment (only in test-runner), so the radio IDs would start at 1.
The repercussions were not fully investigated, but if they were it would
have been seen that hwsim creates radios IDs starting at zero. This left
test-runner and hwsim with unsynchronized radio IDs, and radio zero
never got deleted after each test causing each successive test to
discover old radio IDs.
-nodefconfig doesn't exist anymore and according to the docs it either
had the same meaning or was implied by -no-user-config so it wouldn't be
needed anyway. -balloon doesn't exist anymore and according to
https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06985.html
"-balloon none" was a nop, but I suspect -nodefaults may have already
had the effect of disabling creation of the virtio-balloon device.
This patch allows the host machine to pass through its PCI/USB network
cards into the test-runner virtual machine. By doing this we can run
nearly all the same autotests using physical/real wireless hardware.
First off, utilizing this feature requires a properly configured host
machine. There are kernel boot parameters and config files that need to
be configured before any of this will work. Unfortunately there is no
way around this, and hence this feature is not particularly aimed for
"the masses", but rather for specially configured test machines.
A new configuration file was introduced (tools/hw.conf) which is just an
example, it should be edited to work with the host machine using it. This
file merely holds the PCI addresses/USB bus of the devices you wish to pass
through to qemu.
Passing in this hardware config file with --hw <file> tells test-runner
that you are attempting to use this new feature. The tests themselves
do not need to change, its the initial test setup that required some
changes.
Since we are no longer creating radios we must discover the radios that
are present (once in the VM). This is done using borrowed code from IWD
to dump wiphys and interfaces. As the wiphys/interfaces are dumped, we
build up the wiphy list. In the hwsim case we still build this list up
when we create the radios, which hasn't changed. This does lead us to
have some special cleaup, where in the native case we just 'reset' the
list into its state pre-test (removing any hostapd flags). And as before
with the hwsim case we fully destroy and free the wiphy list, since a
new list will be created on the next test (along with new radios).
There should not need to be any changes to the tests themselves, but
potentially to some hw.conf files. A new key was introduced, 'needs_hwsim'
which need to be set on any tests that require the hwsim dbus API. This
tells test-runner to skip this test, otherwise it would fail in native
mode.
One last minor detail; the wiphy->id was changed to an unsigned int. This
is to match the type the kernel uses when dumping wiphys. Because of
this '0' is now the error case for both hwsim and native mode rather than
-1. Error checks were updated accordingly.
Move the interface creation code from configure_hw_radios to
configure_hostapd_instaces so as not to create unneeded interfaces on
the wiphys that IWD is going to manage. We pass a wiphy whitelist to
IWD later and IWD now creates the interfaces it needs on those managed
wiphys. Change TEST_WIPHY_LIST format to only include the interface
name for the wiphys used by hostapd.
Note that we still remove interfaces just before removing the hwsim
radios on exit, it seems like there's no point removing the interfaces
in that case.
Drop a pointless asignment of has_hw_conf to false when it's already
false, fix index when accessing radio_conf_list. Apparently the SETUP
group is not used in any of our test and wasn't tested itself so could
as well have removed the code.
The option may be present but may not parse correctly as a list in which
case has_hw_conf will be true but radio_conf_list will be NULL and we
might crash.
HT/VHT require setting the regulatory domain to something other
than 00. This adds an option to the hardware config which allows
the regulatory domain to be set to any country.
HT/VHT require channels who's use is restricted depending on
country. When using these channels, cfg80211 tries to load the
regulatory.db file in /lib/firmware and verify the signature.
This poses a problem as the host machine may not have a signed
regulatory.db, or it may have not been signed with the expected
signature which would cause cfg80211 to fail to load the database.
If cfg80211 fails to load the database the country will be set to
00, which is the most restrictive "world roaming" setting. This
does not allow HT/VHT to work properly.
In the context of test-runner we can simply disable the verification.
Unforunately this is not a very common practice, so CONFIG_EXPERT
must be enabled.
If your system does not have /lib/firmware/regulatory.db you must
get it. More info can be found here:
https://wireless.wiki.kernel.org/en/developers/regulatory/wireless-regdb
When using --valgrind, you must also use --verbose iwd, and, depending
on the tests you may also need to include pytests in the verbose flag.
Since anyone using --valgrind definitely wants to see valgrind info
printed they should not need to enable verbose printing. Also, manually
parsing valgrind prints with IWD prints mixed throughout is a nightmare
even for a single test.
This patch uses valgrind's --log-file flag, which is directed to
/tmp/valgrind.log. After the tests runs we can print out this file.
If ofono/phonesim is not found on the system, any test requiring those will
be skipped. In this case we would still try and remove symlinks that were
never created. An error would be printed, but the actual source tree files
were getting removed. This adds a new goto label where we can skip the tmpfs
removal in this error case.
-v dbus may help debug find client authentication problems but if
everything is ok it will not print any messages. -v dbus-monitor starts
the dbus-monitor.
Since we don't catch all hwsim attribute types in unicast_handler
you see tons of "Unknown attribute type: X" prints. Since this is
not an error, we should only print if the attribute does not exist
in the attribute list.
CONFIG_DRIVER_NL80211 isn't an option for the kernel. Maybe it was
mistakenly added based on the hostap configuration instructions in
doc/test-runner.txt
If the intent was to add CONFIG_CFG80211 and CONFIG_MAC80211, they're
already in the relevant kernel defconfigs.
Kernel command line arguments were not being parsed properly, $PATH in
particular was completely screwed up and causing commands in user's
$PATH to fail
In mac80211_hwsim each radio is assigned two addresses, the second (addr
1) being used over nl80211. In SendFrame we'd hardcode the mapping of
address 0 to address 1 even though we track all the addresses in radio_info,
so instead use that data to find the radio that has given address 0. Also
if no address 0 matches what was supplied over DBus try to find a matching
address 1.
There are ways userspace can request different addresses when managing
mac80211_hwsim radios and the hardcoded mapping would become wrong.
There's a new "-Werror=format-overflow" warning in gcc 8 that detects
potential overflow or truncation with sprintf/snprintf, so the
test-runner build fails with gcc 8. Using l_strdup_printf allows the
build to succeed, and moves a few large buffers from the stack to the
heap.
If the hw.conf option 'start_iwd' is false, then iwd is started
inside the python test, which means it will not know if the
--valgrind option was passed to test runner. If this is the case
an environment variable is set so the python test knows the
option is being used.
Without this subsequent tests may be affected by hwsim not being
restarted.
Additionally in 4.13 the kernel will not use the registered hwsim
wmedium for wiphys created after the HWSIM_CMD_REGISTER call and
there's no way to re-register it without disconnecting from netlink
which is a bit of work. It's a one line fix in 4.13, I've not yet
checked if this has changed in current git.
When running multiple BSSes in one ESS this solves the communication
between them (called RRB) for purposes like preauthentication,
FT key pull and push and FT-over-DS without complicated bridges. At the
same time we're unlikely to have a scenario where we need the
communication to fail so there's no need for this to be configurable.
The supporting code for multiple hostapd processes is left in place so
that configure_hostapd_instances can decide how many processes to run
based on hw.conf and policies. start_hostapd now uses "-i wln0,..."
which is no longer documented in hostapd manual page or usage() but
still supported in current git and required if interface names are not
provided in the config files (possibly unless -b is used which is also
undocumented.)
There was an unnecessary l_strsplit call when parsing the verbose
options for test-runner. The verbose options get parsed inside
qemu already, so this call was removed.
Can now pass -V, --valgrind to test-runner to run valgrind on
iwd during autotests. Note: the verbose option for iwd must
also be passed in order to see output (-v iwd).
Added a new method SendFrame() under the radios .Interface
interface. This method takes two byte arrays as parameters,
first is the station address to send the frame to, and the
second is the raw frame data.
The following verbose options can be used:
"unit" - see output from unit tests
"kernel" - see hwsim kernel prints
"pytests" - see prints from python autotests
The existing verbose option would turn on debug logging for most
processes in the test-runner framework. This change makes the verbose
option more granular. The --verbose or -v option should now be
followed by a comma separated list of the test-runner processes you
want to see debug logging from. Currently iwd, hwsim, ofonod, and
hostapd are valid options to pass to the verbose flag e.g.
-v iwd,hwsim,ofonod,hostapd
Or any single/combination of the above applications.
A new option 'sim_keys' can be put in the SETUP group. If
sim_keys=ofono, then test-runner will attempt to start ofono
and phonesim to be used with EAP-SIM/AKA. If ofono OR phonesim
are not found the test will be skipped.
Any other value of 'sim_keys' should be a 'keys' file
e.g. /tmp/sim_keys.conf which should contain hardcoded
SIM/AKA key values.
Replace is_multicast_addr with util_is_broadcast_address usage.
is_multicast_addr seems to have been wrong, first because we're not
interested in just any multicast address (defined to be same as "group
address"), but rather specifically the broadcast address, as we don't
know of any specific address groups other than broadcast. And also
wrong because the "Individual/Group bit" is the LSB of byte 0, not the
MSB of byte 0 apparently.
Derive a floating-point interval value from integers using math rather
than integer->string->float conversion. The string technique triggered a
-Wformat-overflow warning.
Use the same machine type regardless of has_virt: q35 with accelerators
list set to kvm:tcg, this will use KVM if available and fall back to TCG
if not available so there's no point checking has_virt. We can't
reliably know if KVM is usable anyway: even if the CPU has the
virtualization extensions, the support might not be enabled in the QEMU
build or there may be no kernel module or we may be calling a different
qemu executable than the one supporting KVM.
Set CPU type to "host" if KVM available and "max" otherwise because
"host" is not supported without KVM. We actually want to emulate the
host CPU as closely as possible so that host executables can run even if
optimized for the specific CPU. "max" seems to be the new way (since
Feb only) to request "host" cpu without KVM. It seems that the idea is
for "max" to become same as "host" if KVM is enabled so at some point we
will want to switch to using "max" in both cases.
The "level=9" flag seems to have been an error, there's no CPU with
cpuid max level of 9 and I can't see the purpose of setting the cpuid
level other than the host cpu's level.
-enable-kvm is redundant with accel=kvm:tcg in current qemu.
Again I'm not able to test this patch on a cpu that would be affected
by it but I hope this fixes some situations which are currently broken
anyway.
Update check_virtualization to check for the SVM on x86, KVM supports
both VMX and SVM. Fix the clobber list of the VMX check: memory is not
clobbered while ebx and edx are (ecx is already marked as output).
I'm not able to test this patch on a cpu that would be affected by it
but I don't think test-runner has been tested on those cpus yet.
Communicate the final wiphy and interface configuration data after the
hostapd and iwd processes are started, to the test scripts as an
environment variable TEST_WIPHY_LIST. This way the test scripts can
know which interfaces are used by hostapd instances, which hostapd
config they're using, and which are used by iwd. The typical value will
be
rad0=wln0=hostapd,ctrl_interface=/var/run/hostapd,config=ssidOpen.conf
rad1=wln1=iwd
Handle NewLink events to detect interface mac address changes. In my
current testing I don't see nl80211 events on address change so it's
best to react to both types of events.
The phy creation logic was a bit complex, using a hashmap to map between
phy names in the config file (e.g. rad0, rad2, etc) and interface names
created for that phy.
We take advantage of the fact that hwsim can create a radio with a given
name to simply assign the radio name from the autotest config file
directly. Then we name any interfaces created in order of phy creation.
A few extra sanity checks are also added.
Actually run hwsim in the daemon mode to work as the wireless medium now
that hwsim supports it. The current test-runner code, which assumed a
different command line syntax, wasn't functional but it didn't affect
test-runner in any way.
Don't require the full length of a Management MPDU as a condition to
forward the frame, only require data up to the three addresses we need
to know where to forward the frame.
This check was failing with some frames during a deauthentication. We
could possibly forward shorter frames too if needed (send to all
possible recipients.)
Don't handle the hwsim netlink events we use to track radios and
interfaces if we're not in daemon mode. This quiets dbus errors when
using hwsim through the command line.
The name attribute in the NEW RADIO command needs at least 4 bytes for
the attribute header (struct nlattr), all the characters of the name
string and a NUL byte, and up to 3 bytes of alignment padding.
Otherwise, depending on the name length and whether the NO_VIF attribute
was the last, that attribute could end up being dropped and we were
ending up with too many interfaces inside test-runner.
Implement a hwsim wireless medium inside hwsim.c. This doesn't do
anything to the frames it moves around yet, only tries to implement
the same logic that the kernel medium contains.
In daemon mode start a basic passive DBus interface to expose the
information on radios attached to mac80211_hwsim. In this version
interfaces have objects of their own. It might be simpler to only
show them as an array property on the radio object (array of pairs of
string, one string for address, one for name).
Read wiphy addresses from sysfs and perform the wiphy name to wiphy idx
mapping using sysfs. Do this directly on a new radio notification and
stop using new wiphy notifications except for updating the radio names.
Having the wiphy index available synchronously when parsing a new radio
event we store the wiphy index in the radio_info_rec struct directly and
drop struct wiphy_info_rec as there was a 1:1 mapping. With this, and
knowing that all radio_info data is available when new interface
notifications are received, the tracking is simplified because dbus
objects can be created and destroyed within the notification handlers.
We also now store both the wiphy hardware address data and separately
the interface MAC addresses and can use them more appropriately in the
medium implementation.
The kernel expects the radio name attribute to include the string's zero
byte. Things may still work without this if there is padding after the
attribute.
This has been now patched and the zero byte will be optional when that
patch makes its way through different trees.
Add a daemon mode that is entered when no action was specified on the
command line. In this mode hwsim tracks information on radios through
the netlink events. The interface to make use of the information is
added in the next patch.
killall doesn't wait for the iwd process to clean up, so using it is not
enough when cleaning up between tests.
Using killall -w also doesn't work since iwd is launched by the script.
By the time killall is invoked, the script process is also cleaned up.
So when iwd is killed via killall, nobody is there to reap the zombie
process (test-runner is running as init, but doesn't do this)
The easiest solution is to make the test script itself clean up any iwd
processes it launches. This is what has been implemented in the
previous patch.
Some unit tests expect to be running from the top level iwd tree
directory to load certificates and such. Make sure that test-runner
chdirs to the appropriate directory prior to running the unit tests.
The argv buffer allocated by alloca is only large enough to hold the
original argument list. Additional arguments (such as --kernel) are
appended at the end, which results in the stack corruption. Make sure
to allocate space for additional arguments.
When KVM is available, use the -cpu host option. This is useful on
distributions that might use architecture specific optimizations, which
might not be available on the default machine being emulated by qemu.
For example, a Gentoo box compiled with "-march=core-avx2" would result
in several binaries not executing properly inside the qemu environment
due to instruction set mismatches.
James Prestwood