get_ordered_network() now scans automatically and has been updated
to use the StationDebug.Scan() API rather than doing a full
dbus scan (unless full_scan = True). The frequencies to be scanned
are picked automatically based on the current hostapd status
(hidden behind ctx.hostapd.get_frequency()).
This changes all tests to use the default get_ordered_network behavior
rather than some custom or incorrect logic. Any use of
scan_if_needed=True has been removed since this is now the default.
Also any explicit scanning has been removed for tests which do not
require it (where the default behavior is good enough).
With the addition of connect_bssid/roam very few tests actually
require hwsim. Since hwsim can lead to problems with scan results
its best to have it off by default and have each test that needs
it explicitly turn it on.
Tests which previously turned it off have had that option removed.
Tests that do require hwsim still are vulnerable to scan result
problems, so for these tests beacon_int was added to the hostapd
config which seems to help with reliability somewhat.
There is a common block of code in nearly every test which is incorrect,
most likely a copy-paste from long ago. It goes something like:
wd.wait_for_object_condition(device, 'not obj.scanning')
device.scan()
wd.wait_for_object_condition(device, 'not obj.scanning')
network = device.get_ordered_network("ssid")
The problem here is that sometimes the scanning property does not get
updated fast enough before device.scan() returns, meaning get_ordered_network
comes up with nothing. Some tests pass scan_if_needed=True which 'fixes'
this but ends up re-scanning after the original scan finishes.
To put this to rest scan_if_needed is now defaulted to True, and no
explicit scan should be needed.
In addition the send_bss_transition call was updated to only send a
single BSS. By sending two BSS's IWD is left to pick whichever one
it wants which makes the test behavior undefined.
This will use the Roam() developer method to force a roam to
a certain BSS. This is particularly useful for any test requiring
roams that are not testing IWD's BSS selection logic. Rather than
creating hwsim rules, setting low RSSI values, and waiting for the
roam logic/scan to happen Roam() can be used to force the roam
logic immediately.
Several tests tests for connectivity with the expectation that it
will fail. This ends up taking 30+ seconds because testutil retries
3 times, each with a 10 second timeout. By passing expect_fail=True
this lowers the timeout to zero, and skips any retries.
After some code changes the FT-FILS AKM was no longer selectable
inside network_can_connect_bss. This normally shouldn't matter
since station ends up selecting the AKM explicitly, including
passing the fils_hint, but since the autotests only included
FT-FILS AKMs this caused the transition to fail with no available
BSS's.
To fix this the standard 8021x AKM was added to the hostapd
configs. This allows these BSS's to be selected when attempting
to roam, but since FT-FILS is the only other AKM it will be used
for the actual transition.
testScan was creating 10 separate hidden networks which
sometimes bogged down hostapd to the point that it would
not start up in time before test-runner's timeouts fired.
This appeared to be due to hostapd needing to create 10
separate interfaces which would sometimes fail with -ENFILE.
The test itself only needed two separate networks, so instead
the additional 8 can be completely removed.
The scan ranking logic was previously changed to be based off a
theoretical calculated data rate rather than signal strength.
For HT/VHT networks there are many data points that can be used
for this calculation, but non HT/VHT networks are estimated based
on a simple table mapping signal strengths to data rates.
This table starts at a signal strength of -65 dBm and decreases from
there, meaning any signal strengths greater than -65 dBm will end up
getting the same ranking. This poses a problem for 3/4 blacklisting
tests as they set signal strengths ranging from -20 to -40 dBm.
IWD will then autoconnect to whatever network popped up first, which
may not be the expected network.
To fix this the signal strengths were changed to much lower values
which ensures IWD picks the expected network.
Break up the SAE tests into two parts: testSAE and testSAE-AntiClogging
testSAE is simplified to only use two radios and a single phy managed
by hostapd. hostapd configurations are changed via the new 'set_value'
method added to hostapd utils. This allows forcing hostapd to use a
particular sae group set, or force hostapd to use SAE H2E/Hunting and
Pecking Loop for key derivation. A separate test for IKE Group 20 is no
longer required and is folded into connection_test.py
testSAE-AntiClogging is added with an environment for 5 radios instead
of 7, again with hostapd running on a single phy. 'sae_pwe' is used to
force hostapd to use SAE H2E or Hunting and Pecking for key derivation.
Both Anti-Clogging protocol variants are thus tested.
main.conf is added to both directories to force scan randomization off.
This seems to be required for hostapd to work properly on hwsim.
[General].APRange is now [IPv4].APAddressPool and the netmask is changed
from 23 to 27 bits to make the test correctly assert that only two
default-sized subnets are allowed by IWD simultaneously (default has
changed from 24 to 28 bits)
These test cases depend on setting up the existing hostapd instance to a
set of known addresses, which might be different from what test-runner
sets. During this time, any scans might result in the old and the new
addresses used by hostapd to be found in the scan results.
Fix that by using start_iwd=0 which tells test_runner that the test
wants to start iwd itself. This delays starting iwd until after the
setUpClass routine has been called and hostapd configured properly.
Also use more sensible rssi values for the 'non-preferred' bss.
Otherwise, ranking BSSes by throughput can confuse the test logic
since both BSSes are ranked the same and either can be picked by
autoconnect.
The FT-over-DS procedure now authenticates with multiple BSS's
upon connecting. This causes list_sta() to return our address for
any authenticated APs. It has now been changed to work with this
new behavior, as well as a check that the station fully connected
to the expected AP initially.
The AuthCenter was still not being fully cleaned up in these
tests. It was being stopped but there was still a reference being
held which prevented __del__ from being called.
This file was not included when testNetconfig was introduced
and is required. My system was working fine as it was in my
local tree but has been missing and not passing for others.
The cls object is part of the unittest framework and its lifespan
is out of test-runner's control. Setting objects into the cls
object sometimes keeps those objects around longer than desired.
Its best to unset anything set in cls when the test is tore down.
This test fails randomly, and it appears to be due to excessive
scanning. Historically most autotests start a dbus scan right
away. The problem is that most likely a periodic scan is already
ongoing, meaning the dbus scan gets queued. If a Connect() call
comes in (which it always does), the dbus scan gets delayed and will
trigger once connected, at a time the test is not expecting. This
can cause problems with any assumed timing as well as offchannel
frames.
This patch removes the explicit DBus scanning and instead uses
scan_if_needed with get_ordered_networks. The 'all_blacklisted_test'
was also modified to wait for scanning to complete after failing
to connect to all BSS's. This lets all the networks fully come
up (after being blocked by hwsim) and appear in scan results.
Every single roaming test had one of two problems with watching the
state change between roaming --> connected. Either the test used
wait_for_object_condition to wait for 'connected' which could allow
other states in between. Or it simply used an assert. The assert
wouldn't allow other state changes, but at the cost of potentially
failing due to IWD not having made it to the 'connected' state yet.
Now we have wait_for_object_change which takes two conditions:
initial (from_str) and expected (to_str). This API will not allow
any other conditions except these, and will wait for the expected
condition before continuing. This allows roaming test to reliably
wait for the roaming --> connected state change.
This is similar to wait_for_object_condition, but will not allow
any intermediate state changes between the initial and expected
conditions. This is useful for roaming tests when the expected
state change is 'connected' --> 'roaming' with no changes in
between.
This test occationally failed due to a badly timed DBus scan
triggering right when hwsim tried sending out the spoofed frame.
This caused mac80211_hwsim to reject CMD_FRAME when the timing
was just right.
Rather than always starting a DBus scan we can rely on periodic
scans and only DBus scan if there are no networks in IWD's list.
A scanning check was also added prior to sending out the frame
and if true we wait for not scanning. This is more paranoia than
anything.
Sometimes scan results can come in with a MAC address which
should be in the first index of addrs[] (42:xx:xx:xx:xx:xx).
This causes a failure to lookup the radio path.
There was also a failure path added if the radio cannot be
found rather than rely on DBus to fail with a None path.
The arguments to SendFrame were also changed to use the
ByteArray DBus type rather than python's internal bytearray.
This shouldn't have any effect, but its more consistent with
how DBus arguments should be used.
After recent changes fixing wait_for_object_condition it was accidentally
made to only work with classes, not other types of objects. Instead
create a minimal class to hold _wait_timed_out so it doesnt rely on
'obj' holding the boolean.
The testAPRoam autotest was silently failing on my machine until I
realized that my distribution hostapd (Arch Linux) is not built with
CONFIG_WNM_AP=y. Indeed, it is also disabled by default in upstream
hostapd. This resulted in the send_bss_transition() function of
hostapd.py silently failing. With this change, throw an exception in
case the BSS_TM_REQ command does not succeed to hopefully save others
the time of debugging this problem.
After the test-runner re-write many tests were left with
stale options that are no longer used at all. These were
periodically getting removed as changes were made to
individual tests, but its apparent now that a tree wide
removal was needed.
There were some major problems related to logging and process
output. Tests which required output from start_process would
break if used with '--log/--verbose'. This is because we relied
on 'communicate' to retrieve the process output, but Popen does
not store process output when stdout/stderr are anything other
than PIPE.
Intead, in the case of logging or outfiles, we can simply read
from the file we just wrote to.
For an explicit --verbose application we must handle things
slightly different. A keyword argument was added to Process,
'need_out' which will ensure the process output is kept
regardless of --log or --verbose.
Now a user should be able to use --log/--verbose without any
tests failing.
After the re-write this was broken and not noticed until
recently. The issue appeared to be that the GLib timeout
callback retained no context of local variables. Previously
_wait_timed_out was set as a class variable, but this was
removed so multiple IWD instances could work. Without
_wait_timed_out being a class variable the GLib timeout
setting it had no effect on the wait loop.
To fix this we can set _wait_timed_out on the object being
passed in. This is preserved in the GLib timeout callback
and setting it gets honored in the wait loop.
Tests netconfig with a static configuration, as well as tests ACD
functionality.
The test has two IWD radios which will eventually use the same IP.
One is configured statically, one will receive the IP via DHCP.
The static client sets its IP first and begins using it. Then the
DHCP client is started. Since ACD in a DHCP client is configured
to use its address indefinitely, the static client *should* give
up its address.
Certain classes were still using the default namespace. This
didn't matter yet since testAP was the only test using namespaces,
and the AP interface was the only one being used.
For an IWD station on a separate namespace all objects need to
be accessable, so the namespace is passed along to those as needed.
Due to timing this test sometimes does not pass because it was
just asserting on the device state rather than waiting for a
change. This generally worked but not always.
Both these tests create many radios which sometimes causes timing
problems when hwsim is running. Since hwsim is not required for
these tests we can disable it and increase test reliability.
When network namespaces are introduced there may be multiple
IWD class instances. This makes IWD.get_instance ambiguous
when namespaces are involved. iwd.py has been refactored to
not use IWD.get_instance, but testutil still needs it since
its purely based off interface names. Rather than remove it
and modify every test to pass the IWD object we can just
maintain the existing behavior for only the root namespace.
The agent path was generated based on the current time which
sometimes yielded duplicate paths if agents were created quickly
after one another. Instead a simple iterator removes any chance
of a duplicate path.
If running multiple tests testNetconfig would fail due to the
hardcoded wln0 in the dhcpd.conf file. dhcpd can actually start
by passing in the interface to the run command rather than
inside the config file.
The existing AP tests needed to be modified to start IWD from
python since the DHCP test uses a different main.conf.
Also removed some stale hw.conf options that are no longer used.
If the caller specifies the number of devices only return that many.
Some sub-tests may only need a subset of the total number of devices
for the test. If the number of devices expected is less than the total
being returned, python would throw an exception.
If a test does not need any hostapd instances but still loads
hostapd.py for some reason we want to gracefully throw an
exception rather than fail in some other manor.
Add the new wpas.Wpas class roughly based on hostapd.HostapdCLI but only
adding methods for the P2P-related stuff.
Adding "wpa_supplicant" to -v will enable output from the wpa_supplicant
process to be printed and "wpa_supplicant-dbg" will make it more verbose
("wpa_supplicant" is not needed because it seems to be automatically
enabled because of the glob matching in ctx.is_verbose)
The host systems configuration directories for IWD/EAD were
being mounted in the virtual machine. This required that the
host create these directories before hand. Instead we can
just set up the system and IWD/EAD to use directories in /tmp
that we create when we start the VM. This avoids the need for
any host configuration.
This module is essentially a heavily stripped down version of iwd.py
to work with EAD. Class names were changed to match EAD but basically
the EAD, Adapter, and AdapterList classes map 1:1 to IWD, Device, and
DeviceList.
This is somewhat of a hack, but the IWDDBusAbstract is a very
convenient abstraction to DBus objects. The only piece that restricts
it to IWD is the hardcoded IWD_SERVICE. Instead we can pass in a
keyword argument which defaults to IWD_SERVICE. That way other modules
(like EAD) can utilize this abstraction with their own service simply
by changing that service argument.
The interface was hard coded to wln0 which works when running single
tests but not when running multiple. Instead use the actual ifname
that hostapd is using.
Tests that DHCP using IWD's internal netconfig functions properly.
The actual IP address assignment is not verified, but since IWD does
not signal the connection as successful unless DHCP succeeds we
can assume it was successful by checking that the device is connected.
The process of actually starting dhcpd and configuring the interfaces
is quite simple so it was left in the autotest itself. If (or when)
more tests require IP capabilities (p2p, FILS, etc) this could be
moved into test-runner itself and be made common. The reason I did not
put it in there now is a) because this is the only test and b) more
complex DHCP cases are likely to develop and may require more than this
simplistic setup (like multiple APs/interfaces)
The AdHoc functionality in iwd.py was not consistent at all with
how all the other classes worked (my bad). Instead we can create
a very simple AdHocDevice class which inherits all the DBus magic
in the IWDDBusAbstract class.
Many tests waited on the network object 'connected' property after
issuing a Connect command. This is not correct as 'connected' is
set quite early in the connection process. The correct way of doing
this is waiting for the device state to change to connected.
This common code was replaced, hopefully putting to rest any random
failures that happen occasionally.
Some cleanup code got removed by mistake which cleared out any
hwsim rules before the next subtest. Without this the second test
would end up getting erroneous signal strength numbers in the scan
results causing a failure.
This got added in the re-write but a __del__ method was also
added to the Rule class as well. This caused problems if hwsim
cleaned up since it removed the rules, which caused each rule
to call __del__. Since the rule had already been removed there
was no longer a DBus interface which raised an exception.
Before the re-write there was interesting escapes being used for
set_neighbor. Curiously now hostapd fails to set the neighbor due
to these escapes so they have been removed.
Switched around hwsim rules with the IWD initializer to avoid
IWD periodically scanning before hwsim rules are in place. Removed
some unneeded code during teardown.
Changed to wait for DeviceState instead of network object as well
as moved hwsim rules ahead of the IWD initializer to avoid IWD
scanning before the rules are fully in place.
This test occationally failed, and it uses the old style of waiting
for connected on the network object instead of the device object.
The hwsim rule was also moved ahead of the IWD() initializer which
ensures that IWD doesn't scan before the rule can be set/processed.
This test occationally fails due to no hwsim rules. Basically we
were just expecting iwd to connect to one of 3 access points but
the ranking was equal, so it chose the first in the scan list.
Now a signal strength is assigned to each AP to steer IWD into
connecting to the expected AP.
As with other tests, wait on device state instead of the network
object. The connectivity test was also changed to not check for
group traffic since AP does not negotiate the IGTK at this time.
There were a number of fixes here. The waits were changed to wait
on the device state instead of the network state and hwsim rules
were removed after the test as to not interfere with future tests.
One of the rules was setting the signal to -10000 wich was causing
the ranking to be zero.
Updated testFT-SAE-roam to use the TestContext APIs as well as
fixed the failure which was introduced after requiring stricter
AKM logic for SAE networks. The new failure was due to the hostapd
config not including the standard SAE AKM which is actually
required by the spec.
Slower systems may not be able to make some timeouts that tests
mandated. All timeouts were increased significantly to allow tests
to pass on slow systems.
It is not safe to assume that the python dbus implementation will
wait for a method to return. The documentation says this with
respect to reply_handler/error_handler:
"If both are None, the implementation may request that no reply is sent"
To stay on the safe side we should always include the error/reply
handlers and wait for the operation to complete.
test-runner now supports interface name replacement inside hostapd
config files. Since a given test configuration doesn't know what
interface names there will be $ifaceN can be specified instead e.g.
rsn_preauth_interfaces=$iface0 $iface1
The $ifaceN values will be replace with actual interface names when
the test is started.
This patch also removes ctrl_interface inside the hostapd config
files as this is no longer required.
This test was unreliable since it was assuming a periodic scan would
happen at just the right time. Instead since we are expecting autoconnect
we can just wait for DeviceState.connected then after we are connected
verify the network was correct.
This test was never 100% reliable, and after the test-runner re-write
it became extremely unreliable. The issue came down to the very common
block of code thats present in many tests where we wait for obj.scanning
then not obj.scanning. This is fine when a dbus scan() is explicitly
done before, otherwise it could lead to problems. Without a dbus scan
explicitly called we are assuming a periodic scan will happen. If it
already happen the initial wait for obj.scanning will never return and
time out.
This probably needs to be changed in several tests, but for this specific
case we can remove the waits completely. Since
check_autoconnect_hidden_network has a 30 second wait on
DeviceState.connected this will ultimately time out if anything goes
wrong. There isn't any great reason to wait for scanning (for this test
specifically).
A minor style change was also made when initializing IWD. The values
passed in this test are now the default, so no arguments need to be
passed.
iwd.py was updated to use the TestContext APIs to start/stop
IWD. This makes the process managment consistent between starting
IWD from test-runner or from the IWD() constructor.
The psk agent is now tracked, and destroyed upon __del__. This is
to fix issues where a test throws an exception and never
unregisters the agent, causing future tests to fail.
The configuration directory was also chaged to /tmp by
default. This was done since all tests which used this used /tmp
anyways.
The GLib mainloop was removed, and instead put into test-runner
itself. Now any mainloop operations can use ctx.mainloop instead
Before hostapd was initialized using the wiphy_map which has now
gone away. Instead we have a global config module which contains
a single 'ctx'. This is the centeral store for all test information.
This patch converts hostapd.py to lookup instances by already
initialized Hostapd object. The interface parameter was removed
since all tests have been converted to use config= instead.
In addition HostapdCLI was changed to allow no parameters if there
is only a single hostapd instance.
This patch completely re-writes test-runner in Python. This was done
because the existing C test-runner had some clunky work arounds and
maintaining or adding new features was starting to become a huge pain.
There were a few aspects of test-runner which continually had to
be dealt with when adding any new functionality:
* Argument parsing: Adding new arguments to test-runner wasn't so
bad, but if you wanted those arguments passed into the VM it
became a huge pain. Arguments needed to be parsed, then re-formatted
into the qemu command line, then re-parsed in a special order
(backwards) once in the VM. The burden for adding new arguments was
quite high so it was avoided (at least by me) at all costs.
* The separation between C and Python: The tests are all written in
python, but the executables, radios, and interfaces were all created
from C. The way we solved this was by encoding the require info as
environment variables, then parsing those from Python. It worked,
but it was, again, a huge pain.
* Process management: It started with all processes being launched
from C, but eventually tests required the ability to start IWD, or
kill hostapd ungracefully in order to test certain functionality.
Since the processes were tracked in C, Python had no way of
signalling that it killed a process and when it started one C had
no idea. This was mitigated (basically by killall), but it was
no where close to an elegant solution.
Re-writing test-runner in python solves all these problems and will
be much easier to maintain.
* Argument parsing: Now all arguments are forwarded automatically
to the VM. The ArgParse library takes care of parsing and each
argument is stored in a dictionary.
* Separation between C and Python: No more C, so no more separation.
* Process management: Python will now manage all processes. This
allows a test to kill, restart, or start a new process and not
have to remember the PID or to kill it after the test.
There are a few more important aspects of the python implementation
that should now be considered when writing new tests:
* The IWD constructor now has different default arugments. IWD
will always be started unless specified and the configuration
directory will always be /tmp
* Any non *.py file in the test directory will be copied to /tmp.
This avoids the need for 'tmpfs_extra_stuff' completely.
* ctrl_interface will automatically be appended to every hostapd
config. There is no need to include this in a config file from
now on.
* Test cleanup is extremely important. All tests get run in the
same interpreter now and the tests themselves are actually loaded
as python modules. This means e.g. if you somehow kept a reference
to IWD() any subsequent tests would not start since IWD is still
running.
* For debugging, the test context can be printed which shows running
processes, radios, and interfaces.
Three non-native python modules were used: PrettyTable, colored, and
pyroute2
$ pip3 install prettytable
$ pip3 install termcolor
$ pip3 install pyroute2
The tests basically remained the same with a few minor changes.
The wiphy_map and in turn hostapd_map are no longer used. This
was already partially converted a long time ago when the 'config'
parameter was added to HostapdCLI. This patch fully converts all
autotests to use 'config' rather than looking up by interface.
Some test scripts were named 'test.py' which was fine before but
the new rewrite actually loads each python test as a module. The
name 'test' is too ambiguous and causes issues due to a native
python module with the same name. All of these files were
renamed to 'connection_test.py'.
First, looking for DeviceState.connected gives a much better indication
if we are actually connected vs the connected property on the network
object. Second, its good practice to also check that hostapd sees that
the station is connected.
Restarting hostapd from python was actually leaking memory and
causing the hostapd object to stay referenced in python. The
GLib timeout in wait_for_event was the ultimate cause, but this
had no come to light because no tests restarted hostapd then
used wait_for_event.
In addition, any use of wait_for_event after a restart would
cause an exception because the event socket was never re-attached
after hostapd restarted.
Now we properly clean up the timeout in wait_for_event and
re-initialize the hostapd object on restart.
Many tests force a reauth after the initial connection. When the tests
were written there was no way of ensuring the reauth completed except
waiting (IWD.wait()). Now we can wait for hostapd events in the tests,
which is faster and more reliable than busy waiting.
This test was not reliably passing. Busy waiting is not really reliable,
but in this specific case its really the only option as the blacklist
must expire based on time.
