Apple announced with iOS4 the ability to do automated UI testing on the iPhone as a part of the Instruments development tool. Such testing is done with the help of a number of JavaScript libraries which allow you to access an application on a device and tap buttons, scroll views and varify text as if it were a human interacting with the hardware, and not a script.

Why is this – other than the fact it’s something from Apple – cool? It essentially means that it is possible for us as app developers and testers to automate some of the tests we always do to ensure we have not broken anything in our latest development cycle. There is nothing worse than working hard on adding a new feature to an app, testing it and releasing it only to discover that your testing missed a regression in your previous functionality. This forces you to make a brown paper bag release to fix it, for which you have to test everything again in addition to the things not fully tested the first time. This takes up time and effort which could be used to add more features, and more importantly can lead to your users having a broken app.

Automating the tests does not mean that they are better tests than if they were done manually. What it does mean is that tester time is saved on doing the mundane day to day tests so that they can spend more of their time on testing edge cases and new features, where the harder to find bugs are more likely to be and less likely to be found by the developers as they write code.

What Airsource is doing with it

At Airsource we’ve been busy automating tests for our app, Optiscan. Doing this we have found that, despite what Apple have tried to do with using timeouts, it is sometimes necessary to have delays in the tests. The main shortcoming in this area is the fact UI Automation swallows taps on elements which are not visible; meaning that your tests script may expect a changed state and fail because the tap to change the state was never registered. The reason that timeouts do not work on this is that the element you are attempting to tap exists and valid, so it can be tapped, but it may not be visible (due to an animation sequence for example) thus the tap is never registered by the app. This is a big shortcoming in the system of timeouts, it would be great if Apple fixed this by attempting taps until the tap occurs on a visible element or the timeout is reached.

When you are writing a test you do not want to have to worry if the button is ready to be tapped or not, you just want to tap the button! So as a part of the testing effort we have written a bunch of useful little functions which encapsulate all the delay logic and mean than we don’t worry about delays when we’re writing the tests.

The code below allows us to scroll to an element with a particular name in a scroll view, before waiting for it to become visible and then tapping it. This simple encapsulation of a common task means that there is no need for the test writer to worry about the button becoming visible to tap before tapping it, just worry about how to actually test the app.

// Allows you to scroll to an element with a particular name and tap it.
function scrollToElementWithNameAndTap(scrollView,name)
{
	if (! (elementArray instanceof UIAScrollView))
	{
		throw ("Expected a UIAScrollView");
	}

	var e = elementArray.scrollToElementWithName(name);
	waitForVisible(e,5,0.25);
	e.tap();
}

// Poll till the item becomes visible, up to a specified timeout
function waitForVisible(element, timeout, step)
{
        if (step == null)
        {
                step = 0.5;
        }

        var stop = timeout/step;

        for (var i = 0; i < stop; i++)
        {
                target.delay(step); // for the animation
                if (element.isVisible())
                {
                        return;
                }
        }
        element.logElement();
        throw("Not visible");
}

Using JavaScript objects’ prototype property it is even possible to make scrollToElementWithNameAndTap(name) a method of each instance of UIAScrollView. Hence if you have a constructor for your tests you can use

UIAScrollView.prototype.scrollToElementWithNameAndTap = function(name){
                        scrollToElementWithNameAndTap(this,name)
                        };

and now you can call the method on all your UIAScrollViews just as if it were a native UI Automation method.

Doing more than it says on the tin

An unfortunate aspect of the UI Automation kit is that there is no reset button or method. The result of this is that if a test fails, then your app is in an unknown state, thus it is very likely that all subsiquent tests will also fail. Ideally you do not want it to appear as if 20 tests have failed when in fact only one has failed. Having the ability to target.reset() to reset the app to a known state would be very useful, and it is a pain that Apple have not implemented it in some form.

To remedy this we’ve had to add a small amount of code to our apps which makes it possible for us to interact with them on the level of the program, not just the UI level. UI Automation allows us to change the volume on the device using methods in UIATarget, so we’ve added a listener for the device volume to our app, which means we can run code when we detect a volume change. Code for a volume listener is shown below.

// Set up a listener to act on volume changes
AudioSessionInitialize(nil, nil, nil, nil);
AudioSessionSetActive(true);
AudioSessionAddPropertyListener(
                         kAudioSessionProperty_CurrentHardwareOutputVolume,
                         applicationVolumeDidChange,
                         self);

// Note that this callback will only be called if the mute button is off.
void applicationVolumeDidChange(void *inClientData,
                         AudioSessionPropertyID inID,
                         UInt32 inDataSize, const void *inData)
{
        NSLog(@"Volume changed");

        // Do something like reset the system
}

This way calling target.clickVolumeUp(); just after UIALogger.logFail(...) in your testing script will mean that you can reset the app before the next test is run. Note that it is important that the hardware mute button is off when you are testing otherwise no signals will be sent that the volume has changed!

Shortcomings

Other than the swallowed taps and lack of reset method mentioned above, there are a couple of other shortcomings with UI Automation. The most obvious being the lack of screenshots when running the tests on the simulator; this makes it hard to verify UI and record what the app’s state was when a test failed. To be honest it feels rather bizarre that you can take remote screenshots on the actual hardware iPhone, but not in the simulator. I hope that Apple fix this in future releases.

The less obvious, but rather annoying shortcoming (especially if you are building a large library of testing scripts) is the lack of a decent preprocessor. JavaScript does not have native support for preprocessing or any kind of #import structure, so it is good that UI Automation scripts support #import, but the lack of define statements and logic such as #define and #ifdef is a pain when you want only to run a certain initialisation script once, but it is called by each test you run. Emulation of this behaviour using JavaScript variables is not helped by the fact that variables can remain defined between different runs in Instruments.

Conclusions

Automated UI testing does not mean bug free code. No testing on any non-trivial piece of software can guarantee bug free code, and the testing is only as good as the tests themselves. Using automated testing means that it is easier to stop regressions by having a test in place for each previous bug which has cropped up, it also means that the tester’s time can be utilised more effectively. Apple’s UI Automation ticks many boxes for good automated testing despite its shortcommings, and we hope will prove an invaluable tool in our development cycle.

This value indicates the time relative to the device CPU time base register. Compare acceleration event timestamps to determine the elapsed time between them.

Assuming iPhone OS is similar enough to Mac OS X, it must be using mach_absolute_time():[1]

Uptime is the highest resolution (64-bits) timer on any PowerPC processor. It counts bus cycles (usually 1/4 the CPU speed) since the last reset (or (re?)boot). All other system timers and clocks are based off of this counter (including gettimeofday).

QA1398 appears to be the earliest[2] official documentation of mach_absolute_time() and gives two definitions of GetPIDTimeInNanoseconds(). The first relies on CoreServices’ AbsoluteToNanoseconds() which isn’t in iPhone OS; we’re interested in the second (casts added for a little extra clarity):

// Do the maths.  We hope that the multiplication doesn't
// overflow; the price you pay for working in fixed point.

elapsedNano = (uint64_t)elapsed * (uint32_t)sTimebaseInfo.numer
             / (uint32_t)sTimebaseInfo.denom;

On OS X/i386, it’s fine — the several timers are all converted to nanoseconds (this seems to miss the point entirely), so numer = denom = 1 and there’s no overflow.

On an iPhone 3G S, it ticks at 24 MHz and (on OS 3) mach_timebase_info() returns 1000000000/24000000.[3] The multiply overflows after 2⁶⁴/10⁹ ≈ 1.84e10 ticks — about 768.6 seconds or 12.8 minutes. Bingo.[5]

But UIAcceleration.timestamp is a double — so it’s converting timestamps to nanoseconds, overflowing, and dividing by a billion (or multiplying by 1.0e-9) to get the timestamp in seconds. I’m not sure why they do all this; I’d just calculate numer/(denom*1.0e9) at startup and multiply timestamps by it. It’s also pretty easy to convert to nanoseconds without unnecessary[6] overflow:

nanos = elapsed/denom*numer + elapsed%denom*numer/denom;

I ended up fixing the bug by throwing away accelerometer updates (but storing the new timestamp) when time has gone backwards. Who cares if you skip an accelerometer update every 12 minutes?

Afterword

I’m pretty sure that mach_absolute_time() is still the “best” way to measure timestamps — it’s existed since Mac OS 10.0 and seems to be a highly stable API. CoreAnimation and GCD mention it, along with QA1643 which suggests either using it directly or inlining it. Since iOS 4, there’s also CVGetCurrentHostTime() which the docs say is equivalent to the CoreAudio timebase (and thus mach_absolute_time). There’s a reasonably accurate article with more details (but I’d divide by 10⁹ instead of multiplying by 10⁻⁹).

For completeness, there’s also AudioQueueDeviceGetCurrentTime() which requires an audio queue and returns a lot of extra things you probably don’t care about. In Snow Leopard/iOS 4 there’s [[NSProcessInfo processInfo] systemUptime]. AudioGetCurrentHostTime() exists but is “private” due to the lack of an iPhone OS header (the Mac OS X header says since iPhone OS 2).

I used to be lazy and use CFAbsoluteTimeGetCurrent(), but since iOS 4 it can change arbitrarily (due to app backgrounding, NTP/cell network updates, or leap seconds). [NSDate timeIntervalSinceReferenceDate] is equivalent, [NSDate date] is much slower (~4000 CPU clocks!). Moreover, both seem to round to the microsecond (presumably they’re based on gettimeofday()).

1. UpTime() is the Mac OS 8 name for mach_absolute_time(). On a PowerPC, it retrieves the value of the timebase register using the mftb and mftbu instructions. It’s not always the “bus clock”; my iBook G4 says it runs at 18431630 Hz, which doesn’t correspond to any bus unless the CPU has a 57.5× multiplier (apparently it’ll be a 18.432 MHz UART clock, despite the lack of a serial port). As far as I know, ARMs don’t usually have a time base register; mach_timebase_info() on a 3GS takes over 700 CPU clocks, consistent with using an external device.
2. Launch Time Performance Guidelines: Measuring Launch Speed: Using Explicit Timestamps has been around for longer and says mach_absolute_time reads the CPU time base register and is the basis for other time measurement functions, but it’s unclear which revision it was added in and is more of a brief mention than documentation. Interestingly, it’s the only other document mentioning “CPU time base register”.
3. The iPhone and iPhone 3G use 6 MHz. The iPhone 3G S, iPhone 4, and iPad use 24 MHz. The iPod Touches presumably are the same as the corresponding iPhone. On OS 4, mach_timebase_info() reduces the fraction, returning 125/3 on a 3G S, postponing overflow for 195 years, and making it difficult to tell whether UIAcceleration has been fixed.[4]
4. UIAcceleration.timestamp values are still all close to a nanosecond boundary, so it still seems to convert to nanoseconds first. Presumably it’ll break noticeably on devices with an odd clock frequency (33.333333 MHz?). It’s also odd that it’s not fixed given the existence of -[NSProcessInfo systemUptime] which returns a double without nanosecond-rounding.
5. It shouldn’t be difficult to figure out how many times it’s overflowed and thus get the corresponding mach_absolute_time() value, but that just seems like too much effort and is likely to break across releases.
6. If the interval is more than 2⁶⁴ ticks, the input has overflowed. If the interval is more than 2⁶⁴ nanoseconds, the output must overflow (or be clipped, or something). Otherwise, it’ll work; the “price you pay” for integer arithmetic is just that you need a little more care to prevent intermediate overflow. It’s “fixed point” either.
7. Interestingly, CMSampleBufferGetSampleTimingInfo() says that frames from AVCaptureVideoDataOutput use a nanosecond timescale, despite the lack of a nanosecond clock.

We’ve had a note from Greg at SET Japan: their latest designer QR code promotional video features Optiscan, which seems to scan it better than the competition. Check it out: http://youtube.com/watch?v=M5lAT3gVzFc.

Also Optiscan is now featured on the readers page by MSKYNET, which has a rather nice online QR code generator. They “rebrand” QR codes as sparqcodes in the US.

We will only license the library for use on the App Store if the intended use does not directly compete with Optiscan. To request additional details or an evaluation package, send details of your intended use to optiscan-licensing@airsource.co.uk.