A bot's browser you can watch, drive and replay

Since 0.11.0 you could watch a bot's browser beside the conversation and take it over. What you could not easily see was what the bot had just done, and once the page was yours you had clicks and typing. In 0.11.3 the browser is a canvas tab, and both halves are finished.

What the bot did, drawn where it did it

Each thing the bot does is drawn on the live page: a ripple where it clicked, an outline round the element, a line along the top while a page loads. A list beside the page says the same in words, newest first: "Clicked “Sign in”", "Typed into “Search”", "Opened example.com".

The names come from what labels an element, never from what is in it. An unlabelled text area would otherwise be named by its contents, and the live view must not show what was typed. Card numbers and token-shaped names are dropped. When nobody is watching, nothing is measured.

With reduced motion on, the marks still appear, as a fade. The app's reduced-motion rule cuts every animation to nothing, and here that would cut information.

Take Over means the whole page

After Take Over you have an address bar, Back, Forward and Reload, right, middle and double clicks with modifiers, drags, sideways scrolling, key chords and paste. Two limits are enforced, not suggested: only http and https addresses open, and chords that would close the browser or reach your computer (Ctrl+Alt+Delete, Ctrl or ⌘ with W or Q, Alt+F4, app switching) are never sent.

You may open any address, not only the sites the bot is allowed, because a sign-in bounces through hosts nobody listed. The list limits where a bot can be steered, not where you can go. After Hand Back, the bot's first call leaves any page it is not allowed on.

A bot that asks instead of guessing

A bot that reaches a sign-in, a one-time code, a CAPTCHA or a payment can now stop and ask for you. Its browser opens on the request with its reason. Take Over, do the step, Hand Back, and it carries on, told the page's title and address. Not Now tells it to find another way. A request nobody answers expires after ten minutes, and an unattended run, which has nobody to hand the page to, is refused at once.

Since 0.11.0 a bot never types a card number anywhere. Now it has something better to do at a payment page than give up.

Replay

A run that browsed keeps a picture of the page after each step. Replay the Browser on a finished run steps through them like a filmstrip, with what each step did in words and the call as it was stored, redacted where it was redacted at the time. Left and right step through; Space plays.

How a bot's browser works, and why it never handles your password, is in the app docs.