The Apple Experience Augmented — Part 1: User Experience

· Concepts, Product, UX · Patrick Smith

I want to cover what the user experience of an AR Glasses product from Apple could look like, and how it might integrate with today’s products. First, let’s survey Apple’s current devices and their technologies for input & output:

iPhone input methods

  • Multitouch: use your fingers naturally for UI interactions, typing text, drawing, scrolling
  • Voice: request commands from Siri such as change volume or app-switching, dictate instead of typing

iPad input methods

  • Multitouch: use your fingers naturally for UI interactions, typing text, scrolling, drawing
  • External Keyboard: faster and more precise typing than multitouch
  • Pencil: finer control than multitouch, especially for drawing
  • Trackpad: finer control than multitouch, especially for UI interactions
  • Voice: request commands from Siri such as changing volume or app-switching, dictating text

Mac input methods

  • Keyboard: dedicated for typing text, also running commands
  • Trackpad / Mouse: move and click pointer for UI interactions, scrolling, drawing
  • Function keys: change device preferences such as volume, screen/keyboard brightness; app-switching
  • Touchbar: change volume, screen/keyboard brightness; enhances current app with quick controls
  • Voice: request commands from Siri such as changing volume or app-switching, dictating text

Apple Watch input methods

  • Touch: use your fingers for UI interactions, awkward typing text, scrolling
  • Digital Crown: change volume, scrolling, navigate back
  • Voice: request commands from Siri such as changing volume or app-switching, dictating text

AirPods input methods

  • Voice: request commands from Siri such as changing volume, dictating text
  • Tap: once to play/pause, double to skip forward, triple to skip backward

So what would a rumoured Apple Glasses product bring?

Apple’s Design Principles

Deference to Content

The most sparse approach might be to rely on voice for all input. Siri would become a central part of the experience, and be the primary way for switching apps and changing volume and dictating text. Siri currently can be activated from multiple devices, such as personal hand-held devices such as iPhone or shared devices such as HomePod. So it makes sense that the glasses would augment this experience, providing visual feedback that accompanies the current audible feedback.

Contrast Siri’s visual behaviour between iPadOS 13 and 14:

The Siri screen showing that Siri is asked to “Set an alarm for 8 a.m.,” and in response, Siri replies “The alarm’s set for 8 AM.” A notification from the Clock app shows that an alarm is turned on for 8:00 a.m.
Siri in iPadOS 13 takes over the entire screen
Siri on the Home Screen. A notification from the Clock app shows that an alarm is turned on for 8:00 a.m. A button at the bottom right of the screen is used to continue speaking to Siri.
Siri in iPadOS 14 layers discretely over your screen with a compact design

This provides a glimpse of the philosophy of the Apple Glasses. Instead of completely taking over what the user current sees, Siri will augment what you are currently doing with a discrete compact design.

This also relates to the Defer to Content design principle that has been present since iOS 7 which was the opening statement from Apple’s current design leadership. So we can imagine a similar experience with the Glasses, but where the content is everything the user sees, whether that’s digital or physical.

Content from a traditional app could be enhanced via augmentation. A photo or video in a social media feed might take over the user’s view, similar to going into full screen. Text might automatically scroll or be spoken aloud to the user. Content might take over briefly, and then be easily dismissed to allow the user to get back to their life.

Widgets such as weather or notifications such as received messages might be brought in from the outside to the centre. I can imagine a priority system from the viewer’s central vision to the extremes of their field-of-view). Content could be pinned to the periphery and be glanced at, while periodically in the background it receives updates.

If worn together with a set of AirPods, an even more immersive experience would be provided, with the AirPod’s tap input for playing and skipping. The active noise cancellation mode would probably pair well with a similar mode for the glasses, blocking the outside world for maximum immersion. Its counterpart transparency mode would allow the user to reduce the audible and visual augmentation to a minimum.

Clarity

So with a Glasses product, what is the content? It’s the world around you. But what if the world sometimes is an iPhone or Mac you use regularly through your day? Do the Glasses visually augment that experience?

With AirPods you can hop from an iPhone to a Mac to an iPad, and automatically switch the device that is paired. Wouldn’t it make sense for the AirPods and Glasses to perform as synchronised swimmers and pair automatically together to the same device that someone decides to use?

Can the Glasses recognise your device as being yours and know it precise location in the Glasses’ field-of-view? That sounds like what the U1 chip that was brought to iPhone 11 would do, as 9to5Mac describes it “provides precise location and spatial awareness, so a U1-equipped device can detect its exact position relative to other devices in the same room.”

Perhaps instead of tapping your iPhone screen to wake it, you can simply rest your eyes on it for a moment and it will wake up. The eyes could be tracked by the Glasses and become an input device of their own. If precise enough they could move the cursor on an iPad or Mac. The cursor capabilities of iPadOS 13.4 brought a new design language with UI elements growing and moving as they were focused on, and subtly magnetised to the cursor as it floated across the screen.

The cursor becomes the object of focus.

Similar affordances could allow a Glasses user’s eyes to replace the cursor, with the realtime feedback of movement and size increase enough to let the user know exactly what is in focus. The Mac might not need touch if the eyes could offer control.

In the physical world, a similar effect to Portrait mode from iPhone could allow objects in the world to also be focused on. The targeted object would remain sharp, and everything around it would become blurred, literally putting it into focus.

AirTags could enhance physical objects by providing additional information to their neighbour. Instead of barcodes or QR codes, the product itself could advertise its attributes and make it available for purchase via Apple Pay.

Use Depth to Communicate

If the Glasses not just show you the world around you but see the world around you, then your hands gesturing signals in the air could also be a method of input. Simple gestures could play or pause, skip ahead or back, change the volume. The could also be used to scroll content or interact with UI seen through the Glasses.

These gestures would close the loop between input and output. The iPad’s multitouch display works so well because of direct manipulation: your fingers physically touch the UI your eyes see. As your fingers interact and move, the visuals move with it. The two systems of touch input and flat-panel-display output become one to the user. Hand gestures would allow direct manipulation of the content seen through the Glasses.

Speculated Apple ‘Glasses’ input methods

  • Voice via AirPods or nearby device: request commands from Siri such as changing volume, dictating text
  • Eyes: interact with devices that have a cursor, focus on elements whether digital or physical
  • Air Gestures: use your hands for UI interactions, scrolling, changing the volume, playing, pausing, skipping
  • U1: recognise nearby Apple devices and interact with them
  • Plus whatever device you are currently using (if any)

So the Glasses could offer a range of novel input methods from a user’s eyes to their hands, or it could simply rely on the ubiquitous voice-driven world that most Apple devices now provide. The U1 chip seems to hint at an interaction between Glasses and hand-held device, perhaps modest like simply recognising it, or perhaps augmenting its input and output allowing a new way to interact with iPhones, iPads, and Macs. The Glasses accompanies what the user already sees and interacts with every day, enhancing it visually but deferring to the outside world when it needs. It could offer an immersive experience for content such as video and games, or future formats that Apple and other AR-device-makers hope will become popular.

Vary variables not rules in CSS media queries

· CSS · Patrick Smith

A common pattern you’ll see when implementing responsive design with CSS is to redefine rules in media queries:

h1 {
  font-size: 2rem;
}

@media (min-width: 720px) {
  h1 {
    font-size: 2.5rem;
  }
}

@media (min-width: 1200px) {
  h1 {
    font-size: 3rem;
  }
}

Here we are writing our media queries mobile first. This means the styles for small devices (like phones) are defined first, such as our font size of 2rem for h1 elements. Then a wider device is targeted with a media query that increases the font size of h1 elements to 2.5rem when the viewport width is 720px or greater. And then another one, increasing to 3rem for viewports 1200px or wider.

So what are the issues with this approach?

First, these media queries can get bloated. You’ll find that you’ll define your base rules first using a mobile-first approach, and then repeat those rules with overrides within media queries. You could put all of these rules for the same media query together underneath, or could you have the media query override live next to its mobile-first rule and rely on some sort of CSS optimiser or compression so the repetition doesn’t contribute to an increase in file size.

Second, knowing what to override can lead to complex code. Perhaps you only need to override the rules for nav a and not for nav elements within your media queries. These specifics lead to harder to understand code.

nav {
  background-color: lightskyblue;
}
nav a {
  display: block;
  padding: 0.5rem;
}

@media (min-width: 720px) {
  /* We override the rule for `nav a` but not for `nav` */
  nav a {
    padding: 1rem;
  }
}

Perhaps you used quite complex selectors that now need to be repeated for each media query. SCSS allows the media query to be written nested inside the rule, avoiding the need to repeat it as an author. But this still adds a layer of complexity that can make understanding which rule applies in what case that much harder.

Here’s the above two examples but written to take advantage of SCSS’s nesting of media queries:

h1 {
  font-size: 2rem;

  @media (min-width: 720px) {
    font-size: 2.5rem;
  }
  @media (min-width: 1200px) {
    font-size: 3rem;
  }
}

nav {
  background-color: lightskyblue;
  
  a {
    display: block;
    padding: 0.5rem;

    @media (min-width: 720px) {
      padding: 1rem;
    }
  }
}

The result is very direct — these rules have these specific media query overrides. But there a lot of indentation going on, and things get hard to read once you add more rules and more overrides.

In summary, by redefining our rules inside media queries they have a lot of responsibility. They contain knowledge about our selectors. They override some rules but usually not all of them.

Varying CSS custom properties

So what’s the alternative? If we start using custom properties (aka CSS variables), which we can do if we no longer support Internet Explorer, then our media queries can take on a lot less responsibility and thereby be a lot simpler.

/* Define our variables, with mobile-first values */

:root {
  --heading-1-size: 2rem;
  --nav-link-padding: 0.5rem;
}

/* Override just the variables for different viewports */

@media (min-width: 720px) {
  :root {
    --heading-1-size: 2.5rem;
    --nav-link-padding: 1rem;
  }
}

@media (min-width: 1200px) {
  :root {
    --heading-1-size: 3rem;
  }
}

/* Define all our rules, consuming those variables */

h1 {
  font-size: var(--heading-1-size);
}

nav {
  background-color: lightskyblue;
}
nav a {
  display: block;
  padding: var(--nav-link-padding);
}

Making our variables responsive lets our rules be a lot simpler. Everything is flatter without the multiple levels of indentation. And we don’t need to use SCSS to attempt to reduce repetition, we can just stick with CSS!

This works not only for viewport width checks but also for media queries that check for dark mode or reduced motion or which features are supported.

:root {
  --text-color: white;
}

@media (prefers-color-scheme: light) { /* We respect a user's browser preference */
  :root {
    --text-color: black;
  }
}

[data-mode=light] { /* We allow users to switch light/dark mode */
  --text-color: black;
}

p, h1, h2, h3, h4, figure, ul, ol {
  color: var(--text-color);
}

Our CSS rules can be none the wiser to what value a --text-color is under the hood, which is set to white by default and black when the user prefers a light color scheme. Our variables vary and our rules stay the same.

Tips for concatenating URLs in JavaScript

· JavaScript · Patrick Smith

You might have a base URL from configuration, that you then concatenate with paths to produce a full URL. Say an API URL you fetch from.

But should you provide the base URL with or without a trailing slash? e.g. Should you have https://api.example.org/ or https://api.example.org?

If you simply combine the strings together, you can end up with double slashes:

const baseURL = 'https://api.example.org/';

const productsURL = baseURL + '/products';
// 'https://api.example.org//products'

We could try and check for the double slash, but that’s a pain:

const productsURL = baseURL +
(baseURL.endsWith('/') ? 'products' : '/products');

Would you just rather not worry about this? Turns out we can with the built-in URL class. It’s supported in all modern browsers (not IE).

const baseURL = 'https://api.example.org/';

const productsURL = new URL('/products', baseURL).toString();
// 'https://api.example.org/products'

Isn’t that better? It’s also available in Node.js and Deno, so we can use the same technique everywhere.

Tim Bray on subscribing to publications

· Product · Patrick Smith

Tim Bray writes why he won’t subscribe:

Their arithmetic didn’t consider their chance of getting me to click on “Subscribe.” In my particular case, that chance is almost exactly Zero. I subscribe to enough things and I am acutely reluctant to give anyone else the ability to make regular withdrawals from my bank account. … It’s exactly because I’ve done some subscribing that I’m just not gonna do any more.

Biasing your business to producing value in a public manner

· Marketing · Patrick Smith

Some ancient 2011 advice from Patrick McKenzie:

Businesses create value with almost everything they do.  The lion’s share of the value is, like an iceberg, below the waterline: it cannot be seen from anywhere outside the business.  Create more value than you capture, certainly, but get the credit for doing so, both from Google and from everybody else.  This means biasing your business to producing value in a public manner.  Did you write software not core to  your line of business?  Great, OSS it.  Get the credit.  Have you learned something novel about your customers or industry?  Write about it.  Get the credit.  Are your business’ processes worthy of emulation?  Spread what you know.  Get the credit.

37Signals is amazing at this. You can do it, too.

Get good at SEO — You need more links. Create ways to justify people who aren’t in a commercial relationship with you linking to you anyway. My favorite way for doing this is getting the credit for things you do, as described above.

https://www.kalzumeus.com/2011/06/17/software-businesses-in-5-hours-a-week-microconf-2010-presentation-1-hour/

There are also slides for the talk.