By January 28, 2020 February 29th, 2020 No Comments

When I tried to leave for work, the front door was locked.

I jiggled the handle up and down a couple of times while my brain caught up to my muscle memory. Huh. Patrick was supposed to unlock the door when my watch came within two metres. In the months since I upgraded, that feature had worked seamlessly. It was so slick that on the first day I questioned whether the door was locking at all. I had tried to open it with my watch off just to make sure.

“Hey Patrick, unlock the front door,” I said softly in that flat way you talk to assistants. It felt odd to ask him that – I had gotten used to him being two steps ahead of me. I checked my watch. 9:35AM.

“Sorry, Finn, there’s a safety threat outside. I’ll unlock the door as soon as it’s safe.”

That delivered a tiny hit of adrenaline. A safety threat? I checked the door cam for signs of trouble, but didn’t see anything concerning. The screen showed the usual empty, less than clean hallway. Walking to the living room window, I peered down to the street. Everything seemed normal. People bustled by on their personal trajectories, forming a current in aggregate. Patrick could be overly cautious – a week ago he asked if he should call 911 because there was a drunk man in the hallway. He was still learning that humans, especially female humans, have to be comfortable in the omnipresence of danger.

“What is it?” I asked.

“A faulty cable.” A cable? He really was a nervous one…

“…In the door?” I asked.

“Your door doesn’t have a cable, Finn,” he chimed pleasantly. This has to be a bug, I thought. I wouldn’t be able to get to the bottom of it until after work though. It was 9:36, and I was officially running late.

I decided to take my chances. “Open the door anyway, Patrick,” I muttered.

“Sorry, Finn, I’m not allowed to do that. There’s a safety threat outside. I’ll unlock the door as soon as it’s safe.” His calming tone was patronizing. I was going to miss my bus.

I pulled my phone out of my purse and tried manually unlocking the door. A popup on the screen echoed Patrick and wouldn’t let me proceed. I rattled the door handle in vain, letting out an audible moan. Patrick quietly turned on the Spa playlist I listen to when I read in the evenings. It helped, I admitted begrudgingly to myself, noticing my body relax. Then I saw 9:37 on my watch and retensed. If I missed my bus, I would miss the 9:50 subway and be late to the morning briefing. Gabriel would condescend, even though I hadn’t been late once in the two years since we joined the same management training cohort.

“Override front door lock,” I said in a commanding tone, aping sci-fi movies I’d seen. I was pretty sure that real assistants didn’t work that way, but it was worth a try. Patrick repeated his spiel. I checked the door cam and window again. “There’s nothing out there, Patrick!” I whined. “I’m confident that there is a faulty cable,” he replied earnestly. How can computers be so smart and so stupid simultaneously? Regardless, I surrendered to the fact that I wouldn’t be leaving until his “threat” had dissipated. I marched over to the couch, threw myself down with a huff, and stared fixedly at the door.

When I upgraded, there was a lot of buzz surrounding the new Patrick. He used a cutting edge “universal” algorithm, as opposed to the patchwork of specific task algorithms that make up other assistants. The new algorithm let him learn things without being explicitly programmed for them. He was simply integrated into your home and given the objective of making you comfortable (with some constraints, of course). It was a huge improvement from the previous version. Within a week, Patrick was brewing my morning coffee just in time for when I arrived in the kitchen, and responding to routine emails for me. He adapted to variations in my schedule deftly. The shower always turned on 30 seconds before I was ready to step in, even when I woke up late in the morning after a night out. The temperature as I moved from room to room was consistantly perfect, while my latest power bill had been much lower than usual.

Until this morning I hadn’t encountered a single bug with Patrick. I was so impressed that I’d been preaching his merits to all my friends, pushing nobly past the point of becoming tiresome. But being locked in my apartment like this was unnerving. I knew it was unreasonable, but I detected a faint seed of panic in the back of my mind. I inhaled deeply and focused on the spa music. Patrick crossfaded in a guided meditation.

At 9:41, I heard the mechanism of the lock move. “It’s safe again, Finn. Sorry about the wait,” Patrick announced. Before he finished his sentence I was out the door and half-running toward the stairwell. I didn’t have time to wait for the elevator. As I hurried down the stairs, I rummaged through my purse for my phone, found it, and held my thumb on the Trolley icon to summon a car. If I was going to make the subway, I’d have to pony up the cash to take the tunnels there. Underground highway robbery… I landed on the ground floor, crossed the lobby, and pushed out onto the street. The car was just pulling up, its door sliding open in time for me to step in without breaking my stride. I sat down in one of the forward facing seats and we glided into the flow of traffic. Breathing heavily, I set my route in the app – the arrival time read 9:50. If the traffic gods blessed me, I could still make it.

As the car maneuvered towards the tunnel, I pulled up that week’s notes on my phone and started reviewing for the briefing. There had been some big developments over the last few days and I wasn’t sure how they would come together. I tried imagining the different scenarios, preparing as much as I could for each. I kept my head down like that until I sensed we were nearing the 12-car tunnel elevator, then looked out the window to size up the congestion. It was 9:46, but I could see one more spot on the platform as we approached. I was going to make it. But just as I started to relax in my seat, the car suddenly braked. Another Trolley car appeared out of nowhere and slid into the space. “Come on!” I yelled at the passenger, louder than I intended. They startled in their cabin and looked around for the source of the noise. Immediately embarrased, I looked down at my lap – they couldn’t control the fleet any more than I could. They obviously just had Plus. My heart sank as the elevator descended below street level. It would be about a minute before it came back up for the next batch – more time then I had. The Trolley app updated my ETA to 9:51. I could perfectly visualize Gabriel’s poorly contained grin.

I wallowed in self-pity during the 10:00 trainride downtown, listening to my Sad playlist and watching the concrete blur past the windows. When we pulled into the station, I stepped briskly off the train and up the stairs to street level. My office building was only a block away, and I closed the distance in record time. As I entered the lobby, the vintage elevator doors were just closing, forsaking me at ground level. I rolled my eyes at my luck and stood back, waiting for the carriage to climb it’s way up and back down the building. I watched the floor numbers light up in sequence above the safety doors.

When it got to the 18th, I heard a heavy SNAP from inside the shaft, then another. Seconds later, a sickening impact that travelled through the concrete and shook my body.

When I requested the logs, it took a couple weeks for the Patrick Company to get back to me, but eventually they did. The file was 31MB of plain text. The mass of activity Patrick did behind the scenes took me off guard. For every item that seemed directly linked to my interactions with him, there were a thousand more that didn’t. He crawled the social media pages of each of my contacts multiple times a day. He looked up the manufacturer of any new thing I bought. When I took a vacation to Spain last month, he went down a rabbit hole researching European history. He was constantly building out the context of my life in his model.

At first, the only thing I could find related to my near death experience was Patrick accessing the Trolley API at 9:31 that morning, but it turned out I wasn’t looking far enough back. I had been combing through the logs for 3 hours when I finally found what I was looking for – 10 seconds of activity almost 4 months ago. Here’s what he did in that time:

  • Accessed the safety inspections for my office building on an unlisted page of the city’s website
  • Reviewed footage from the lobby and elevator’s CCTV streams (they were private but unencrypted)
  • Looked up the specific make of the elevator on a specialty site dedicated to vintage elevators
  • Researched woven steel cables
  • Looked at my historical location data and step count from 9:35AM (when I leave for work) to 9:56AM (when I get there)
  • Looked at my Trolley ride history
  • Performed a complex equation I couldn’t follow
  • Checked my calendar for 121 days in the future, March 4
  • Looked up the subway schedule for March 4
  • Scheduled the door to emergency lock on March 4 from 9:33:42AM to 9:41:11AM


This story was inspired by something I read in Stuart Russell’s book, Human Compatible. He talks about how humans can’t behave rationally because life is too complex. We can’t truly optimize for our goals because it’s impossible for us to predict the way things will pan out with so many interacting factors. We can’t even predict the way a chess move will pan out. But computers are much better at chess than us. It’s a type of computation they are better suited for, while we’re better suited for other things (for now). So, I imagined a scenario where an AI assistant could accurately predict a catastrophe by modelling the interactions of the different factors in a way our brains can’t. I think this would be possible for a specific type of situation with relatively few interacting factors, like in the story.