This summer ⛱ we paused our current development for one week to focus on innovation. This focus was an opportunity to invest in both our engineering, product, and design staff, as well as invest in future product ideas. We worked on new product ideas to extend our product’s use cases, applications for internal tools, new products to make our own workplace better, improving our current technology, and improving user experiences, e.g. accessibility.

Why stop forward development to experiment? Well, work should be fun, creativity should be common, a good idea 💡 can come from anyone. So, why not build something we believe will be great?

Orchestrating a hackweek ⚒

Our staff pitched their ideas/projects then organized by teaming up based on personal interest. We posted internal help wanted notices on a shared spreadsheet. Next we teamed up and joined project channels in our Slack chat. To create time to hack, we paused our current work in progress, likewise our scheduled ceremonies. This provided the freedom to hack on the ideas which our people believe will make a better future for our mission, to challenge the status quo of workplace technology. We focus on creating a better and safer workplace; and deliver solutions for visitor and employee experiences in the office that stand out. So our aim was to challenge ourselves to build and demo as many prototypes of something awesome we can, in about a week.

How can our technology make a difference? Can we expand upon our platform to transform mundane tasks and even make complex tasks more efficient? Our application services for mobile apps and back office tools include a variety of software choices, we experimented to discover where we might fill gaps and expand our capabilities. We asked ourselves… Can we do better with data? Can our complex and automated solutions get faster and more accurate? Do we have any use cases that surround our existing solutions that we should explore? What can we quickly prototype or even build a MVP for? What’s missing in our office? (Lets build an app for that.)

At the end of the week we demoed 🎬

• 😲 Improvements to streamline our visitor sign-in with TouchId/FaceID.
• 🏭 A new application for tracking facilities maintenance tasks.
• 🚧 An icon repository for our graphic library.
• 🎡 A queues app for managing a wait list.
• 🗣 Improvements for accessibility of our dashboard application.
• 🎛 Machine learning to leverage patterns for improving OCR recognition.
• 🗓 Event registration and sign-in.
• 📷 Facial recognition capabilities to by-pass manual checkin.
• 🚢 A new deployment solution for our dashboard application.

…what a great demo this was 👏

A quote coined while demoing our A11y improvements:

“Workplaces must be accessible. So too must workplace technology.”
— Steve Szczecina

A few screenshots from our live demo to close out the week…

As the week finished, a few questions arose…

• What can we ship?
• What can we use internally?
• Which ideas can be segmented for beta testing by an existing customer, or even a new customer?
• How can we make work always feel like a hackweek?

We do not have all the answers yet, only time will tell. One thing we know for sure is — we want to do this again :)

When we take action through experimentation we discover that we can make the impact we strive for, to make workplaces better, safer, more fun, and do so with style. We will able to ship some of the internal projects so that we may learn more; and iterate. Also, we made plans to iterate upon some of the prototypes we demoed.

Since we innovate for the office, we strive to solve problems that meet our own interests. Likewise, we strive to develop solutions for the challenges that our customers experience. We are a distributed company, meeting together for our hackweek and demoing solutions was a rewarding experience. Our team had plenty of fun when we all met up at our headquarters in San Francisco, and it was a pleasure to connect and work together onsite. We rolled up our sleeves and built solutions not only for our own distributed office 🏢 but also for the workplaces of the tomorrow 🚀. We would love to ship some of the prototypes that we presented at the close of our hackweek — time will tell which become winning solutions as we iterate 📐and experiment ⚗.

A few observations from our experience hacking together…

What were your initial expectations for the hackweek?

- I figured that it would be fun 🎪, but was uncertain if we’d finish anything. What happens if we get pulled back on an issue for ongoing work? I was optimistic that we could build something cool to demo at the end of the week. I was not sure yet if I would have something valuable to contribute as a new member on the engineering team but hopeful to make a contribution.

- Building a half-working product that is 🐝 buggy and unusable.

What were the highlights, what stood out?

- The fact that all week long, ceremonies for organizing work were not really needed, we met as needed and at impromptu times of the day. We skipped formal processes in favor of just organizing ourselves. There was plenty to demo at the end of the week. I really liked the fact that many of the projects were something that we could actually put into play after the week was over. I was impressed by a mobile solution that simplified the steps users had to take by using features baked into the OS which created a streamlined and more natural experience. The efforts made on the dashboard project to apply changes to make the application more accessible to users was great, a demo with voice-over 🎙 helped us understand the experience of users who rely on voice over features to use a web application.

- Being able to see potential of our teams — building 🏗 a new features in a week, explore different solutions for a goal.

What made you laugh?

- During a demo an engineer mentioned “when you have a problem…” and at the same time the camera 🎥 used in the demo was pointed at a specific individual, the presenter started giggling and the crowd broke out in laughter 🤣. Not because the person was problematic, instead that as a group the team already knows how to work through problems and this individual proves that point.

- 🐶 Dogs in the office. :D

What did you enjoy about the demos, or team effort?

- I was fortunate to be on a team with a designer, product lead, API engineer and an iOS engineer. I contributed to the user interface portion of the feature. I was impressed that, as a newly formed team, we were able to identify the specific outcome we wanted to demo — even an ambitions scope of work. We were able to demo a full experience that involved coordination between multiple devices 📱 and users. It was fun to explore the possibilities of what we could deliver as a prototype in only a few days. As a new engineering team member, it was a pleasure to work with people I have not yet had the privilege to work aside yet. I enjoyed the collaboration with individuals from our product and design teams. The variety of demos was impressive, from internal tools to solutions our which customers may benefit from.

- To see how far our team can go in one week.

Where there any risks you took and/or challenges you faced?

- I thought it was pretty risky to take on the scope of the prototype that we wanted to demo/present in only a few days. Along with other all-hands 🙌 activities 🍻 🍷 🍩 🍹 🎉 occurring the same week - we only had a few days of focus. I tried out a library for a small mobile web app which I had not used before. It appeared to have the tooling I needed to move quickly and provided testing as well (I favor test driven development). Our designer did a great job, I knew I may not have the time to apply all the polish that would make the design really shine. But I made the effort to keep up with the iterations of the designs; while also developing a solution.

- Not being able to socialize as much as want to for all-hands week because I was focusing on hackweek project 🚧.

What did you learn?

- I learned that the leadership team values experimentation and discovering solutions for the workplace; even by looking for solutions in our own environment 🕵️‍. I learned that talented people will do the right things to reach their goals, they will collaborate and have fun while taking on an ambitious goal. I learned how to use Glimmer.js for a small mobile web application and also tried out Tailwind CSS, a utility first library.

- How to build an iOS app.

What did you take away from the experience?

- I would like to ship 🛳 the project we worked on, I know it was only a prototype but I think we did show a viable solution. I’ll lean on our product team to know if that solution is also desirable for our customer base or even new customers. It was a pleasure to see the innovation that resulted from dedicating a week toward hacking on ideas brought up throughout the engineering, product and design teams. I would enjoy doing this again. I think I also have less reliance on the ceremonies we keep for producing software and instead a stronger reliance on the people to do so. We pulled off a nice prototype/demo without any traditional agile ceremony, just ad-hoc meeting and syncing up as questions arose.

- Learning new things and being able to work with different people 👾 👩‍🚀.

In Closing…

Hopefully sharing our hackweek experience can provide some insight into the kind of people we employ at Envoy, as well as the values we hold. It’s a pretty accurate snapshot 🖼 of our internal culture that we hope inspires other teams to experiment, to create margin for innovation, and to attract like-minded product owners, designers and software engineers to work with too.

Hackweek 2018 Recap was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This time we needed to build a Chrome extension, so we asked that question again; the answer ended up being… “maybe”. There are a ton of great starter and boilerplate projects on Github for building extensions, but we found out that with ember-cli and the right mix of add-ons, it would be more than enough to have a solid environment to work with.

So here I’m going to share the path we took to build a Chrome extension using Ember.js and the things we would have wanted to know a couple weeks ago.

Final app

Some context of what wanted to build: we needed a single Chrome extension with multiple components sharing a single Ember code-base. These extension components were:

Browser/Page Action popup: A small window that’s opened with a click on the corresponding UI element.

Content Scripts: A script that runs alongside pages; for compatibility reasons, it runs in an isolated world, but it shares the DOM with the page, and as such can modify it.

One of the coolest parts of the Content Scripts isolated world is that you can inject any library or code into a website and not worry about any conflicts with existing JS running there. You can have different versions of frameworks or trigger different events, plus you can share the DOM 🔝.

Initial setup and configuration

ember-cli-deploy-chrome-app is the initial boilerplate for an Ember Chrome extension and the easiest way to integrate CI and all the deployment processes to the Chrome Web Store.

It generates a chrome folder and a key.pem file in the root of your project. In chrome you'll find background.js and a default manifest.json that you can change according to your needs.

This chrome folder will have mostly configuration files and references to our Ember app. It also includes some handy symlinks pointing to the dist folder to access those files.

⚠️ Because we will need a Content Script extension component (which injects scripts into a page), symlinks do not work in this scenario. So every time the Ember app is built, we need to copy files from dist into chrome.

Obviously there is an add-on for that: ember-cli-post-build-copy

You may have noticed we’re only copying two files here, app.js and app.css. But a normal Ember app has also a vendor.js and avendor.css. We used ember-cli-concat to have a single file of JS and CSS so we can manage them more easily.

We won’t have an index.html here either. Therefore, we have to turn off storeConfigInMeta, so that all the config data is included in the app.js.

Finally our ember-cli-build.js will look something like this:

Booting up the Ember app

In order to share a single code-base with these extension components, we needed to manually control the booting of the application. Specifically for the Content Script part, where we want to wait until the user performs an action to render the app.

We can easily configure it by setting autoboot: false and calling

require('chrome-extension/app')['default'].create({ rootElement: '#selector' });

later when needed.

⚠️ We need to make sure we’re loading the JS files in the right order, so we can have access to require('chrome-extension/app')['default']

// chrome/manifest.json

"content_scripts": [
  {
    "matches": ["https://website/where-i-will-run-this-code/*"],
    "css": ["assets/app.css"],
    "js": [
      "assets/app.js",
      "content-script.js"
    ],
    "run_at": "document_idle"
  }
],

If you want to learn more deeply about how an Ember app boots, I highly recommend:

How does Ember Boot?

Because we set autoboot: false, we also need to manually boot up the Ember app on the pop up side.

// chrome/manifest.json

"browser_action": {
  "default_icon": {
    "16": "images/logo-icon-16.png"
  },
  "default_title": "Ember",
  "default_popup": "popup.html"
},

Lastly, we need to know where we are booting the Ember app from, so we can know which logic to use or which components to render. For this, we add a <meta> tag to the popup.html file, so can get its content from the application controller.

// popup.html

<meta scope=”ember-app-from” content=”popup”>

Working with the Chrome API inside an Ember app

Access to the Chrome extension API is available inside the Ember app from any file at any time. You can either use /* global chrome */ on files where you’re using it or just add it to your .eslintrc.js file so ESLint doesn’t yell at you.

// .eslintrc.js

module.exports = {
  globals: {
    $: true,
    chrome: true,
    window: true
  }
}

One of the APIs we needed was chrome.storage. We use it to save and share the “session” between the popup component and the content script. It provides the same storage capabilities as the normal localStorage API that you would have access to on any web app.

When using storage, the extension’s content scripts can directly access user data without the need for a background page.

To use almost all of these APIs you need request permission when the user is installing the extension. Just specify these permissions in your manifest file.

// chrome/manifest.json

"permissions": ["tabs", "activeTab", "storage", "management"]

One of the must-install add-ons in any Ember app is Ember-Concurrency (EC), as it makes it so easy to write concise, robust, and beautiful asynchronous code. We certainly wanted to use it here. But Chrome extension API uses a callback design structure API. In order to convert all these “callback hell” functions into something we can plug-in inside any EC task we needed: chrome-promise. With it, we canyield any call to the Chrome extension API.

Installing the Chrome extension locally

• Open the Extension Management page by navigating to chrome://extensions.
• Enable Developer Mode by clicking the toggle switch next to Developer mode.
• Click the LOAD UNPACKED button and select the extension directory (/chrome).

Live reloading

⚠️ Because we’re in an encapsulated environment, the WebSocket connection created by ember-cli-inject-live-reload won’t work here. Instead we need to watch for file changes under thechrome folder. For this we can use https://github.com/xpl/crx-hotreload. When a change is detected, it reloads the extension and refreshes the active tab (to re-trigger the updated scripts).

🚨 Known limitations

• Chrome apps currently seem unable to handle HTML5 pushState. Make sure locationType is set to hashin your config/environment.js file.
• Depending on your configuration, you might need to disable fingerprinting. If you have symlinked chrome/window.htmlto dist/index.html, the asset urls in app/index.html will be compiled like normal. However, if you have a different window.html, it will not be compiled and thus will unable to handle fingerprinted assets. In that case you have to disable it in ember-cli-build.js by setting fingerprint to { enabled: false }.

Conclusions

We felt really confident working and publishing this extension, it has a good test coverage, it shares the same patterns we use on our main Ember app and we know that any engineer can jump on it and add new features or fix bugs. We’re basically speaking the same language as if we were working on a normal web app. At the end an Ember app is simply a .js file plus a .css, so with the correct configuration you can basically port it anywhere you want.

Finally, by sharing this, we hope to encourage more people to have Ember.js as an option when creating Chrome apps. The Chrome API and the Ember.js ecosystem is a powerful mix for building ambitious Chrome extensions.

(((Ian Irving)) 🍁 on Twitter

@esbanarango always forget that #EmberInspector is a (brilliant) Ember app and Chrome extension that I use every day. #EmberJs

A framework for ambitious Chrome Extensions was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: this isn’t about how you build a multi-location engineering org, meaning teams located in different geos. In my experience the tactics to make that work are different. This is about building a team composed of folks in different locations.

Here are the things we do that make it work for us and keep it going as we hire more folks. We hope these ideas can help you too.

It starts with your goals and mindsets

The biggest factor is how our team thinks of our distributed environment. Do we hire folks outside of San Francisco because they’re cheaper? Do we believe being in the same room is unilaterally better for communication? Are we doing this to compete for the best talent?

Goals and mindsets affect your approach, strategy, and day to day behaviors so it’s important to be clear, explicit, and on the same page about them.

For us, we do it simply because we’re out to hire the best, and don’t think the best only lives in San Francisco. The talent pool in the world is only getting larger with the evolution of the industry.

The mindsets that help us strategically and day to day:

1. Everyone is a 1st class citizen. We prefer the word “distributed” over “remote” as there’s no implied hierarchy.
2. Communication is critical. The biggest difference between working colocated with someone and not is the communication channels can be different. It never means that communication is somehow less important.
3. We hire and value folks that can succeed in a distributed environment. It’s not for everyone, and not everyone will be good at it.
4. A strong engineering culture > distributed culture. We believe that promoting shared values, mindsets, skills, and behaviors in our people promotes fairness, regardless of location (and ethnicity and gender and sexual orientation and age and …).

We hire and construct our team around these goals & mindsets

If the goals & mindsets are the “what”, the “how” is in part about who you hire and how you set your teams up to work. These are the guidelines we use for hiring and constructing teams:

1. We hire folks primarily in or near our timezone (our specific guideline is continental US timezones) so folks can more easily collaborate real time. Timezones cause more pain than location in that regard. If we go into other timezones it will be with the intent of constructing a self-sufficient team there but right now we don’t want or need to.
2. 50/50 split between HQ & distributed is our target, though that’s flexible. This is because much of our company is still HQ bound, so it’s important to have a significant enough presence in HQ.
3. Balance. It’s more ideal when there’s a good balance of senior folks working in and outside of HQ, so a location doesn’t have an implicit importance. Similarly, we try and avoid situations where there is a lone person working from home on a team. This should be a lagging indicator of our approach, but is nice when it’s achieved. 2 of our 4 managers and half our senior engineers work out of their home, for example.
4. Evidence of success working from home. This usually means we won’t hire junior/early-career folks that work out of their house, and we’ll be extra judicious with experienced folks working from home full time for the 1st time. This might change as we get bigger and more able to afford the time to work kinks out.
5. Empathy, “High I/O” (good communication and open mindedness), strong accountability … basically we hire to the desired behaviors in our job bands. This enables our culture everywhere.

Note: as mentioned earlier, these are tactics for constructing teams made up of folks distributed across locations. At a certain size, we’ll probably mix in having entire teams at locations or perhaps teams made up entirely of folks working from their homes. We’ll evolve our tactics then.

How we work

It’s not just how we hired and constructed our teams. We also put in systems to ensure it works day to day.

1. Overcommunicate everywhere. This works whether we’re talking to someone a few desks away or in another city. We’ve focused a bunch of energy on great Github etiquette (verbose, templated pull requests and reviews), task tracking (Jira), heavy use of Slack and video, recording meetings if folks can’t attend, ceremonies like weekly demos/Show & Tells and all-engineering gatherings, and many more things. Related, we invest in the highest quality tools for collaboration (Zoom is pretty nice). It pays off.
2. Nothing beats face to face for bonding. No matter what we do, you can’t replace the human connection which is best created face to face. To this end, we have a quarterly co-location, usually at SF HQ, and loose budgets for traveling in between. That way we spend at least a month a year face to face if not more.

Conclusion

We believe this is a template we can follow at scale. We feel as long as we have teams composed of folks distributed across locations, we will be able to perpetuate our success using this formula. We hope this helps you too!

Building our (remote, distributed) engineering culture was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This feature may look trivial, but it’s not! There was a bug here that took us several months to track down. Today we are going to share the story of hunting the bug like finding a needle in a haystack.

It just works! Until it doesn’t 😅

After we launched the Passport app, some users reported that it occasionally didn’t work. The most difficult bugs to find aren’t those ones that happen all the time, but the ones that happen sporadically.

Besides that, there were even more factors that made this bug harder to find:

• It involved real devices interacting with each other, and it couldn’t be reproduced on a simulator.
• It was hard to tell if the iPad or iPhone app was causing the problem.
• Because iBeacon is facilitated by iOS, we didn’t know whether it was a bug from iOS or our app.
• The Passport app itself is very complex.

Given the massive scope and factors, this wasn’t an easy bug to fix. Here’s how we hunted the bug down step by step.

Round 1. Simplify

At the very beginning, we had no clue whether it was Apple’s bug or ours. Either way we realized that finding out was our first priority. If it’s Apple’s bug, we can only report and wait for them to fix.

Passport is not a simple app, it talks to an API server, gets data from it, caches it, it updates the UI, and responds to user interaction. There are too many factors involved. And just like what Sherlock’s author said

Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth

— Arthur Conan Doyle

To make it easier for us to find out the truth, we need to eliminate as much of the impossible as possible up front. Say if it’s really a bug in iOS, then we should be able to reproduce the glitch with only simple BLE functionality, all other functionalities are just noises. So the strategy we adopted here is to simplify.

We built very simple apps, one for iPad and one for iPhone. They do nothing but BLE signal broadcasting and monitoring. The iPad app has an “I’m here!” button in it like this

We set it up at the front desk and asked our co-workers to help. They install another simple app on their iPhone, and every day they come to work, they tap the button and select their name to sign-in. Now we have simple apps which do only BLE broadcasting and listening. Next, to know the truth, we need to collect data.

Round 2. Logs, logs, and more logs

As an engineer, I loved to watch a TV show when I was a kid — Air Crash Investigation

It’s a TV show about the investigation of airplane crashes. There are many invaluable engineering lessons to learn from a tragedy. One very important lesson I’ve learned from this TV show was: write logs.

When an airplane crashes and the black box gets lost, it’s very hard to find out what happened. Software also runs and crashes — that’s why we should keep logging what’s happening.

To know what’s going on with the BLE stack, We started to log relevant data for our investigation — like

• Start broadcasting signals
• Entering an iBeacon region
• Exiting an iBeacon region
• … so and so on

We could then read and investigate that data to see why the BLE was not working as expected.

Round 3. Centralize the log

To make it easier to investigate on a larger scale, we developed a simple API server that writes BLE activities event log into a database.

Now, we are in the big data business! With all the events available in a database, we can query and find exactly the pattern we are looking for.

Round 4. Suspect found, but not the murderer

With the BLE data, we soon quickly identify and confirm a bug in iOS BLE stack, we published the report here. We found out that iBeacon region stops monitoring after the device reboots sometimes. Unfortunately, this doesn’t look like the major root cause of the unstable BLE issue, since people don’t usually reboot their iPhone. We reported this bug to Apple, they fixed it and we kept looking.

To know why sometimes it works, or doesn’t work, we need more details, like

• When did the user enter the building?
• When do they take out their iPhon?
• When do they tap the sign-in button?
• How do they carry their iPhone
• Do they put it in a metal box that blocks radiation signal or what?

To find out, we look into office security camera video footage.

After a long time of data collection and analytics, we came to a conclusion:

There are few iPhones seem to be malfunctioning, tend to not respond to iBeacon signals, but for almost all iPhone devices always work as expected. It may sound like we can easily jump into it’s the flaw of few devices as the final conclusion, but the problem is, some of our co-workers reported their passport is not working, and their device actually always works with the simple experimental app.

Hmmm, okay…. 🤔

Sorry, there is no conclusion yet. But at least for now, we can stop blaming Apple. We know there must be a bug in our code, or at least something different from the simple BLE app causes the malfunctioning.

Round 5. Sysdiagnose is here to help!

We contacted Apple for help. Apple asked us to install Bluetooth log profile and run sysdiagnose. We followed the instructions and found out that the sysdiagnose tool is super helpful for investigating the root cause of the bug, as it logs detailed events about Bluetooth activity in OS level.

To run sysdiagnose, visit this page, find the functionality you would like to debug, it’s Bluetooth for iOS for our case, we download the profile and install it on our iOS devices. Normally iOS won’t bother to log too many details, but this profile enables it. Next, use your iOS device as normal, and when encountering the problem, you long press

Volume Up + Volume Down + Power

and release. You will feel a short vibration, which means iOS started collecting sysdiagnose logs for you. Wait few minutes, then plug in your iOS device to your Mac, use iTunes to sync it, and you should be able to find the sysdiagnose log file at

~/Library/Logs/CrashReporter/MobileDevice/[Your_Device_Name]/DiagnosticLogs/sysdiagnose

Then you can open the “system_logs.logarchive” file with the Console app and you should be able to see the logging entries popping up

Once all logging entries are loaded, you can scroll to the time that specific issue occurred, and see if there is anything strange.

Caught in the act!

With the help of sysdiagnose, we finally found something promising. We realized that iOS was doing a great job in terms of seeing the iBeacon region.

After entering the iBeacon region, iOS should launch the Passport app and notify it about this event. However, instead of our app being notified, we found a dead body:

Yep, that’s our app and it crashed! After looking into the crashing report, we found that its error code is 0x8badf00d (bad food), we googled and found out it means the app takes too long time to launch and not responsive, thus end up getting killed by an iOS demand process call watchdog.

With that in mind, we look into how the app launches and realized that it actually takes a long time to load your previous sign-ins from RealmDB when you have many of them. That’s a pretty ironic finding, the more you use the app, the more likely it’s not going to work. Fortunately now we know and caught this serial killer in the act!

Solution

As the app got killed for taking a long time to launch for iBeacon event, we decided to make launch as lightweight as possible, we change the code a little bit by looking into “UIApplicationLaunchOptionsLocationKey” in “launchOptions”

- (BOOL)application:(UIApplication *)application 
   didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
    if (launchOptions[UIApplicationLaunchOptionsLocationKey]) {
        // Do not load UI and any other heavy duty loading stuff
        // Kick start iBeacon here
        return YES;
    }
    // ...
}

So if the app is launched for a location event, then it only runs necessary code for iBeacon setup. Since we’ve fixed the problem, Passport now always works like a charm!

Takeaways

That’s a long story to tell, but actually, there are still some details not covered. To keep it short, here are some lessons we’ve learned:

• If your app takes too long time to launch and not responding, it may get killed by watchdog, and of course, your app won’t see the iBeacon event
• If your app launches in short time but didn’t setup iBeacon in time, it’s also possible your app also won’t see the iBeacon event
• If your app happens to enter a region and another with the same UUID, it won’t be notified even the major or minor number is different, as iOS only cares about the region UUID when monitoring. And it only notifies your app when it enters or exits a region.
• App running with TestFlight may take longer time to launch, as it seems iOS needs to talk to server to ensure the testing is still valid
• Avoid heavy-duty loading when app is waking up for a location event
• Setup iBeacon as soon as possible when the app is launched for a location event
• Writing logs about software activity is a good practice, it helps a lot while debugging
• Simplify the code as much as possible to reproduce a very hard to reproduce bug
• Luke, use sysdiagnose!

Finally, hope you enjoy our bug hunting story. If you’re like us, feel excited about finding the root cause of bizarre bugs, strive to build great products and pursue it-just-works quality, Envoy is hiring!

iBeacon Bug: Hunting for a needle in a haystack was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Let’s look at the data.”

“It would be great to get some data around how users are using feature x.”

These are utterances we hear at Envoy quite often, and everywhere it’s becoming ever more common with companies collecting and consuming more and more data. Our goal, however, has not changed too much — we want to identify clear trends in the data that can help us make better product decisions about what to build, how to best build it, and for whom. As we grow with our product teams, becoming multi-product, exploring more complex use-cases, and modernizing ever more offices, data becomes increasingly important and increasingly challenging to scale. In this post, we share some best practices we’ve found and lessons we learned.

Analyzing usage is key

Unsurprisingly, staying abreast of how your users are using your product is critical to success. Keeping up with usage allows you to identify problems faster, recognize changing trends in a timely manner, and prioritize features. Data helps keep our product teams honest and makes sure the right things are being built and results are tracked and monitored.

Developments in technology has really helped close the gap between us and our users. At Envoy, we use tools such as Heap and Fullstory to be able to meet our customers right where they using our product, in their environment. This not only helps eliminate bias from our user sessions, but also saves us time. We’re able to replicate the results of traditional UX research sessions for a fraction of the hours and cost, and scale as our number of users grow. Casting a wider net also helps us identify new use cases and workflows previously unanticipated. We can then spend more of our time understanding the findings to build empathy for use cases and pain points to help prioritize our roadmap.

Having a hunch

We’ve found product usage analytics has helped us think through a hunch. We often have hunches about how users get maximum value out of our product. It could be something reactive such as how successful adoption of a new feature is. It more often is forward-looking such as whether a series of actions or behaviours lead to surfacing more value. In times like this, it is less intuitive to get existing data to make a decision. But we’ve found that clustering users to find those who exhibit the hypothesis, or in some cases a correlated proxy action, gives us enough insight to set a direction. Hypothesis testing like this allows us to iterate quickly and choose a fork in the path, even if the path remains unclear.

What data cannot do

On the other hand, we are carefully pessimistic about what data cannot do as well. Data cannot innovate your product for you. Being data-driven is great for corroborating intuition, iterating quickly, identifying problems promptly, and making difficult decisions with no clear direction. It can optimize your current product and take you to a ‘local maximum’, but it will always be blind to the ‘global maximum’. Ultimately, to create the better office experiences of the future, a deep empathy for our users, the problems they are trying to solve, and a brilliant team of innovators are required to be successful.

Follow us for monthly updates on how we keep using data to enhance our product!

How data helps us design our products was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last weekend at a Friendsgiving dinner I asked another guest, “What do you do?” In between bites of pumpkin pie, she replied in typical San Francisco fashion, “I work at a startup doing data ops, but it’s small so I really do a bit of everything. Very Series A.”

“I do a little bit of everything.”

This line seemed particularly relevant to me given my first six months at Envoy. In that short time span, I’ve had the opportunity to be part of both the Marketing and Data Science teams. This experience would be hard to find anywhere except at a small startup, and I’m very grateful for the perspective I’ve gained from working in two very different functions within the organization.

Why the switch?

As a recent college grad, and someone with a fresh perspective on marketing, it’s clear to me that the modern marketer must have an understanding of data and new technology to survive.

Hot Topics recently held a roundtable discussion about the future of marketing with six senior marketing executives:

When asked to describe what a B2B Marketing organization would look like in the future, the importance of data science dominated the discussion. Peter Thomas, Managing Director of Integrated Marketing at Accenture said of his marketing team, “It will have data scientists in it. We wouldn’t have had those two or three years ago, we wouldn’t have even known why we need them.”

Scott Allen, CMO of Microsoft UK describes the ideal member of his team as “part scientist or part creative, and those two need to come together. I’ve got a lot of data and insight professionals on my team, and I only see that being beefed up as it becomes the muscle of the CMO over the next couple of years. But there are other roles that have to be in there alongside that, which are the traditional creative roles.”

I’ve seen the new age of data-driven marketing embodied at Envoy. A prime example of the fusion of data and marketing is the tools we use to try and understand our customers. The parts of our tech stack that are especially pertinent to me as a marketer are:

• Heap Analytics to track every action people take on our website and inside our product;
• Clearbit as a data enrichment tool to provide information about company size, industry, geography and job titles;
• Intercom as a marketing automation and segmentation tool;
• Segment to bring data from all these sources together;
• Stitch Data for ETL to send data to Amazon Redshift, our data warehouse.

The reason for my jump from marketing to analytics come from a place of discomfort. I knew we had these amazing tools, and I somewhat knew what they did — but I didn’t know how to use them to do great marketing.

I would have questions, such as how can I find a lookalike audience for my best customer, and what are the marketing touch points that prompt them to start a trial? Or, are there cohorts of customers that are more likely to convert after they started a trial, and what actions did they take while in trial? My head was spinning with questions, and I knew the answers were there, but they seemed inaccessible to me. It was a very discomforting position to be in.

I wanted to be the new breed of marketer who combines a deep understanding of his customers and business model with an analytical skillset. I made my ambition known to my friends, mentors, and managers at the company. I actively sought out projects that involved analytics or working with software tools I wasn’t familiar. Through these projects, I realized I enjoyed analyzing data and uncovering insights more than other pieces of marketing.

Analytics was calling me, so I scheduled a meeting with my manager to drop the bomb that that I wanted to switch teams. I was nervous, in my head, I repeatedly went over the checklist of things I wanted to say. Justify the switch by linking marketing to analytics, check. Reference past projects and experiences that have prepared me for this roll, okay got it. Don’t forget what you’re supposed to say!

Lucky for me Envoy is still small, and I get to do a bit of everything. The conversation went well, and I was able to switch from Marketing to Data Science. I’ve been in my new role for a month and a half, and have learned several important things about the role of a data analyst:

We are partners with internal stakeholders

People from different parts of the organization from Revenue, Success, Product, Marketing up to the CEO all have questions, gut feelings, and instincts about their work. Our role as analysts involves providing them with the answers to these questions and giving them access to the knowledge they need to build a successful company.

Get over feeling that you’re “not technical”

I have a finance background, which means I have a lot to learn about data analytics. One management concept I came across in college that stuck with me is the importance of high self-efficacy. The basic idea is that if you believe you can succeed in a challenging new situation, then you are more likely to succeed.

Whenever I encounter a concept I don’t know, I ask questions, I do research and I always believe that I will be able to eventually understand it. Since joining the analytics team I’ve analyzed time series data from multiple data sets to help answer business-critical questions like how well our salesforce is performing, what is responsible for fluctuations in MRR and what are the funnel metrics for our new product. These are all questions I wouldn’t have been able to answer a month ago.

Ask questions

Your team is there to help you. We all have the same goal, to provide best in class analytics for Envoy. Helping each other makes our team stronger and makes it easier for us to achieve our goals. Don’t be afraid to ask questions, it can literally save you hours of work, what seems difficult for you may be easy for someone else and in most cases they will be happy to lend a hand.

Insights must bring about change

Insights form data are only valuable if they can bring about organizational changes that help us meet our business objectives. My journey as a marketer and a business leader is not over. I think it’s important to keep on learning, not just about data analytics, but about other departments as well. If there’s a problem that needs solving at Envoy I should be able to use all the tools in my arsenal, from marketing, analytics and elsewhere to solve it. After all, working at a small startup we are expected to be able to do a little bit of everything.

Switching from marketing to analytics was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

There’s been burgeoning adoption of Net Promoter Score (NPS) as a measure of customer satisfaction and a predictor of growth. NPS excels as an industry benchmark because it is, in majority cases, determined by asking a single survey question:

How likely is it that you would recommend our company/brand/product to a friend or colleague?

Here’s how it works: respondents to NPS surveys answer on scale of 0 to 10. Score a 9 or 10 , you’re a Promoter. 7 or 8, you’re a Passive. And anything lower, that makes you a Detractor.

Promoters get assigned a +1, Passives a 0 and Detractors a -1. This figure is averaged out and multiplied by 100 to give an NPS, which can fall anywhere between -100 (worst) and 100 (best).

Many of the world’s top companies regularly conduct NPS surveys and benchmark themselves against their industry average, which runs a wide gamut from 31 for telecommunications to 74 for healthcare. Tesla, for example, is doing great at 97, while Pacific Gas and Electric Corp not so much at -6.

The sampling problem

At Envoy, we run NPS surveys daily, both in-app and through email. In particular, we care about 30-day NPS — calculated by taking the mean NPS over the last 30 days.

Because we don’t want to spam our customers with surveys all the time, we throttle survey-sending to once every 90 days. This immediately presents a sampling problem: How can we determine the number of customers to survey each day, if we want this number x to draw a smooth curve over time?

For starters, we obviously can’t survey 100% of eligible customers at the start of every quarter, because then we’d have no respondants in the second and third months and our 30-day NPS would be irrelevant during those periods.

Intuitively, we ought to sample 1/90 of our customer base each day to send surveys out to. This is easier said than done. An approach we employed was to pull lists of eligible customers each day, pick 1.11% of them at random, and send them surveys.

This worked great on the first day.

On the second day, what needs to happen is: people surveyed on the first day have to be taken out of the pool. New eligible customers from the day before have to be added to the pool. When you include such complexities as only sending surveys on weekdays during work hours, and taking into account global timezones, this gets messy real fast.

On day 91, we have to do all that, plus add the people that were surveyed on day 1 back into the pool before sampling. This also means that 90 days is the least amount of time between surveys, when what we really want is for 90 days to be the exact amount of time between surveys.

# Sample data frame
set.seed = 66
df_sample <- df[sample(nrow(df), nrow(df)/90), ]

Looking back, this was a terrible idea.

Smarter sampling

After some iterating, it turns out that the better solution is simply to assign each customer a random number between 1 and 90. This way, day 1’s cohort would comprise all the 1’s, day 2’s all the 2’s, and day 91’s all the 1’s again. We can write it this way:

# Cohort eligible customers
set.seed = 66
df <- mutate(df, cohort = sample(1:90, nrow(df), replace = TRUE))

Then, pick an arbitrary start date and determine today’s cohort:

# Determine current cohort
freq <- 90
start <- as.Date("1970-01-01")
cohort <- as.integer(Sys.Date() - start) %% freq + 1

Finally, put it all together by identifying the correct cohort to survey:

# Sample data frame
df_sample <- filter(df, cohort == cohort)

Remember how we were talking about drawing a smooth curve of customers surveyed over time? If you model out what we’ve put together, you’ll find a curve with decreasingly small steps around day 91 and every 90 days thereafter. The slope of the curve also closely mimics that of your customer growth, which is exactly what we’re looking for.

Why sample manually?

Several NPS survey providers (Wootric and Delighted, to name a couple we’ve used) offer solutions to this very problem. They do a great job throttling surveys — all you really need to do is upload a CSV file of your customers, and they’ll take care of the rest.

So why would you roll your own sampling solution?

The simple answer is control. At Envoy, we always want to provide the best possibly customer experience, so our criteria for selecting customers to survey includes:

1. Have been on the platform for at least a month;
2. Have used the product at least once over the last 7 days;
3. Can only receive surveys between 8:00 AM and 5:00 PM on weekdays;
4. Belong to a particular set of users defined by permission level;
5. Work at companies that are active paying customers.

What we’ve discovered is that putting in more thought into your sample results in a more accurate and authentic NPS. This process, which we run on Heroku off an R buildpack, helps us to continue building great products around customer satisfaction and feedback.

Sampling for NPS surveys was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

We host our assets in S3. Before Ember CLI Deploy, the easiest way to upload assets would have probably been to use a custom grunt task, which getting to work could easily turn into several hours of hitting your head against a wall.

Today, the same process can be achieved by running ember install ember-cli-deploy-s3 and then configuring the deploy file with your AWS credentials.

We are using S3 to serve our index.html instead of Redis. To do so, we are using ember-cli-deploy-s3-index which also allows us to upload different revisions before making it available to our users.

Our config file looks like the following:

With both plugins configured, deploying our dashboard is a matter of running ember deploy staging or ember deploy production. Then we can activate a revision with ember deploy:activate production -r revision.

If we are not sure about which revision to activate, we can look at the list running ember deploy:list staging.

Automate the mundane

After getting Ember CLI Deploy to work, something didn’t feel quite right yet. We were still doing some manual tasks like running the deploy command to ship PRs to staging or even uploading and activating the latest commit in production.

One of Envoy’s core values is automate the mundane, and honoring that value we decided to go deeper and simplify further our deployment process.

Our requirements were:

1. Deploy automatically to staging every PR that passes all the tests.
2. Deploy every merge into master to staging and production.
3. Attach a notification to the PR in GitHub so we can test easily in staging.
4. Make master the default index in staging.
5. Notify us in slack that a new revision is ready to be activated in production.

Travis

To deploy automatically to staging we created an after_sucess hook in Travis that calls a custom script after the tests have passed. Our script is called travis_deploy.sh and it has the following content:

The previous script always deploys the commit to staging, and if we are on “master” then it also deploys to production. Great! One chore less to run, but we were still missing steps 3 to 5: the PR notifications and slack message.

GitHub

GitHub’s API allow us to create deployment events and attach them to different PRs. We use that to attach a notification when the deployment starts and then include a link to the revision in staging once it has been deploy.

We use the following script called “deploy-to-staging.rb” to generate the notification above:

And then, we edited our deployment script, so the script above is called only if the current branch is not master. Also, we automatically activate staging if the branch is master:

Once we solved requirements 3 and 4 the only thing we had left was to add slack notifications.

Slack

To get a notification in slack we added an incoming webhook and then put the following after activating staging:

Every time master is deployed to production (remember we don’t activate by default), we get a message like the following:

As easy like that, we got our fifth requirement.

Everything was happiness and all that jazz. But, we were still running a manual command to activate the latest revision in master. Using as example the revision above, one of our engineers would have to go to their terminal an run something like ember deploy:activate production -r 926e624, which turned into a boring task.

Dockbit

We started to use Dockbit to automate our API deployments through Slack, and we decided to do the same for our dashboard.

After adding the repository, we created a new pipeline with a single step doing the following:

Dockbit was the missing piece in our workflow, and it help us to automate our deployment process. Now, anyone can activate running a single command in Slack. We edited our “after_success” script to include the activation command, making it easier for us to activate:

Wrapping up

The term “lighting fast deployment” was probably brought to the Ember community after Luke Melia’s talk at RubyConf 2014: Lightning Fast Deployment of Your Rails-backed JavaScript app.

Two years after that, the same process can be replicated quite easily thanks to the tools mentioned above.

Ember CLI Deploy makes a breeze the assets handling part. Travis, GitHub and Slack take care of uploading and notifications, and Dockbit allows everyone on the team to finally ship code to production.

Do you have a different deployment workflow for your Ember apps? We would love to hear about it! At Envoy we love learning from others and we’re always looking for new ways to improve our processes.

Automated Ember deployments was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Question: have you ever wondered how those beautiful stamps are made?

The truth is, unfortunately, rather brutal: to make beautiful stamp images for the thousands of companies who use Envoy, we put our poor designers (Amy Devereux, Sze Wa Cheung, Wells Riley and Jon Rundle) on call 24 hours a day, 7 days a week. Their task? Manually create stamp images with Photoshop whenever a new customer signs up.

Just kidding! That’s not how things are done in Envoy (phew!). At Envoy, one of our core values is:

Automate the mundane

As such, we actually built software for making stamp images. Even though it might just seem like an image you show on your phone, we spent a lot of time and effort to make sure it looks perfect.

Why creating stamps is difficult

At first glance, turning a company logo into a stamp image is not that hard. With basic image processing knowledge, you simply convert the company logo as a monotone color, then draw it as a mask with the theme color on the canvas. Voila! Done.

This actually works fine for most company logos. However, the world is more colorful than you may expect. Most companies use only a single color in their company logo, but there are still many which do not.

Oops! As you can see, the problem is that there are components with different colors in the logo. When they are converted into monotone colors, there is no way the distinguish the boundaries originally defined by differing colors.

The solution: look for the edges

To address the problem, a straightforward approach is to stroke the edge between color components. But here’s another problem: how can we know where the edge is? For example, given a real-life photo like this:

…how can you find the edges? By definition, it’s the pixels where the difference between neighborhood pixels is huge.

There is actually a well-developed image processing technique called edge detection that can be used to solve this problem. We tried different edge detection algorithms, and realized that a Sobel filter provides the best result.

The fundamental idea of a Sobel filter for edge finding is this: there are two filter matrixes to be applied on the picture pixels:

These two matrices subtract nearby pixel values from horizontal and vertical directions to, essentially, produce the difference of nearby pixels. By combining the signal strength of nearby pixel difference

We get a edge stroked image:

With the edge image, we can finally make the edges of colorful company logos distinguishable in stamp images.

Let’s make it even better

The story doesn’t end here. Although the non-distinguishable edge in stamps is now distinguishable, we noticed there was a drawback to this approach.

Since the original distinguishable edge, like the outer edge between the visible and invisible area, is also stroked, it actually slightly shrinks the logo image.

Even though shrinking of the logo size on the edge is barely noticeable by the human eye, we wanted to make the stamp image as perfect as possible. To achieve that, we came up with an idea: the alpha channel also has an edge.

We can also get the edge for it.

As we only wanted to stroke the real non-distinguishable edges, we used the edge of the image of the RGB channel to subtract the the image of Alpha channel. Here are the real non-distinguishable edges:

Finally, we stroked the real non-distinguishable edges on the stamp and made the perfect stamp image.

The result looks great (if we do say so ourselves.)

Envoy is hiring

Admittedly, we’re a bit crazy. If you’re as crazy about pursuing quality and building the best product in the world, join us. As an exponentially growing startup, the work you craft will directly impact millions of people around the world. Here are the open positions:

iOS Product Engineer

Ruby Backend Engineer

Shoot us your resume if you’re interested!

The secret sauce of Envoy Passport stamp images was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Look familiar? That’s Envoy, our visitor registration app, which has signed in more than 5 million visitors in three years at companies all around the world.

Recently, we sometimes found ourselves trapped in a loop of fixing bugs instead of creating new features for the iPad app. The fundamental problem here is that we’ve incurred too much technical debt.

Software development is hard. To deliver the highest quality software in the industry and continually evolve, we need to pay back the technical debt we incurred by adopting automatic tests.

Mocking is hard

To write unit tests for the legacy code base, we need to refactor the design objects and architecture to make them testable first. However, without existing tests to ensure all functionalities are still working as expected, it’s difficult to refactor. As it turns out, it’s a chicken or egg problem.

Fortunately, in addition to writing unit tests against classes, we can write tests for UI interaction against the whole app. As long as we have good coverage from the UI surface down to the app, we can then refactor the legacy code inside the app later without worrying too much about breaking it.

Although UI testing solves the chicken or egg problem of testing and refactoring, it’s not an easy thing to do. The major issue we’ve encountered is that due to the way UI testing works, there are two standalone processes: 1) the UI testing runner and 2) the target app. The UI testing runner launches the target app and uses the accessibility infrastructure to inspect and control the app.

To test our app, we need to control the data returned from the API client object, but as they are two standalone processes, you cannot control in-memory values directly from one process to another. You’re unable to do this:

Then replace the API and set the mock data to return:

Iteration #1: Pass environment variables

Despite that you cannot mock the API directly (like what you do for unit tests), you can pass environment variables and arguments to the target app when you launch it like this:

With that in mind, our first approach was to write a mock API client that mocks API calls according to environment variables.

Passing environment variables and mocking the API function is easy and works, but there are many drawbacks to this approach:

• Need to write code for reading environment variables and mocking the API
• Hard to control the API, like delaying the response or no response to test timeout handing
• API behavior is not customizable. For example, if we want to return different values according to the received parameters
• Hard to check API calls and the data passed in

Iteration #2: Embassy and Ambassador

After writing a few test cases with the environment variables approach, we realized that it’s obviously not the best way to write tests. In the end, what we mocked is an APIClient that connects to the HTTP server, but then we thought: why not just mock the HTTP API server instead?

We looked around to find open source HTTP servers written in Swift, but since Swift is a relatively new programming language and Apple just open sourced it a while ago, the resources for server-side Swift were very limited. We did find swifter, however, it’s a sync style server and we want async. We also found Perfect and Kitura, but they‘re both geared towards being the production ready web application solution in Swift, which is too much heavy weaponry for us.

You may ask why not use existing resource from other language community, certainly we can, but as if it’s not in Swift, it brings extra burden for our iOS engineers to context switch between different languages. And there are things you cannot do or hard to do, like in-line assertion in the API response handler just right inside the test case.

Since we couldn’t find anything that met our needs, we finally decided to build the wheel ourselves. It’s called Embassy, a super lightweight async HTTP server built purely in Swift — and of course, it’s open source.

Some awesome features:

• Super lightweight, only 1.5 K lines
• Zero third-party dependency
• Async event loop-based HTTP server, makes long-polling, delay and bandwidth throttling all possible
• HTTP Application based on SWSGI, super flexible
• IPV6 ready, also supports IPV4 (dual stack)

For this HTTP server, we defined a gateway interface called SWSGI, a hat tip to Python’s WSGI (Web Server Gateway Interface). It’s basically a simple function defined as:

It decouples web applications from the implementation details of the web server and also allows middleware to be wrapped around other web applications. It’s very easy to set up an HTTP server and run it inside your UI tests. Dealing with SWSGI is also easy, but to make mocking APIs even easier, we built Ambassador, a lightweight web framework based on SWSGI.

Automatic UI testing in action

With Embassy and Ambassador, mocking APIs and writing UI tests couldn’t be easier. You can install them with CocoaPods: add Embassy and Ambassador to your Podfile target to the UI test like this:

and run “pod install.”

Here’s an example how you run a simple HTTP API server:

Next, configure your app to connect to “http://localhost:8080/api/v2/users” instead of the real API server. It should be able to get the provisioned JSON payload, which looks like this:

To avoid writing the handlers for the same endpoints again and again, we provide a DefaultRouter that has decent default handler build-in, like so:

Then, we have a base class UITestBase for all UI tests cases to inherit:

As you can see, we pass ENVOY_BASEURL here as the API base URL for your own API client. You also need to make it read from the environment variable.

Finally, you can now write test cases with mocked APIs:

You simply overwrite the endpoint, you can assert and whatever you want in the response handler.

More than just mocking APIs

One good thing about bringing embedded web servers inside UI tests is that you can also mock other services, like PubNub, we use it to allow server to notify the iPad app for certain events. We use a classic HTTP long polling technique introduced in the Ajax/Comet era to allow the test runner to push messages to the target app in real-time. This way, the test cases can push a message to the app whenever they want.

As the iPad app supports badge printing, we also created a protocol for badge printer and created a mock client connecting to an HTTP API endpoint for printer instead of real printer. This means we can examine the printed document from the iPad.

Wanna build cool stuff with us? Envoy is hiring

As a language, Swift was introduced mainly for building iOS apps and desktop apps. Currently, there are limited resources available, but we see huge potential in bringing server side technology into the Swift community.

The SWSGI gateway interface we defined for Embassy demonstrates how the server side ecosystem can be built on top of it. What’s great is that this was originally a 20% time project and meant for UI testing purposes only.

These are just a few of the cool things we’ve built at Envoy. If you also enjoy building cool stuff and solving hard problems, Envoy is hiring. Here are the open engineering positions:

iOS Product Engineer

Ruby Backend Engineer

Feel free to shoot us your resume if you’re interested. Thanks! :)

Embedded web server for iOS UI testing was originally published in Envoy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.