Learnings from the GitHub OAuth user token attack

Hi everyone,

if you work on the internet last weekend may have been jarred by the announcement by GitHub of an attack in which an abuser has possibly used stolen credentials (OAuth tokens) from the services Heroku and TravisCI to download data from private repositories.

We won’t go into the details of the attack here, but suffice to say this announcement was quite concerning for us: here at OpenCage we are users of all three services. GitHub hosts our source code, we use Travis to run automated tests, and our website is hosted on Heroku.

The good news is that all services immediately took action to address the situation, and there is no evidence that our source code was compomised. TravisCI has since said, “We thoroughly investigated this issue and found no evidence of intrusion into a private customer repository (i.e. source code) as the OAuth key stolen in the Heroku attack does not provide that type of access.”

That is reassuring. And as we noted on our Twitter account immediately after becoming aware of the situation, no user data or production credentials are ever stored in source code. So there was no reason for panic, but still we thought to be prudent we have to now assume an attacker has a full copy of the contents of our private repositories, so we immediately launched into a detailed audit of what that might imply.

We did a full review, for full transparency, and in the hopes that it might help other online businesses, here are our findings and resulting actions:

For sending emails we use the service Postmark. We used the same API token in production and test. We deactivated that token and now we use new, different tokens in testing and production. The production token is no longer stored in source code. Postmark has a full audit trail of every email sent the last 30 days. We reviewed and found no evidence of any abuse.
We kept the offsite backup encryption passphrase in a script. Sounds bad, but here was no ability to read or write backups as that is only possible via SSH. We have changed the passphrase and it is now no longer stored in the script.
Postgresql database replication for one of the internal services we use was documented including password for ssh access between servers and password of the replication database user. All have been changed, database user accounts are now limited by IP address and replication protocol.
We were using the same Slack webhook tokens for Rails testing and production to send team members various alerts. In theory this could have been used to send notifications to our company Slack. No such messages have been sent though. All tokens have been regenerated and now different tokens are used for testing and production.

Overall nothing particularly major, but that is also because over the past years we have devoted significant time to regularly reviewing security best practices. This incident serves as a good real world example of why it pays to be vigilant, and the findings were reassuring. It was a good reminder that work on security is worth it, and also neever “finished” (though, all things equal, it would be great if next time that reminder doesn’t come on a holiday weekend).

Finally, if you are a user of OpenCage, and worried your geocoding API key may have been compromised (in this attack or otherwise) you can always create a new key (or keys if you are a subscription customer) in a few clicks in your account dashboard.