HackedThat: Minding the backdoor

Daniel DeGroff

Hack This

Earlier this summer, we published a comprehensive Guide to User Data Security detailing steps to harden a server and secure applications. We provisioned a couple Linode servers and hardened them to the guides specifications to stand by our claim. We shared the IP addresses and proposed a challenge. 

Github: https://github.com/inversoft/2016-security-scripts

Hack This: https://hackthis.inversoft.com

We dared anyone to hack our database. To add incentive, we offered a fully loaded MacBook Pro as a reward. 

Challenge accepted.

Polynome successfully breached our security and have taken the time to publish a thorough post detailing how they did it. They walk you through their thought process, outline false attempts and share the hack that ultimately proved successful. Click on the following link to read their account of the hack.

Breaking into a Hardened Server Via the Back Door

Our account of the events leading up to the hack.

Before promoting the contest we reviewed and verified all necessary security precautions were in place on the “Hackthis” servers. We discussed all possible attack vectors and felt confident that the measures we had employed would keep all attempts to breach our systems at bay.

We moved forward, launched the Security Guide and promoted the contest. Weeks of unsuccessful attempts bolstered our confidence. As Polynome said, “Their hardening guide was and remains correct; there was no way we were getting through the front door of their servers…”

The chink in the armor.

Prior to the successful hack we had been preparing the release of some new cloud offerings. In support of this effort our production server required an API key to request new nodes on our cloud hosting provider, Linode.

Admittedly our production servers did not have the same rigorous security measures in place as the Hackthis servers that we used to taunt “would be” hackers.

The team at Polynome was able to identify our weakest link and exploit it, essentially bypassing all intended security measures on the Hackthis servers. We had assembled a small army to guard the front door and left the back door unprotected. Once Polynome obtained shell access to our production server through a known ElasticSearch vulnerability and found the Linode API key, it was only a matter of time.

Because the penetration came from our own production server the exploration by the Polynome team went mostly undetected. The final phase of the attack required them to clone the cloud instance and mount it on a newly created node and swap private IP addresses to gain database access through the firewall. While we quickly detected these events it was not immediately clear who or what was triggering them.

If you’re familiar with Linode API keys you know that they act as a proxy for a user. The API key privileges are synonymous with the owning user of that key. The actions taken by the Polynome team with our API key appeared to be coming from our own server by one of our developers.

The Polynome team knew they had to move quickly once they began the sequence of destructive steps. To claim success they needed to dump the Passport database undetected.

They did in fact move quickly. While we were feverishly working to identify who or what was causing these events we received a note from the Polynome team. It was over, they had succeeded.

Lessons learned.

We learned a few things from this exercise.

First. We had not applied the same set of rules we touted in our guide to our own production server. Duh.

Second. Our production server was not a good place for our product demo instances of CleanSpeak and Passport. ElasticSearch was running as part of the services required to run our product demos and was not properly firewalled. “It’s a demo, who cares if it gets hacked, there is no customer data on that search index.” Famous last words.

Third. The principle of limiting privilege was not applied to the API key we kept to create cloud nodes for our cloud offerings.

The irony of course is that we already knew these things. I would wager that in most successful security breaches, the necessary measures were already thought of, discussed and then not implemented for a variety of reasons.

The same reasons companies don’t test their backup and restore procedures until they lose critical data is the same reason many wait until they have been hacked to shore up their security.   

Job well done.

We want to thank Polynome for the time and effort they put into the contest and the ethical way in which they handled themselves. Congratulations on a job well done.  We hope you enjoy your MacBook Pro!

About Polynome
Polynome is a consulting firm providing rapid, high-quality design and implementation of software projects with a focus on scalability and security. Our expertise ranges from mobile and web applications to distributed server systems. Our team has decades of experience building secure applications for Microsoft, Amazon and numerous successful startups.

Join the dicussion on HackerNews: https://news.ycombinator.com/item?id=12361318


2 responses to “HackedThat: Minding the backdoor”

  1. sims11tz says:

    Good on you for how you handled this and how transparent you have been in your response to the public. I have always respected your company and have enjoyed your filter. Its always nice to have someone else learn the hard way and then plead with the rest of us to not repeat the mistakes!


  2. Matson, Gregory P. says: