…and how/why it works.
Thanks!
In: Technology
Team A sells door’s lock
Team B sells components to Team A so that they can build their locks
Guy C get incorporated in Team B
Guy C prove he can do very good things, and get promoted
Guy C add some strange and hard thing to understand in the component’s building-testing stage, but people trust it, because not that much people work in Team B
Guy D early tester of the first door’s lock released by Team A incorporating the latest components of Team B, find out the door don’t work as good as the older versions, and notify some people around the world
People around the world investigate and notice that the new doors have now a “master key”. Presumably that Guy C has a key that can open all the doors.
If it was not cached early, he could have a key that opened around 60% of the doors in the world.
Linux is full of small pieces of software, developed by tiny teams (sometimes a single person), that do important things in the OS.
XZ is one of those pieces of software, it handled compression (think like zip on windows) and was developed by just a single person.
A couple of years ago people on forums (now thought to be linked to the hacker) started complaining at him for being too slow to update it. Then the main attacker, Jia Tan, came in and offered to help with the project. This isn’t unusual in open source.
So Jia Tan is doing a good job keeping xz updated for a few years and the main developer is so happy with their work they let them push new versions out.
However Jia Tan had very cleverly hidden a hack in the download file. It modifies the remote access software that Linux comes with to add their own secret user to it.
So if this hadn’t been found as quickly as it was the hacker could have logged in to any Linux system using the hacked xz version.
xz is a compression utility used in a number of open source projects. Someone joined this project and contributed a lot of good changes. The creator and maintainer of xz had some issues and could not come up with the time to do all the tasks in the project so they promoted the attacker as another maintainer to help out. After some time the attacker added some test files to the project which were seemingly malformed compressed data but in reality it was the compiled code of a malware written specifically for this. After even more time a release came out with a modified build script that would package the malware into the xz utility itself. These changes were not done to the source code repository which all the developers used but only to a packaged version of the source code made as an official release. This entire process took over a year of dedicated work by the attacker.
The malware was only active if it ran in an application called OpenSSH. This is a tool which allow administrators to connect to a server in a highly secure and encrypted way and run commands or forward network traffic. OpenSSH is watched very closely for just such malware. But in this case OpenSSH does not include xz so it is not checked. Some distributions like Debian, Ubuntu and RedHat have made changes to OpenSSH to include closer interaction with SystemD. SystemD is a service manager used on many Linux distributions to manage the various services which run on the server, including OpenSSH. The changes the distributions added some features when running OpenSSH through SystemD. It involved including a SystemD library into OpenSSH which meant that the SystemD code is running inside the OpenSSH process. The SystemD library depended on xz so this code was now running in the OpenSSH library and the malware got triggered.
What the malware did was to go through the OpenSSH process looking for tables of where each function is located. A particular entry in this table, concerning how to validate an incoming authentication reply, was then changed to point to some code in the malware. I have not seen this exact code but I presume it checks if the person trying to connect to the server is the attacker and then lets them inn. If not they run the original code instead.
Because of all the ways the malware was hiding itself. None of the official project code repositories include any of the code. It only happens when you use the official xz release and not the repository, and only if you apply the SystemD changes to OpenSSH, and only when you run OpenSSH under SystemD, and only when you connect to the SSH server with this special key. It was very hard to find and might have been able to escape detection entirely. If it was not for the fact that we tend to take things slowly and give plenty of time between accepting new code into a distribution and releasing that distribution the backdoor would have been used on millions of servers by now. As far as I know only a few desktop distributions were affected. But next year both Debian and RedHat, the two largest distributions, would have released their latest version with this backdoor in it.
The entire thing was discovered by an employee at Microsoft who were doing some testing on a database called PostgreSQL, probably related to it being offered on Microsoft’s cloud platform. He was running on a test release of Debian with all the new software set to be released next year. And somehow he noticed that OpenSSH took half a second longer to start then usual. This peaked his interest and he started digging into the software to find out why. This is when he discovered that it was because the malware was going through the OpenSSH binary looking for the table with the functions. This was done in an inefficient way which could take half a second to do. If it had been written in a different way which were much faster then he would not have noticed it. Maybe nobody would have noticed it.
Already discussed here recently: https://www.reddit.com/r/explainlikeimfive/comments/1brf749/eli5_the_recently_discovered_xz_backdoor/
Latest Answers