…and how/why it works.
Thanks!
In: Technology
xz is a compression utility used in a number of open source projects. Someone joined this project and contributed a lot of good changes. The creator and maintainer of xz had some issues and could not come up with the time to do all the tasks in the project so they promoted the attacker as another maintainer to help out. After some time the attacker added some test files to the project which were seemingly malformed compressed data but in reality it was the compiled code of a malware written specifically for this. After even more time a release came out with a modified build script that would package the malware into the xz utility itself. These changes were not done to the source code repository which all the developers used but only to a packaged version of the source code made as an official release. This entire process took over a year of dedicated work by the attacker.
The malware was only active if it ran in an application called OpenSSH. This is a tool which allow administrators to connect to a server in a highly secure and encrypted way and run commands or forward network traffic. OpenSSH is watched very closely for just such malware. But in this case OpenSSH does not include xz so it is not checked. Some distributions like Debian, Ubuntu and RedHat have made changes to OpenSSH to include closer interaction with SystemD. SystemD is a service manager used on many Linux distributions to manage the various services which run on the server, including OpenSSH. The changes the distributions added some features when running OpenSSH through SystemD. It involved including a SystemD library into OpenSSH which meant that the SystemD code is running inside the OpenSSH process. The SystemD library depended on xz so this code was now running in the OpenSSH library and the malware got triggered.
What the malware did was to go through the OpenSSH process looking for tables of where each function is located. A particular entry in this table, concerning how to validate an incoming authentication reply, was then changed to point to some code in the malware. I have not seen this exact code but I presume it checks if the person trying to connect to the server is the attacker and then lets them inn. If not they run the original code instead.
Because of all the ways the malware was hiding itself. None of the official project code repositories include any of the code. It only happens when you use the official xz release and not the repository, and only if you apply the SystemD changes to OpenSSH, and only when you run OpenSSH under SystemD, and only when you connect to the SSH server with this special key. It was very hard to find and might have been able to escape detection entirely. If it was not for the fact that we tend to take things slowly and give plenty of time between accepting new code into a distribution and releasing that distribution the backdoor would have been used on millions of servers by now. As far as I know only a few desktop distributions were affected. But next year both Debian and RedHat, the two largest distributions, would have released their latest version with this backdoor in it.
The entire thing was discovered by an employee at Microsoft who were doing some testing on a database called PostgreSQL, probably related to it being offered on Microsoft’s cloud platform. He was running on a test release of Debian with all the new software set to be released next year. And somehow he noticed that OpenSSH took half a second longer to start then usual. This peaked his interest and he started digging into the software to find out why. This is when he discovered that it was because the malware was going through the OpenSSH binary looking for the table with the functions. This was done in an inefficient way which could take half a second to do. If it had been written in a different way which were much faster then he would not have noticed it. Maybe nobody would have noticed it.
Latest Answers