Initially I titled this «My laptop was hacked» but because the way they got in was by guessing a password, so it’s difficult to consider this a hack. Furthermore, what they did was pretty standard stuff, so although they technically broke into my laptop, I feel more like I invited them, they had a couple of beers and left. Either way, it was a back-to-basics experience for me and I want to share it with you.

When a process goes crazy to a point that I can no longer open a terminal for further analysis, I am really glad that I have permanent real-time resource monitors on my screen. This has been a valuable tool multiple times at least to understand what is happening just by peeking and take quicker and better action than if I hadn’t. This screenshot is from Xfce 4.14:

Today was one of those days. Not that the computer was slow, but worse: the CPU monitor was showing this:

Two out of four CPU virtual cores were being used to their maximum capacity but the other monitors were fine. I had not left anything running through the night so it was possible that my computer was hacked. But… how?! Follow the story step-by-step so you can follow my train of thought and learn from my mistakes.

I immediately ran top to find out more and I found process 311250, called kswapd0, using 200% CPU.

I made the mistake of not issuing the ‘c’ command (which toggles the command name or line) in top. I also did not notice the process owning user, considering that the laptop is only used by me. More on this, later.

I did a quick Web search just to find out more about kswapd0 but this was just a red herring. I confirmed that it is a legitimate Linux kernel process which can actually have high CPU load in particular circumstances and there was a lot of information about the real kswapd0. I spent maybe a minute or two on that, max. It didn’t make any sense to follow that track so I took it on my own.

So, I wanted to immediately kill the process, but I needed to get whatever data I could about it for later analysis before killing it. So, I did cd /proc/311250 and issued:

head -n -0 $(find -type f) > ~/311250.txt

It should export the contents (head -n -0) of all found files [$(find -type f)] and write that to ~/311250.txt. I opted for specifying the files by using $(find -type f) instead of * because head does not recurse directories like grep -r does. What I like about head -n -0 is that it dumps out the contents of the specified files, indicating the name in the output. Demo:

$ mkdir head-demo; cd head-demo
$ echo hello > a; echo world > b
$ ls
a   b
$ head -n -0 *
==> a <==
hello

==> b <==
world

Well, it hanged. Being the /proc file system, most likely it was not an actual hang. I might just be dumping a huge virtual file, something similar in nature to copying all of /proc and hitting /proc/kcore in the process. This would mean that it was just the output being a never-ending set of data being correctly written out to 311250.txt so I canceled it. Because usually most of /proc/<pid> contents are small, the quickest I thought to do was to do less ~/311250.txt and manually page down to a stream of continuous data and find out what that file name. Sure enough, it was /proc/311250/pagemap. I easily amended my command to remove the pagemap file from the list in the hope that no other huge file would get in the way:

head -n -0 $(find -type f | grep -v pagemap) > ~/311250.txt

It worked! Let’s see the first 10 lines of the resulting head -n -0 output:

# ls -lh
total 1.5M
-rw-r--r-- 1 root root 1.5M Jun 18 10:44 311250.txt

# head -n 10 311250.txt 
==> ./task/311250/fdinfo/0 <==
pos: 0
flags: 0100000
mnt_id: 23

==> ./task/311250/fdinfo/1 <==
pos: 0
flags: 0102001
mnt_id: 23

Browsing through it, I noticed that some sockets were open, referring to port 22 being open as a local port. This isn’t good but it’s not necessarily bad, as long as no one has guessed a password and the sshd process doesn’t have a vulnerability before bad logins are denied.

Before continuing with the rest of ~/311250.txt I decided just to take a quick look at /var/log/auth.log for accepted passwords. I found a large amount of failed password messages so this was likely a result of a brute-force attack. I cheated a little bit and I briefly logged in to my own laptop using ssh 127.0.0.1 to quickly find out how successful logins were recorded:

Jun 18 10:51:47 alvarezp-samsung sshd[420113]: Accepted password for alvarezp from 127.0.0.1 port 57306 ssh2
Jun 18 10:51:47 alvarezp-samsung sshd[420113]: pam_unix(sshd:session): session opened for user alvarezp by (uid=0)
Jun 18 10:51:48 alvarezp-samsung sshd[420113]: pam_unix(sshd:session): session closed for user alvarezp

I went with the first line and grepped for ‘Accepted password for’:

# grep -ir 'Accepted password for' /var/log/auth.log
Jun 18 09:18:35 alvarezp-samsung sshd[308256]: Accepted password for demo from 132.232.30.87 port 39128 ssh2
Jun 18 10:51:47 alvarezp-samsung sshd[420113]: Accepted password for alvarezp from 127.0.0.1 port 57306 ssh2

Yeap, somebody broke into my laptop!

For sure I created a demo account for a presentation and forgot to erase it afterwards. I couldn’t guess the password I had set for the demo account. TBH I just tried demo, it didn’t work and gave up. 😀 Either the attacker changed it after logging in, or, most likely, I set a password that I thought as being hard enough for a simple brute-force attack but it really wasn’t.

Now, you think changing the SSH port is good enough? Think again. My laptop is connected through a NAT router and I had an external port forwarded back into it, that port not being 22, and yet, someone found it. It’s more important to have great passwords. I have never seen port changing as being effective, to be honest.

Let’s see what’s there on the demo account:

$ find -name kswapd0
./.configrc/a/kswapd0

There we go! Let’s see what’s there under the hidden .configrc directory:

$ find .configrc -type f
.configrc/dir2.dir
.configrc/a/kswapd0
.configrc/a/dir.dir
.configrc/a/a
.configrc/a/bash.pid
.configrc/a/run
.configrc/a/stop
.configrc/a/init0
.configrc/a/.procs
.configrc/a/upd
.configrc/cron.d
.configrc/b/sync
.configrc/b/dir.dir
.configrc/b/a
.configrc/b/run
.configrc/b/stop

Remember head -n -0? I used it once again to glance through the source code. Interesting things:

1. It installed a crontab on the demo user.
2. It tries loading the msr kernel module, then it tries to modify the MSRs depending on the CPU. This failed because of not being root. This section supports Intel and Ryzen.
3. It wrote some files into /tmp. This is a typical technique.
4. It uses some pseudonyms like Donald, ld-linux, kswapd0, xm64, Macron, anacron…
5. kswap0d is included as a binary.
6. Inside there’s a base64 encoded pack-obfuscated Perl script. After decoding and depacking the script it turns out it has variable names in Spanish and is used to connect the computer to an IRC server with a Netherlands IP address, most likely a C&C for botnet purposes. The IP most likely corresponds to a hacked computer too. Nothing out of the ordinary.
7. It install an SSH key in .ssh/authorized_keys, not before deleting the whole ~/.ssh directory first and recreating it.
8. It includes a script to kill all miners known by name. Some names may pass for common system services.

It’s pretty standard stuff, nothing fancy. Boring sample of what I found:

You can clearly see that even as a non-root user it’s easy to ramify the implications beyond just entering and running a file… just so you know.

Usually, whenever I get a hacked system I just format it to get it back into a trusted state. About this specific attack it seems to be a pretty standard, botnet-related, cryptocurrency-purpose attack. A fresh install does not seems to be necessary. However, after any attack it’s impossible to assess with certainty if something invisible might still be in there or not, or if the laptop was hacked before, so I am seriously considering wiping it out and starting fresh anyway.

This is what I have done, for the time being:

• Killed all processes related to that user.
• I changed the password to some difficult string —hopefully—.
• Disabled the account by issuing usermod -L demo.
• Renamed the account name.
• Removed the crontab.
• Removed the SSH key.
• Added the account to /etc/cron.deny.

More on the brute-force attack

Browsing through /var/log/auth.log I found a lot of attempts from different IP addresses, so this seems to be a brute-force attack that started who-knows-when:

# grep -ir 'Failed password' /var/log/auth.log | head -n 1
Jun 14 00:31:24 alvarezp-samsung sshd[2284]: Failed password for invalid user ubuntu from 111.161.74.106 port 40074 ssh2

# head -n 1 /var/log/auth.log
Jun 14 00:00:36 alvarezp-samsung pkexec: pam_unix(polkit-1:session): session opened for user root by (uid=1000) 
# grep -ir 'Failed password' /var/log/auth.log | wc -l
31953

This should give us a clue. Observe the huge different in log file sizes (file dates roughly means the date of the last entry in the file).

# ls -l /var/log/auth.log*
-rw-r----- 1 root adm 20934857 Jun 18 12:45 /var/log/auth.log
-rw-r----- 1 root adm 28213399 Jun 13 23:59 /var/log/auth.log.1
-rw-r----- 1 root adm  1533685 Jun  8 00:00 /var/log/auth.log.2.gz
-rw-r----- 1 root adm    90530 May 31 00:04 /var/log/auth.log.3.gz
-rw-r----- 1 root adm    52949 May 24 00:00 /var/log/auth.log.4.gz

So, looking at auth.log.2.gz, we see the first line is from May 31st, and the first failed attempt was from June 3rd:

# zgrep 'Failed password' /var/log/auth.log.2 | head
Jun 3 13:01:58 alvarezp-samsung sshd[1030212]: Failed password for root from 123.207.85.150 port 48990 ssh2
Jun 3 13:12:50 alvarezp-samsung sshd[1031013]: Failed password for root from 123.207.85.150 port 60460 ssh2
Jun 3 13:13:12 alvarezp-samsung sshd[1031032]: Failed password for root from 145.239.239.83 port 33414 ssh2
Jun 3 13:14:57 alvarezp-samsung sshd[1031120]: Failed password for root from 123.207.85.150 port 59062 ssh2
Jun 3 13:16:57 alvarezp-samsung sshd[1031216]: Failed password for root from 123.207.85.150 port 57650 ssh2
Jun 3 13:17:06 alvarezp-samsung sshd[1031231]: Failed password for root from 145.239.239.83 port 43972 ssh2
Jun 3 13:17:48 alvarezp-samsung sshd[1031261]: Failed password for root from 145.239.239.83 port 49298 ssh2
Jun 3 13:18:30 alvarezp-samsung sshd[1031296]: Failed password for root from 145.239.239.83 port 54624 ssh2
Jun 3 13:18:47 alvarezp-samsung sshd[1031311]: Failed password for root from 123.207.85.150 port 56238 ssh2
Jun 3 13:19:16 alvarezp-samsung sshd[1031332]: Failed password for root from 145.239.239.83 port 59946 ssh2

# zcat /var/log/auth.log.2 | head -n 1
May 31 00:04:28 alvarezp-samsung sudo: pam_unix(sudo:auth): Couldn't open /etc/securetty: No such file or directory

The last command is just to prove that the log file started way before the attack so we don’t misunderstand the start of the file for the start of the attack.

There was a total of 110,939 attempts from 2,337 IP addresses during the course of 14 days.

The ‘c’ command in top

Had I toggled from the command name to the command line in top (by pressing the ‘c’ key while in top), instead of showing kswapd0 it would have toggled to ./kswapd0. This would have quickly revealed that the kswapd0 process was not a kernel process, but a user-space process ran directly from somewhere in the file system. If it had been a kernel process, its command line would have been changed into [kswapd0] instead.

Most of you already know this, but for those who don’t: in Linux, the current directory is not considered for finding executable commands. Only those directories in the PATH variable are looked into. To include the current directory, it must be explicitly added to the PATH variable. This is not recommended, though. So, to run a program from the current directory, provided it’s not listed in PATH, it must be prefixed by dot-slash, ./like-this, so instead of asking the shell to find the program, we ask it to run a directly-specified file.

I could have looked even more patiently into the top output: it was right there: the program belonged to the demo user!

Now, on to wiping it, just as a precaution. 🙁