This morning, as soon as I stepped into the office, my team member informed me that one of our important websites had been hacked, and malicious JavaScript code had been injected into ALL .js files. It was so bad that Google flagged the site:
I immediately entered firefighting mode.
After some investigation, it seemed the hacker might not have gained shell access to the server but simply bypassed the upload mechanism of WordPress through some outdated plugins (we still don’t know which ones) and uploaded what we call a “ground zero” script file. The script would find all .js files in wp-content directory (which is semi-public anyway) and replace all contents with obfuscated malicious code. The script would populate the malicious code again as soon as the files’ contents were cleared by Imunify (an antimalware app we have on our server).
It was a relief that this was contained within one vhost instead of server-wide.
Based on that hypothesis, I thought that simply rolling back our data to a pre-breached point in time would fix this, and then we could patch things up to make sure they couldn’t do it again.
We rolled the server back a month.
The malicious code was still there. Damn! This had been going on longer than we anticipated.
I suggested we rolling it back another month. That is May.
Luckily, this time it worked, because that was the earliest backup we could find. If that hadn’t worked, well, things would have started to get much hairier. We might have lost more data in the past months than we expected.
This stressed the importance of having a sound backup strategy for cases like this.
While my team member was managing things on Plesk control panel, I decided to shell into the server itself, bypassing the management app to investigate at the OS level. Something about the web GUI always gives me a sense of detachment from things that matter.
As soon as the website was recovered to the May checkpoint, we started clearing the .js files that were still there, but now empty because the “ground zero” script had been removed. My team member was in dismay because he couldn’t possibly go through thousands of .js files to delete them all. I did it with a simple find Linux command I usually use at home.
find . -type f -name *.js -s -1M -exec rm {} \;
To be sure that the malicious script was gone, I also ran lsof on the vhost dir to monitor all activities on that website for a few hours.
watch lsof -D /var/www/vhost/<domain>/httpdocs/wp-content
Then we update all plugins, themes and PHP, re-enable Wordfence etc. Though it’s been long known for me that the management and security of the whole websites infrastructure is really bad (obviously, when you have 30+ websites to manage), I guess this is the final straw where I get out of my role as a Marketing Manager and overstep onto server management, to require better management and security practice from the “IT people” in my company.
So at this point, I would say that we have secured the website. Now we only need to wait for Google to review the site to lift the flag.
What I found most satisfying was that, although I was very anxious during the whole event, I knew exactly what we needed to do instead of waiting for our part-time server admin or our web dev in Bangladesh to wake up and come to our rescue. I took the matter into my own hands and fixed it, all directly thanks to my studies in the past year.
I ended the day by adding more hard drives to one of my team member’s PCs and mine as well. Walked out of the office at 6:30pm.
So yeah, today was how I imagine what I want to do for the rest of my life from now on.
Just need to wait for the right time to take the leap of faith.