Physical Address
Metro Manila, Philippines
Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server
Killing a malicious process does not necessarily remove malware from a Linux server.
We learned this during an investigation of a compromised Zimbra mail server running on Ubuntu.
We discovered suspicious processes named javab and idle. We terminated them, but the activity returned.
That changed the investigation.
Instead of asking only:
What are these processes?
we needed to ask:
What keeps starting them?
The answer was hidden in the zimbra user’s crontab:
* * * * * /home/SSL/.khp
A suspicious file named .khp was being executed every minute.
This article explains how we found the cron-based persistence mechanism, how we correlated it with /var/log/syslog, why killing the processes did not solve the compromise, and what Linux administrators should check when suspicious processes keep returning.
Customer names, domains, IP addresses, remote destinations, SSH keys, credentials, and other identifying information have been removed or replaced with placeholders.
Important: This article documents a real incident, but it is not a universal malware-removal procedure. Preserve evidence before deleting suspicious files or modifying a compromised system.
The Problem: Suspicious Processes Kept Returning
During our Zimbra investigation, we discovered processes that did not belong to the expected Zimbra environment.
Two names repeatedly appeared:
javab
idle
One suspicious process had a command resembling:
./javab -o <REMOTE_HOST>:<REMOTE_PORT>
The destination has been removed from this article.
We investigated the processes using commands such as:
ps aux | grep -E 'javab|idle'
and:
pgrep -a javab
pgrep -a idle
The -a option tells pgrep to display the process command line together with the PID.
Once we determined that these processes were unauthorized, we terminated them.
For example:
pkill javab
pkill idle
or by PID:
kill <PID>
At first, this appeared to stop the activity.
It did not solve the actual problem.
The processes returned.
Killing Malware Is Not the Same as Removing Persistence
A running process is only one part of an intrusion.
Something started that process.
If the process starts again after being terminated, another component may be responsible for launching it.
Possible persistence mechanisms on a Linux server include:
- User crontabs
/etc/crontab- Files under
/etc/cron.d/ - Scripts under
/etc/cron.hourly/ - Scripts under
/etc/cron.daily/ - Scripts under
/etc/cron.weekly/ - Scripts under
/etc/cron.monthly/ - systemd services
- systemd timers
- SSH
authorized_keys - Shell startup files
- Application startup scripts
- Legacy init scripts
In our incident, cron turned out to be one of the confirmed persistence mechanisms.
The Critical Discovery
Because the suspicious processes were running under the zimbra account, we inspected scheduled tasks associated with that account.
A user crontab can be displayed with:
crontab -u zimbra -l
The -u zimbra option tells crontab to operate on the zimbra user’s crontab, while -l lists its current contents.
Inside the crontab, we discovered this unauthorized entry:
* * * * * /home/SSL/.khp
This line was particularly important because /home/SSL/.khp had already appeared during our investigation of suspicious files.
We had now connected a suspicious file to a scheduled execution mechanism.
Understanding the Cron Entry
A standard user cron entry uses five scheduling fields followed by the command:
minute hour day-of-month month day-of-week command
Our suspicious entry was:
* * * * * /home/SSL/.khp
Breaking it down:
* * * * *
| | | | |
| | | | +-- Day of week
| | | +------- Month
| | +------------ Day of month
| +----------------- Hour
+---------------------- Minute
An asterisk means every valid value for that field.
Therefore:
* * * * * /home/SSL/.khp
means that /home/SSL/.khp is eligible to execute every minute.
The Linux crontab(5) documentation states that cron examines entries every minute and executes jobs whose time fields match the current time.
This immediately explained why killing javab and idle was insufficient.
We were eliminating the visible process while leaving the mechanism responsible for recreating the activity.
What the Persistence Chain Looked Like
At a high level, our investigation established this relationship:
zimbra user crontab
|
v
* * * * * /home/SSL/.khp
|
v
/home/SSL/.khp executes
|
v
suspicious activity returns
This was far more important than simply identifying a strange process name.
Once the scheduled execution was discovered, we could explain why the server appeared to become reinfected after the suspicious processes were killed.
Why the Cron Entry Was Not a Normal Zimbra Job
Zimbra legitimately uses scheduled cron tasks.
This means you should not delete the entire zimbra crontab simply because the server has been compromised.
Zimbra has legitimate scheduled maintenance tasks.
The suspicious part in our case was this specific command:
* * * * * /home/SSL/.khp
It pointed to a suspicious hidden file under a directory already associated with other unauthorized artifacts.
Zimbra’s own security investigation documentation specifically recommends examining the zimbra user’s crontab when investigating a compromised installation.
The Zimbra security guide even provides an example of an unauthorized one-minute cron job that downloads and executes a malicious script.
This made checking the Zimbra user’s scheduled tasks especially relevant to our investigation.
Correlating the Cron Entry With /var/log/syslog
Finding a suspicious crontab entry tells you what is configured to execute.
It is even better if you can find evidence that the command actually executed.
On our Ubuntu server, cron activity appeared in:
/var/log/syslog
We searched for cron commands running under the zimbra account:
grep -hE 'CRON.*zimbra' \
/var/log/syslog \
/var/log/syslog.1 \
2>/dev/null
This command does several things.
grep -hE
uses extended regular expressions and suppresses filenames from the output.
The pattern:
CRON.*zimbra
looks for records containing CRON followed later by zimbra.
We searched both:
/var/log/syslog
/var/log/syslog.1
because rotated logs may contain activity from before the current log file.
The following:
2>/dev/null
suppresses error messages if one of the files does not exist.
The Log Confirmed Execution
We found records similar to:
Sep 2 05:29:01 mail CRON[<PID>]: (zimbra) CMD (/home/SSL/.khp)
This was one of the strongest pieces of evidence in the investigation.
The crontab told us:
/home/SSL/.khp should execute every minute
The log told us:
cron actually executed /home/SSL/.khp as the zimbra user
The process investigation showed:
suspicious processes were running
These findings created a useful evidence chain.
Crontab configuration
|
v
Cron execution recorded in syslog
|
v
Suspicious processes observed
Ubuntu documents /var/log/syslog as a general system log that contains system information when those events are not recorded in more specific logs.
Narrowing the Investigation to a Time Window
Once we knew roughly when the suspicious activity occurred, we searched a smaller section of the logs.
For example:
grep -hE \
'Sep 2 09:(3[0-9]|4[0-9]|5[0-9]).*(CRON|CMD)' \
/var/log/syslog \
/var/log/syslog.1 \
2>/dev/null
This searches for cron activity between approximately:
09:30
and
09:59
on September 2.
Filtering by time becomes useful on production systems because /var/log/syslog can contain a large amount of unrelated information.
Instead of reading thousands of lines manually, you can concentrate on the period surrounding:
- Process creation
- File creation
- Scheduled execution
- Authentication events
- Service failures
- Network activity
This helps build an incident timeline.
Searching the Logs for the Exact Suspicious File
Once we knew the suspicious path, another useful search was:
grep -R '/home/SSL/.khp' /var/log 2>/dev/null
This can reveal whether the same path appears elsewhere in available logs.
You can also search specifically for the process names:
grep -R -E 'javab|idle|\.khp' /var/log 2>/dev/null
Be careful with recursive searches on servers with large log directories because they can consume CPU and storage I/O.
A more targeted search is usually preferable.
Inspecting User Crontabs
The zimbra account was directly relevant to our incident, but other accounts should also be reviewed when investigating persistence.
To inspect a particular user:
crontab -u <USERNAME> -l
For example:
crontab -u root -l
and:
crontab -u zimbra -l
Do not assume that only root matters.
An attacker may gain access to an application account and establish persistence using that account’s permissions.
Cron normally stores user crontabs in its spool area. On Debian and Ubuntu systems, cron uses:
/var/spool/cron/crontabs/
The Debian cron documentation recommends using the crontab command rather than directly modifying files in the spool directory.
Preserving a Suspicious Crontab Before Changing It
Before removing a malicious entry, save a copy.
For example:
mkdir -p /root/incident-evidence
Then:
crontab -u zimbra -l \
> /root/incident-evidence/zimbra-crontab.txt
Record a hash:
sha256sum /root/incident-evidence/zimbra-crontab.txt
You can also record the collection time:
date -Is
This provides a simple record of what existed before you changed the crontab.
These are additional evidence-preservation recommendations. They improve on the initial investigation procedure and should not be interpreted as commands we necessarily executed before every change during the original incident.
Inspecting /etc/crontab
Linux also supports system-wide scheduled jobs.
Inspect:
cat /etc/crontab
or:
less /etc/crontab
A system crontab differs slightly from a normal user’s crontab.
It includes an additional username field.
For example:
*/10 * * * * root /usr/local/bin/example.sh
The fields become:
minute
hour
day-of-month
month
day-of-week
username
command
The Linux crontab(5) documentation describes this difference for system cron files.
Look for:
- Unknown executable paths
- Scripts under writable directories
- Commands using
/tmp - Commands using
/dev/shm - Network download commands
- Encoded shell commands
- Unexpected interpreters
- Recently added entries
- Commands running as unexpected users
Do not assume an unfamiliar entry is malicious. Packages can legitimately install scheduled tasks.
Investigate first.
Inspecting /etc/cron.d/
Ubuntu and Debian systems can also store system cron jobs under:
/etc/cron.d/
List the directory:
ls -lah /etc/cron.d/
Then inspect individual files:
cat /etc/cron.d/<FILE>
You can search all entries:
grep -R . /etc/cron.d/ 2>/dev/null
The Debian cron documentation confirms that cron reads files under /etc/cron.d/ and treats them similarly to /etc/crontab, including the username field.
Check ownership and permissions:
find /etc/cron.d \
-maxdepth 1 \
-type f \
-exec ls -l {} \;
Debian’s documentation states that files under /etc/cron.d should be owned by root and should not be writable by group or others.
Unexpected ownership or writable cron files deserve investigation.
Inspecting /etc/cron.*
Ubuntu commonly uses scheduled-task directories such as:
/etc/cron.hourly/
/etc/cron.daily/
/etc/cron.weekly/
/etc/cron.monthly/
List them:
ls -lah /etc/cron.hourly/
ls -lah /etc/cron.daily/
ls -lah /etc/cron.weekly/
ls -lah /etc/cron.monthly/
Or inspect them together:
ls -lah /etc/cron.*
The Debian implementation uses these directories for hourly, daily, weekly, and monthly scheduled jobs through the system cron configuration.
Useful questions include:
Do I recognize this script?
Which package installed it?
Who owns it?
When was it modified?
Is it executable?
Is it writable by an unexpected user?
Does it execute anything from /tmp, /dev/shm, or a user's home directory?
Check metadata with:
stat /etc/cron.daily/<FILE>
Determine its file type:
file /etc/cron.daily/<FILE>
If you suspect modification, calculate a hash:
sha256sum /etc/cron.daily/<FILE>
Search Cron Locations for Suspicious Paths
If you already know an indicator such as:
/home/SSL
search cron configuration for it:
grep -R '/home/SSL' \
/etc/crontab \
/etc/cron.d \
/etc/cron.hourly \
/etc/cron.daily \
/etc/cron.weekly \
/etc/cron.monthly \
2>/dev/null
Likewise, you can search for known filenames:
grep -R -E 'javab|idle|\.khp|\.rguard' \
/etc/crontab \
/etc/cron.d \
/etc/cron.hourly \
/etc/cron.daily \
/etc/cron.weekly \
/etc/cron.monthly \
2>/dev/null
These commands are recommended additional checks.
Our confirmed persistence finding was specifically the unauthorized entry in the zimbra user’s crontab.
Cron Is Not the Only Scheduler
Modern Ubuntu systems also use systemd timers.
A server may therefore have no suspicious cron entry while still containing a scheduled persistence mechanism.
List systemd timers with:
systemctl list-timers --all
The --all option helps include timers that are currently inactive as well as active timers.
The systemctl documentation describes list-timers as listing timer units and the service units they activate.
Typical output contains information such as:
NEXT
LEFT
LAST
PASSED
UNIT
ACTIVATES
Look for timer units you do not recognize.
Investigating a Suspicious systemd Timer
If you find something suspicious:
systemctl status suspicious.timer
Then inspect its unit definition:
systemctl cat suspicious.timer
Determine what service it activates:
systemctl status suspicious.service
and:
systemctl cat suspicious.service
You can also list timer unit files:
systemctl list-unit-files --type=timer
Search common unit locations:
find \
/etc/systemd/system \
/usr/lib/systemd/system \
/lib/systemd/system \
-type f \
\( -name '*.timer' -o -name '*.service' \) \
-ls 2>/dev/null
These systemd checks are additional recommendations.
We confirmed cron persistence during our Zimbra incident. We are not claiming that we found a malicious systemd timer on that server.
Check the Cron Service Itself
If investigating cron behavior, verify that the service is running:
systemctl status cron
You can also inspect cron-related journal records:
journalctl -u cron
or for a specific time period:
journalctl -u cron \
--since "YYYY-MM-DD HH:MM:SS" \
--until "YYYY-MM-DD HH:MM:SS"
Whether /var/log/syslog, the systemd journal, or both contain the relevant events depends on the server’s logging configuration.
For our incident, /var/log/syslog contained the cron execution evidence we needed.
Do Not Delete the Suspicious File First
Once we discovered:
* * * * * /home/SSL/.khp
it might have been tempting to immediately run:
rm -f /home/SSL/.khp
That would remove a useful piece of evidence.
Before deleting a suspicious file, consider collecting:
stat /home/SSL/.khp
file /home/SSL/.khp
sha256sum /home/SSL/.khp
and:
cp -a /home/SSL/.khp /root/incident-evidence/
You can then hash the preserved copy:
sha256sum /root/incident-evidence/.khp
For a serious business incident, preserve evidence on separate trusted storage where practical.
Do not execute the suspicious file to see what it does.
Containment During Our Incident
We needed to stop the active behavior while continuing the investigation.
After identifying /home/SSL as associated with the suspicious activity, one of the containment measures we used was:
chmod -R a-x /home/SSL
This removes execute permissions recursively from that directory.
We also terminated confirmed unauthorized processes.
These actions helped contain the incident, but they modified the compromised system.
Important Warning
Do not use:
chmod -R a-x <DIRECTORY>
against a directory unless you understand what it contains.
Running it against a legitimate application directory could prevent software from running.
A formal forensic response may also prioritize acquiring evidence before modifying permissions or terminating processes.
Removing a Confirmed Malicious Cron Entry
Once you have preserved the existing crontab and confirmed that an entry is malicious, edit the user’s crontab with:
crontab -u zimbra -e
Remove only the unauthorized entry.
Do not remove legitimate Zimbra scheduled tasks.
Afterward, verify:
crontab -u zimbra -l
Then monitor for recurrence:
pgrep -a javab
pgrep -a idle
and check logs again:
grep -hE 'CRON.*zimbra' \
/var/log/syslog \
/var/log/syslog.1 \
2>/dev/null
If the malicious entry returns after being removed, the server likely contains another mechanism capable of rewriting the crontab.
That should trigger a broader investigation.
An Important Zimbra-Specific Point
Zimbra normally has its own legitimate scheduled jobs.
If you suspect the zimbra crontab has been modified, compare it with expected Zimbra configuration rather than deleting the whole thing.
Zimbra documentation specifically discusses checking the zimbra crontab during compromise investigations.
The key question is not:
Does the Zimbra user have cron jobs?
It should.
The question is:
Are there cron jobs that do not belong to Zimbra or to anything the administrator intentionally installed?
That is what mattered in our case.
Actual Findings Versus Additional Recommendations
It is important to keep evidence separate from general security advice.
What We Actually Confirmed
During our incident, we confirmed:
Suspicious javab and idle processes were running.
Suspicious artifacts existed under locations including /dev/shm and /home/SSL.
The Zimbra user's crontab contained:
* * * * * /home/SSL/.khp
/var/log/syslog showed the zimbra account executing:
/home/SSL/.khp
The suspicious processes returned after being terminated.
We performed process termination and containment activities.
We searched the system for related suspicious files and persistence mechanisms.
These findings came directly from the investigated server.
Additional Checks Recommended in This Article
The following are broader investigation techniques:
Inspect every user's crontab.
Inspect /etc/crontab.
Inspect /etc/cron.d/.
Inspect /etc/cron.hourly/.
Inspect /etc/cron.daily/.
Inspect /etc/cron.weekly/.
Inspect /etc/cron.monthly/.
Inspect systemd timers.
Hash suspicious files.
Collect file metadata.
Capture systemd unit definitions.
Use journalctl where appropriate.
Preserve copies of suspicious configuration before editing it.
These are recommended practices.
They should not be interpreted as additional malicious mechanisms that we confirmed during this particular Zimbra incident.
A Practical Linux Persistence Checklist
If you terminate a suspicious process and it returns, work through something like this:
# 1. Identify the process
ps auxf
pgrep -a <PROCESS_NAME>
# 2. Inspect the executable
readlink -f /proc/<PID>/exe
tr '\0' ' ' < /proc/<PID>/cmdline
echo
# 3. Check the owning user's crontab
crontab -u <USERNAME> -l
# 4. Check system cron
cat /etc/crontab
ls -lah /etc/cron.d/
ls -lah /etc/cron.hourly/
ls -lah /etc/cron.daily/
ls -lah /etc/cron.weekly/
ls -lah /etc/cron.monthly/
# 5. Check systemd timers
systemctl list-timers --all
# 6. Check cron logs on applicable Ubuntu systems
grep -i CRON /var/log/syslog
# 7. Check the journal
journalctl -u cron
# 8. Preserve suspicious files
stat <FILE>
file <FILE>
sha256sum <FILE>
Do not turn this into an automated deletion script.
The purpose is investigation.
Why Persistence Matters More Than the Process Name
The names javab, idle, and .khp were useful indicators during our incident.
But focusing only on names can be dangerous.
An attacker can rename an executable:
systemd-update
apache-helper
java-service
backup
monitor
and make it appear ordinary.
The stronger evidence comes from relationships.
Ask:
Who owns the process?
What executable is running?
Where is it stored?
What started it?
When did it start?
Does it make network connections?
What scheduled job references it?
What logs show its execution?
What else changed at the same time?
In our case, the cron entry supplied one of those missing relationships.
The Key Lesson From This Incident
The command:
pkill javab
solved an immediate symptom.
It did not solve the compromise.
The real breakthrough came when we found:
* * * * * /home/SSL/.khp
and then found corresponding records in /var/log/syslog.
That transformed our understanding of the incident.
We no longer had only a suspicious process.
We had:
a suspicious executable
+
a persistence mechanism
+
execution logs
+
recurring suspicious activity
Those pieces could be correlated into an incident timeline.
Containment Does Not Restore Trust
Removing the cron entry, disabling the suspicious files, and terminating the processes can stop the visible behavior.
It does not prove the server is clean.
If an unauthorized party had enough access to:
write executable files
modify an application user's crontab
run processes under that account
or modify other account configuration
you should assume that other changes may exist until proven otherwise.
For a critical mail server, the long-term response may require:
- Preserving incident evidence.
- Identifying affected accounts and credentials.
- Rotating passwords, SSH keys, API credentials, and other secrets where necessary.
- Reviewing application and operating-system logs.
- Checking neighboring systems for related indicators.
- Building a clean replacement server.
- Applying current security patches.
- Restoring only trusted data and configuration.
- Monitoring the replacement system for recurrence.
Removing the visible malware should not automatically return a compromised production server to trusted status.
Lessons Learned
Our Zimbra incident produced several useful Linux security lessons.
First, killing a malicious process is containment, not persistence removal.
If the process returns, investigate what launches it.
Second, inspect scheduled tasks belonging to application accounts.
The suspicious processes were associated with the zimbra account, and its crontab contained the critical persistence entry.
Third, correlate configuration with logs.
The crontab showed what was configured.
/var/log/syslog showed that cron executed it.
The process list showed the resulting activity.
Fourth, preserve evidence before changing the system.
Crontabs, scripts, file hashes, timestamps, logs, and process information can help reconstruct the incident.
Fifth, cron is only one persistence mechanism.
Modern Linux administrators should also inspect systemd services and timers.
Sixth, do not remove every unfamiliar scheduled job.
Linux packages and Zimbra itself legitimately create scheduled tasks.
Investigate ownership, purpose, installation source, timestamps, and behavior first.
Finally, visible malware is rarely the entire investigation.
The important question is not simply how to kill a suspicious process.
The important question is:
What caused it to run, and what can make it run again?
For us, the answer was hidden in one line:
* * * * * /home/SSL/.khp
That line explained why the suspicious activity kept returning and changed the direction of the investigation.
References
Zimbra: Investigating and Securing Systems
Zimbra’s security investigation guide includes checks for unusual processes, unauthorized accounts, suspicious activity, and modified zimbra crontab entries.
Zimbra Investigating and Securing Systems
Linux crontab(5) Manual
Documents cron scheduling fields, * syntax, user crontabs, system cron entries, and /etc/cron.d behavior.
Linux crontab(5) manual at man7.org
Debian cron(8) Manual
Documents cron behavior on Debian-based systems, including /var/spool/cron/crontabs, /etc/crontab, /etc/cron.d, and the hourly, daily, weekly, and monthly cron directories. Ubuntu inherits much of this Debian cron structure.
Linux systemctl(1) Manual
Documents systemctl list-timers and management of systemd timer units.
Linux systemctl(1) manual at man7.org
Ubuntu: Viewing and Monitoring Log Files
Ubuntu documentation describing /var/log/syslog, /var/log/auth.log, and other standard Linux log locations.
Ubuntu Viewing and Monitoring Log Files
SEO and Social Metadata
SEO Title: Finding Linux Malware Persistence: How Cron Reinfected Our Zimbra Server
SEO Description: See how a malicious cron job kept restarting suspicious processes on a compromised Zimbra server and how we traced the persistence through Linux logs.
Social Title: We Killed the Malware, But It Kept Coming Back
Social Description: Suspicious processes kept returning on our Zimbra server. The cause was a hidden cron persistence mechanism running every minute. Here is how we found it.
Focus Keyword: Linux malware persistence
Tags: Zimbra, Linux Security, Linux Malware, Cron, Crontab, Ubuntu Server, Zimbra Security, Incident Response, Malware Persistence, Linux Administration, Systemd, Server Security
