Processes and jobs
nohup
Run a command immune to the hangup signal, so it keeps running after you log out or close the terminal.
nohup COMMAND [ARG...]
When a terminal closes or an SSH connection drops, the session's processes receive HUP
(hangup), and by default HUP terminates a program. nohup starts a command with HUP
ignored, so it survives the hangup. It also deals with the terminal the command would have
written to: if standard output is a terminal, output goes to nohup.out in the current
directory (or ~/nohup.out), standard error follows it, and input is taken from /dev/null.
When output is already redirected, nohup leaves it alone and prints nothing about it.
The idiom is nohup ./job.sh > job.log 2>&1 &: ignore hangups, choose the log file yourself,
and put it in the background. nohup does not background anything by itself; the & does.
nohup is the oldest and most portable tool for the job, but not the only one: disown (bash,
zsh) detaches an already-running job from the shell, setsid starts a command in a new session
with no controlling terminal, and tmux or screen keep a whole terminal alive. For services that
must restart after a crash or a reboot, a systemd unit or another supervisor is the right
answer. The examples run without a terminal, so where it matters they use script to give
nohup a pseudo-terminal like the one you would have at a prompt.
Examples
nohup with output that is not a terminal
nohup echo hello
ls nohup.out
hello ls: cannot access 'nohup.out': No such file or directory
Here standard output is not a terminal (it is captured to show on this page), so nohup
leaves it alone: hello appears normally, no message is printed and no nohup.out is
created. This is also what happens in cron jobs and scripts whose output is redirected.
nohup at an interactive prompt
script -qc 'nohup echo hello' /dev/null | tr -d '\r'
cat nohup.out
nohup: ignoring input and appending output to 'nohup.out' hello
script runs the command on a pseudo-terminal, as at a real prompt. nohup then tells you
on standard error that input is ignored and output is appended to nohup.out, and the
hello lands in that file. tr -d '\r' only removes the terminal's carriage returns.
Run a long job in the background with its own log
#!/bin/bash
for step in 1 2 3; do
echo "step $step done"
sleep 0.2
done
echo "error: disk almost full" >&2
nohup bash job.sh > job.log 2>&1 &
echo "started job $!"
wait
cat job.log
started job 1641 step 1 done step 2 done step 3 done error: disk almost full
> job.log 2>&1 sends both output streams to a file you chose, & returns the prompt at
once, and nohup makes the job ignore the hangup when you disconnect. After logging back
in, tail -f job.log follows its progress.
See that the hangup signal is ignored
bash -c 'kill -HUP $$; echo "survived"'
echo "without nohup: exit status $?"
nohup bash -c 'kill -HUP $$; echo "survived"'
echo "with nohup: exit status $?"
/bin/bash: line 1: 1646 Hangup bash -c 'kill -HUP $$; echo "survived"' without nohup: exit status 129 survived with nohup: exit status 0
Each inner shell sends HUP to itself, standing in for a closed terminal. Without nohup it
dies (Hangup, status 129 = 128 + 1). Under nohup the signal is ignored, and the ignored
setting is inherited by everything the command starts.
Where standard error goes
script -qc 'nohup ls README.md missing.txt > ls.log' /dev/null | tr -d '\r'
cat ls.log
nohup: ignoring input and redirecting stderr to stdout ls: cannot access 'missing.txt': No such file or directory README.md
Standard output is redirected but standard error is still the terminal, so nohup sends
standard error into the same place as standard output and says so. To keep them apart, give
both redirections yourself (> out.log 2> err.log).
nohup cannot run functions or builtins
backup() { tar czf backup.tgz data; }
nohup backup
nohup: failed to run command 'backup': No such file or directory
nohup is an ordinary program that executes a file, so shell functions, aliases and
builtins are invisible to it: exit status 127, command not found. Put the logic in a
script, or run nohup bash -c '...'.
Detach completely with setsid
setsid -f sleep 300
ps -o pid,ppid,sid,args -C sleep
PID PPID SID COMMAND 1656 1 1656 sleep 300
setsid -f forks and starts the command in a new session: its session ID (SID) is its
own PID, so no terminal hangup can reach it, and the shell is no longer its parent. Unlike
nohup, it does not touch the output, so redirect that yourself.
Common pitfalls
Writing
nohup cmdwithout&and wondering why the prompt does not come back.nohuponly ignores the hangup; add&to background it.Filling the disk with an ever-growing
nohup.outin whatever directory you happened to be in. Always redirect:nohup cmd > /var/log/cmd.log 2>&1 &.Expecting
nohupto protect against everything. It only ignoresHUP: the job still dies fromTERM,kill -9, a reboot or the out-of-memory killer, and some programs reinstall aHUPhandler of their own. systemd-logind can also be configured to kill all of a user's processes at logout (KillUserProcesses=yes).Running a job under
nohupthat reads from the terminal. Its input is/dev/null, so prompts get end-of-file; pass answers through arguments or a file.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
On macOS and the BSDs,
nohupbehaves the same way (POSIX defines it), includingnohup.out. The exact messages differ; theignoring inputpart is GNU's wording.setsidis a util-linux tool and is not installed on macOS.
Related
jobsList the background and stopped jobs of the current shell, with their job numbers and state.bgResume a stopped job in the background, as if it had been started with &.killSend a signal to processes by PID or job number; by default TERM, which asks a process to exit.timeoutRun a command with a time limit, and kill it if it is still running when the time is up.sleepPause for a given amount of time, in seconds by default, with m, h and d suffixes and fractions allowed.- RedirectionHow >, >>, 2>, 2>&1, &>, < and exec move a command's input and output between files and file descriptors, and why the order matters.
- Traps and signalsRunning cleanup code with trap on EXIT, handling INT and TERM, the ERR pseudo-signal, and the timing and scoping rules that trip people up.