Kristian Lyngstøl's Blog

Tools of the trade - Job control

Posted on 2012-11-03

About the Tools of the trade series

After over a decade of using GNU/Linux, you pick up a few tricks. They become second nature to you. You don't even think about them when you're using them. They enter your regular tool chest, so to speak.

This blog post is the first in a series of what I hope to be many posts where I introduce basic tools, tricks and techniques to those of you who are less experienced with GNU/Linux. The goal is not to make you an expert on the tools, but to get you started and show you a few use cases.

As I hope to make this a series, please let me know if the style, topic and level of detail is appropriate, or if there are any particular topics that you're interested in.

If you enjoy the series, feel free to subscribe to the RSS, either for the entire blog (http://kly.no/feed.xml) or just these TOTT (tools of the trade) posts (http://kly.no/feedtott.xml). And of course, I'd appreciate it if you helped me spread the word to others who could find these posts interesting, even if they might not be for you.

Now, let's get started with job control and screen, two very simple tools that can make your life easier.

Job control, you say?

How often do you do this:

  • Open service_foo.conf
  • Edit
  • Save and close service_foo.conf
  • Restart the service foo
  • Get a syntax error
  • Reopen service_foo.conf
  • Navigate to the same position you were at
  • Edit
  • Save
  • Try restart,
  • etc etc

It's pretty common.

Or:

$ long_running_command
# Darn, should've started it in the background instead!
CTRL-C
$ long_running_command &

All of these situations can be dealt with using basic job control in your shell. Most proper shells have some job control, but since it's by far the most common shell, we'll talk about how bash handles it.

It's actually very simple. Here's what you need to know:

Action Effect
CTRL-Z Stops the currently active job
$ jobs Lists all jobs and their state
$ fg Wakes up the job most frequently stopped
$ fg x Wakes up job x, where x can be seen using the jobs command.
$ bg Sends the job most frequently stopped to the background, as if you started it with &.
$ bg x Sends job x to the background.

A job can be any command that would normally run in the foreground. You can also use %prefix instead of the job number, where the prefix is the command you started. For instance if you run man bash to read up on job control, then stop it, you could resume the job with fg %man.

Stopping a job is not the same as putting it in the background. When you stop a job, it actually stops running. For your editor, this doesn't matter. Here's a simple example where I just have a script output the time:

kristian@luke:~$ ( while sleep 1; do date ; done )
Sat Nov  3 03:05:16 CET 2012
Sat Nov  3 03:05:17 CET 2012
Sat Nov  3 03:05:18 CET 2012
Sat Nov  3 03:05:20 CET 2012
Sat Nov  3 03:05:21 CET 2012
Sat Nov  3 03:05:22 CET 2012
Sat Nov  3 03:05:23 CET 2012
^Z
[2]+  Stopped                 ( while sleep 1; do
    date;
done )
kristian@luke:~$ date
Sat Nov  3 03:05:33 CET 2012
kristian@luke:~$ jobs
[1]-  Stopped                 man bash
[2]+  Stopped                 ( while sleep 1; do
    date;
done )
kristian@luke:~$ fg 2
( while sleep 1; do
    date;
done )
Sat Nov  3 03:05:42 CET 2012
Sat Nov  3 03:05:43 CET 2012
Sat Nov  3 03:05:44 CET 2012
Sat Nov  3 03:05:45 CET 2012
^C

Notice how there is no time stamps printed for the time the command was stopped.

If you wanted that, you would have to put the job in the background. When you do put jobs in the background their output will generally pop up in your shell, just like what would happen if you use & without redirecting output.

There are a few shortcuts to job control too, though I personally don't use them. Take a look at the Job Control chapter in man bash for more.

screen

Using your shell's job control is great for manipulating jobs within a single open shell. But it has many limitations too. And it doesn't allow you to stop a job in one shell and open it up again in an other (perhaps at a later time from an other machine).

Screen is most famous for allowing you to keep programs running even if you lose your connection.

Screen is a simple wrapper around any command you run. You typically start screen with just screen and end up in a plain shell. You can also start a single command directly, for instance using screen irssi. Under the hood you've now created a screen "server" which is what your applications are connected to, and a screen "client" which is what your terminal is looking at. If you close your terminal, the client will stop, but the server will keep running and the applications inside it will be unaware of the disappearance of the terminal. You can also detach from the server manually by hitting ctrl-a d. All screen-bindings start with ctrl-a. I'll have a little list further down.

Here's a demo:

kristian@luke:~$ cat screen-demo.sh
#!/bin/bash
while sleep 1; do
        date | tee -a screen-demo.log;
done
kristian@luke:~$ screen ./screen-demo.sh

(date printing starts)
^A d (detach)

[detached from 24859.pts-3.luke]
kristian@luke:~$ date
Sat Nov  3 03:24:53 CET 2012
kristian@luke:~$  tail -n 2 -f screen-demo.log
Sat Nov  3 03:24:53 CET 2012
Sat Nov  3 03:24:54 CET 2012
Sat Nov  3 03:24:55 CET 2012
Sat Nov  3 03:24:56 CET 2012
Sat Nov  3 03:24:57 CET 2012
Sat Nov  3 03:24:58 CET 2012
(keeps running)

The basics of screen are:

  • screen starts screen with a regular shell.
  • screen app starts screen running app. The app-argument can include arguments. screen irssi -! will start screen and irssi -!.
  • All screen-commands start with CTRL-a (^A).
  • CTRL-a d (^A d) detaches from screen. This happens automatically if you close the terminal or your ssh connection breaks or similar.
  • screen -r re-attaches to a screen session. If you have multiple screens running you will have to specify which one (it will prompt you to).
  • screen -r -d re-attaches to a screen session that you are still attached to somewhere else. This means that if you ssh to a server at work and open screen but forget to close it, you can take over that screen session when you get home for example.
  • screen -x attaches to a screen session without detaching any other screen clients. A good use case is ssh'ing to a server, starting screen and having your customer do the same with screen -x so he can see exactly what you're doing and even type himself. It's quite cool, so try it out!

Screen can also have multiple 'windows' inside a session. I mostly use "full screen windows" as they are simplest. Try it out while running screen:

  • Hit ^A c to create a new window.
  • Hit ^A n to go to the next window.
  • Hit ^A p to go to the previous window.
  • Hit ^A a to go to the window you were at last.

You can also show multiple windows at the same time (split screen) and jump to specific windows if you have many (e.g: jump from window 1 to 6 without going through window 2, 3, 4 and 5.). Check the screen manual page for more.

Screen has some quirks with regards to scrolling, though, so you may want to check out the man page for that too.

Tip

Ever need to re-configure network stuff over ssh?

Run the commands in screen.

What I often do is something along the lines of: ifdown eth0; sleep 5; ifup eth0; sleep 60 && ifconfig eth0 some-safe-ip for instance. This ensures that the commands run even if the connection drops. It also allows you to regain your old session if you have to reconnect.

Minor tips

  • Use tee file if you both want to write the out to a file (echo blatti > foo) and want to see the output at the same time. tee -a will append instead of overwrite, similar to what >> does.
  • Simple bash loops are wonderful for testing. I use variations of while true; do blatti; done, while sleep 1; do blatti; done; and for a in foo bar bat; do echo $a; done frequently.