scp tricks

scp tricks

If you don’t know about scp,
then today is your lucky day. It is a great tool for copying file around.
In the following examples, here are the terms I will use:

  • remotebox is the IP address or hostname of the remote box.

Here are a few pratical examples to help you get started:

  • copy a file to a remote system:

    scp a remotebox:

  • copy a file to a remote system, and put it somewhere other than your home directory:

    scp a remotebox:/var/tmp

    That will copy a from your current working directory to
    /var/tmp on the remote box.

  • copy a file to a remote system but a different user id:

    scp a mark@remotebox:

  • copy file from a remote system:

    scp remotebox:a .

  • copy file from a remote system, different login:

    scp mark@remotebox:a .

  • copy recursively from a remote system, different login:

    scp -r mark@remotebox:a .

  • copy a number of files from a remote system to a specified directory locally:

    scp remotebox:\{a,b,c\} ~/my/secret/dir

If you have more good ideas about scp, please put them in the article comments. Thank you.

23 thoughts on “scp tricks”

  1. Amit Chakradeo

    Hi,

    scp uses ssh as its transport for copying files between different machines. You can set up RSA/DSA keys for passwordless login across hosts and then use scp without entering passwords, just like local cp. You can also make use of ssh-agent to achieve passwordless file copying.

    -Amit

    1. For moreAmit Chakradeo wrote:
      >
      > scp uses ssh as its transport for copying files between
      > different machines. You can set up RSA/DSA keys for
      > passwordless login across hosts and then use scp without
      > entering passwords, just like local cp. You can also make use
      > of ssh-agent to achieve passwordless file copying.

      For more on that, see <a href="/secure-file-copy.php">How to copy files around without anyone seeing them</a>.

    2. For working password-less, I highly recommend /usr/ports/security/safesh. This helps you maintain different credentials for different hosts, which avoids identity theft by the machines you connect to.

      It should also automates key generation etc.

      Feedback about it the program is welcome.

      Eivind.

  2. First off, anyone who uses ssh or scp with lots of different hosts should take the time to read the ssh_config(5) man page. It is very useful if you have accounts with different usernames, see the ‘User’ parameter.

    Ok, now, since this article was about scp, let me talk about some nice things you can do with ssh. You all know that copying a whole directory tree sometimes causes problems, and that is why it is common to use tar to ensure permissions, ownership and symlinks are handled correctly. The example in tar(1) is:

    tar -cf – -C srcdir . | tar xpf – -C destdir

    Now, consider this when the destination is a remote host. Ssh can be used to make this very simple, just say:

    tar -cf – -C srcdir . | ssh remote_host tar xpf – -C destdir

    where ‘srcdir’ is on the local host, and destdir is on the remote.
    Similarly, to copy from a remote host:

    ssh remote_host tar -cf -C srcdir . | tar xpf – -C destdir

    Another way I use ssh is to redirect output of a command on one host to a file on another host. For example, if I was compiling something and for some reason wanted the make log on another computer, I would run (using a Bourne shell):

    make buildworld 2>&1 | ssh remote_host cat \> /tmp/make.log

    The output of buildworld is sent over ssh to the cat process on the remote host, which just writes it to it’s stdout. In this case it’s stdout is redirected to /tmp/make.log. Make sure you escape the redirect (>) on the command line so it is evaluated by the remote not the local host.

    Have fun.
    -James

    1. Oops there is a missing – in the second ssh+tar example. It should read:

      ssh remote_host tar -cf – -C srcdir . | tar xpf – -C destdir

      not

      ssh remote_host tar -cf -C srcdir . | tar xpf – -C destdir

      -James

    2. pierre-yves verdon

      i use quite the same trick but i finally gzip the transfered directory.
      tar -zcvf – DIRECTORY/ | /usr/bin/ssh -l LOGIN HOST ‘cat – > file.tgz’

      it’s also possible to use ssh with rsync:
      rsync -rltgpvz –exclude=log/ -e ssh LOGIN@HOST:/remote_dir .

      -pierre-yves

    3. I keep this little script on my machines. Sucks files into the current directory.

      /usr/local/bin/sshtar
      #!/bin/sh
      if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
      printf "\n\n\tExample:\n\n\t\tsshtar username hostname_or_ip /dir/to/copy\n\n"
      exit 0
      fi
      /usr/local/bin/ssh $1@$2 "cd $3 ; tar cf – ." | tar xvfBp –

    4. When using stdio over ssh, it’s sometimes a good idea to use "ssh -e none" to make it have no escape character, as there is a non-zero chance your data may contain the relevant magic to make ssh exit (:

      Also you can do fun things like:

      tar cf – /home | ssh -e none user@host \( cd /home \; tar xvf – \)

      to change dir when untarring stuff over ssh.

      1. frank wrote:

        > useless use of cat award?

        Comments such as the above fall into the "I’m smarter than you!" category. Without explanation, they are worthless.

        Frank, perhaps if you could explain why you think it is useless people might learn.


        The Man Behind The Curtain

  3. If you are running a copy in the background or from a demon, us ethe options

    -o ‘BatchMode=yes’

    which will make sure it won’t try and prompt. A copy which needs a human response will just fail.

    If you are using ssh-agent then rather than running it from each login and relying on inheretence of environment variables to pass the location of the agent around, put the output of ssh-agent into a file

    ssh-agent -s > "$agent_file"

    And then in your .bashrc, or wherever, load the settings
    . "$agent_file"

    This way all your shells will have access to the agent, including for instance if you connect in from a windows box. Also your scripts can source the same file and have access to keys in the agent.

  4. Gaspar Chilingarov

    This also will do good job for you!

    scp user@remotehost:’`echo foo*bar[0-9]`’ ./

    note the backtriks passed to remote host – you can run any command with this. like

    scp user@remotehost:’`find / -mtime -1`’ ./

    will copy all files which are modified less than in one day to local side.

    But also – take care when using this method. Files with same names, but located in different directories will override each other.

  5. scp may come in handy for local files copies as well, when you want to copy files to another account on the same system where you don;’t have permission, and root access is not desired or available:

    scp file.txt otheruser@localhost:/dst/dir

    You could use one of the other tips to set this up to be passwordless if you use it frequently.

  6. pattern matching can be performed with a star ‘*’ if you first escape it with a backslash ‘\’:

    scp remote_host:\*.gz .

    or

    scp remove_host:mail/example\*folder

  7. Once you’ve copied your keys about so that you don’t need
    passwords (hey that’s really not very secure though!) then you can get filename completion for scp with zsh.

    If you installed zsh from the ports collection and your fpath
    variable is properly configured then you should be able to do:

    autoload -U compinit
    compinit

    then try scp user@host:<TAB>

    Don’t look as surprised as I did though, I found it by accident!

    1. Stephen Roome wrote:
      >
      > Once you’ve copied your keys about so that you don’t need
      > passwords (hey that’s really not very secure though!) then
      > you can get filename completion for scp with zsh.

      It is very secure if you have passphrases on your private key. In fact, it’s more secure than having a password.

  8. Here is a little script I use every day. It created 3 links to the script: rget, rput and rs. calling the script by the different links does different things:

    rput: scp a file to a remote machine. For example if you are in /home/foo and you type:

    rput bar

    then this expands to:

    scp bar example.com:/home/foo/bar

    rget reverses this to copy from the remote machine to the local machine.

    And rs calls rsync and pushes all files in the current directory to the remote directory. rs requires a file named .rs in the current directory– this file is a list of all files to exclude from the rsync

    #!/bin/sh
    # $Id: publish.sh,v 1.1 2002/11/24 14:49:27 culley Exp $
    #
    #

    host="example.com"
    dir=`pwd`

    script=$0

    case $script in

    *rput)
    cmd="scp $1 $host:$dir/$1"
    break
    ;;

    *rget)
    cmd="scp $host:$dir/$1 ."
    break
    ;;

    *rs)
    echo "rsync -zvt –modify-window=15 –progress -e ssh –exclude-from=$dir/.rs * $host:$dir/"
    rsync -zvt –modify-window=15 –progress -e ssh –exclude-from=$dir/.rs * $host:$dir/
    exit;
    ;;

    *)
    echo valid commands:
    echo " rs <file_names> using rsync send all changes in the current directory to the server."
    echo " rput <file_name> using scp send a singlular file to the server"
    echo " rget <file_name> using scp get a singlular file from the server"
    exit
    ;;

    esac

    echo $cmd
    exec $cmd

  9. Philipp Friedrich

    not exactly a scp feature but more one of vim

    vim scp://remotehost/file.txt

    makes vim get the file via ssh and after saving it upload it again. so its actually not remote editing but quite comfortable sometimes

    this works also with vim ftp://host/file

  10. For a more efficient ssh recursive copy, use a tar pipe:

    tar cf – dirname | ssh user@host ‘cd /path/to/destination/root ; /usr/bin/tar -xf -‘

    Rather than sending each file individually, this lets you send one tar archive, and unpack on the other end.

Leave a Comment

Scroll to Top