Mar 072003
scp tricks
If you don’t know about scp,
then today is your lucky day. It is a great tool for copying file around.
In the following examples, here are the terms I will use:
remoteboxis the IP address or hostname of the remote box.
- copy a file to a remote system:
scp a remotebox: - copy a file to a remote system, and put it somewhere other than your home directory:
scp a remotebox:/var/tmpThat will copy
afrom your current working directory to/var/tmpon the remote box. - copy a file to a remote system but a different user id:
scp a mark@remotebox: - copy file from a remote system:
scp remotebox:a . - copy file from a remote system, different login:
scp mark@remotebox:a . - copy recursively from a remote system, different login:
scp -r mark@remotebox:a . - copy a number of files from a remote system to a specified directory locally:
scp remotebox:\{a,b,c\} ~/my/secret/dir
Hi,
scp uses ssh as its transport for copying files between different machines. You can set up RSA/DSA keys for passwordless login across hosts and then use scp without entering passwords, just like local cp. You can also make use of ssh-agent to achieve passwordless file copying.
-Amit
For moreAmit Chakradeo wrote:
>
> scp uses ssh as its transport for copying files between
> different machines. You can set up RSA/DSA keys for
> passwordless login across hosts and then use scp without
> entering passwords, just like local cp. You can also make use
> of ssh-agent to achieve passwordless file copying.
For more on that, see <a href="/secure-file-copy.php">How to copy files around without anyone seeing them</a>.
For working password-less, I highly recommend /usr/ports/security/safesh. This helps you maintain different credentials for different hosts, which avoids identity theft by the machines you connect to.
It should also automates key generation etc.
Feedback about it the program is welcome.
Eivind.
First off, anyone who uses ssh or scp with lots of different hosts should take the time to read the ssh_config(5) man page. It is very useful if you have accounts with different usernames, see the ‘User’ parameter.
Ok, now, since this article was about scp, let me talk about some nice things you can do with ssh. You all know that copying a whole directory tree sometimes causes problems, and that is why it is common to use tar to ensure permissions, ownership and symlinks are handled correctly. The example in tar(1) is:
tar -cf – -C srcdir . | tar xpf – -C destdir
Now, consider this when the destination is a remote host. Ssh can be used to make this very simple, just say:
tar -cf – -C srcdir . | ssh remote_host tar xpf – -C destdir
where ‘srcdir’ is on the local host, and destdir is on the remote.
Similarly, to copy from a remote host:
ssh remote_host tar -cf -C srcdir . | tar xpf – -C destdir
Another way I use ssh is to redirect output of a command on one host to a file on another host. For example, if I was compiling something and for some reason wanted the make log on another computer, I would run (using a Bourne shell):
make buildworld 2>&1 | ssh remote_host cat \> /tmp/make.log
The output of buildworld is sent over ssh to the cat process on the remote host, which just writes it to it’s stdout. In this case it’s stdout is redirected to /tmp/make.log. Make sure you escape the redirect (>) on the command line so it is evaluated by the remote not the local host.
Have fun.
-James
Oops there is a missing – in the second ssh+tar example. It should read:
ssh remote_host tar -cf – -C srcdir . | tar xpf – -C destdir
not
ssh remote_host tar -cf -C srcdir . | tar xpf – -C destdir
-James
i use quite the same trick but i finally gzip the transfered directory.
tar -zcvf – DIRECTORY/ | /usr/bin/ssh -l LOGIN HOST ‘cat – > file.tgz’
it’s also possible to use ssh with rsync:
rsync -rltgpvz –exclude=log/ -e ssh LOGIN@HOST:/remote_dir .
-pierre-yves
I keep this little script on my machines. Sucks files into the current directory.
/usr/local/bin/sshtar
#!/bin/sh
if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
printf "\n\n\tExample:\n\n\t\tsshtar username hostname_or_ip /dir/to/copy\n\n"
exit 0
fi
/usr/local/bin/ssh $1@$2 "cd $3 ; tar cf – ." | tar xvfBp –
When using stdio over ssh, it’s sometimes a good idea to use "ssh -e none" to make it have no escape character, as there is a non-zero chance your data may contain the relevant magic to make ssh exit (:
Also you can do fun things like:
tar cf – /home | ssh -e none user@host \( cd /home \; tar xvf – \)
to change dir when untarring stuff over ssh.
useless use of cat award?
frank wrote:
> useless use of cat award?
Comments such as the above fall into the "I’m smarter than you!" category. Without explanation, they are worthless.
Frank, perhaps if you could explain why you think it is useless people might learn.
—
The Man Behind The Curtain
If you are running a copy in the background or from a demon, us ethe options
-o ‘BatchMode=yes’
which will make sure it won’t try and prompt. A copy which needs a human response will just fail.
If you are using ssh-agent then rather than running it from each login and relying on inheretence of environment variables to pass the location of the agent around, put the output of ssh-agent into a file
ssh-agent -s > "$agent_file"
And then in your .bashrc, or wherever, load the settings
. "$agent_file"
This way all your shells will have access to the agent, including for instance if you connect in from a windows box. Also your scripts can source the same file and have access to keys in the agent.
This also will do good job for you!
scp user@remotehost:’`echo foo*bar[0-9]`’ ./
note the backtriks passed to remote host – you can run any command with this. like
scp user@remotehost:’`find / -mtime -1`’ ./
will copy all files which are modified less than in one day to local side.
But also – take care when using this method. Files with same names, but located in different directories will override each other.
scp may come in handy for local files copies as well, when you want to copy files to another account on the same system where you don;’t have permission, and root access is not desired or available:
scp file.txt otheruser@localhost:/dst/dir
You could use one of the other tips to set this up to be passwordless if you use it frequently.
pattern matching can be performed with a star ‘*’ if you first escape it with a backslash ‘\’:
scp remote_host:\*.gz .
or
scp remove_host:mail/example\*folder
You can also enclose them in single quotes
scp ‘remotehost:*.gz’ .
Once you’ve copied your keys about so that you don’t need
passwords (hey that’s really not very secure though!) then you can get filename completion for scp with zsh.
If you installed zsh from the ports collection and your fpath
variable is properly configured then you should be able to do:
autoload -U compinit
compinit
then try scp user@host:<TAB>
Don’t look as surprised as I did though, I found it by accident!
after the scp user@host:–TABCOMPLETE–
I put < TAB > and forgot this strips quotes…
Stephen Roome wrote:
>
> Once you’ve copied your keys about so that you don’t need
> passwords (hey that’s really not very secure though!) then
> you can get filename completion for scp with zsh.
It is very secure if you have passphrases on your private key. In fact, it’s more secure than having a password.
Here is a little script I use every day. It created 3 links to the script: rget, rput and rs. calling the script by the different links does different things:
rput: scp a file to a remote machine. For example if you are in /home/foo and you type:
rput bar
then this expands to:
scp bar example.com:/home/foo/bar
rget reverses this to copy from the remote machine to the local machine.
And rs calls rsync and pushes all files in the current directory to the remote directory. rs requires a file named .rs in the current directory– this file is a list of all files to exclude from the rsync
#!/bin/sh
# $Id: publish.sh,v 1.1 2002/11/24 14:49:27 culley Exp $
#
#
host="example.com"
dir=`pwd`
script=$0
case $script in
*rput)
cmd="scp $1 $host:$dir/$1"
break
;;
*rget)
cmd="scp $host:$dir/$1 ."
break
;;
*rs)
echo "rsync -zvt –modify-window=15 –progress -e ssh –exclude-from=$dir/.rs * $host:$dir/"
rsync -zvt –modify-window=15 –progress -e ssh –exclude-from=$dir/.rs * $host:$dir/
exit;
;;
*)
echo valid commands:
echo " rs <file_names> using rsync send all changes in the current directory to the server."
echo " rput <file_name> using scp send a singlular file to the server"
echo " rget <file_name> using scp get a singlular file from the server"
exit
;;
esac
echo $cmd
exec $cmd
not exactly a scp feature but more one of vim
vim scp://remotehost/file.txt
makes vim get the file via ssh and after saving it upload it again. so its actually not remote editing but quite comfortable sometimes
this works also with vim ftp://host/file
?
For a more efficient ssh recursive copy, use a tar pipe:
tar cf – dirname | ssh user@host ‘cd /path/to/destination/root ; /usr/bin/tar -xf -‘
Rather than sending each file individually, this lets you send one tar archive, and unpack on the other end.
I guess I should probably read all of the postings first. Sorry for the dupe.