Remote Connection Guide

This article tries to explain how to setup a connection to remote computers. A computer is remote when you are not controlling it with an attached mouse or keyboard but indirectly over some kind of network device.

For the typical OS to be found there are slightly different steps to do in order to set up a remote connection:

Before continuing with this section, make yourself comfortable with the UNIX command line environment.

SSH is a command line tool in its original form. And you'll mostly be only able to work in a command line environment on a remote computer.

Linux

for Mac/OS, UNIX, BSD: the setup should be quite similar to Linux

SSH

SSH, or Secure Shell, is a protocol used to securely log onto remote systems, such as Neumann. It is the most common way to access remote Linux and Unix-like servers. In this guide, we will discuss how to use SSH to connect to a remote system.

Basic Syntax

The tool on Linux for connecting to a remote system using SSH is called, unsurprisingly, ssh. The most basic form of the command is:

$ ssh remote_username@remote_host

The remote_host in this example is the IP address or domain name (141.44.8.59 for Neumann) that you are trying to connect to. remote_username is the user name of your account on the remote server. Once you have connected to the server, you will probably be asked to verify your identity by providing a password. Later, we will cover how to generate key pairs to be used instead of passwords.

To log out from a remote terminal run:

$ exit

How Does Key-based Authentication Work?

Key-based authentication works by creating a pair of keys: a private key and a public key. The private key is located on the client machine and is secured and kept secret. The public key can be given to anyone or placed on any server you wish to access. When you attempt to connect using a key-pair, the server will use the public key to create a message for the client computer that can only be read with the private key. The client computer then sends the appropriate response back to the server and the server will know that the client is legitimate. This entire process is done in the background automatically after you set up keys.

The contents of this section have been adapted from Justin Ellingwood, 2013
licenced under CC BY-NC-SA 4.0

Setting up a passwordless SSH connection

  1. Create a SSH key pair on your local machine with ssh-keygen. Open a Terminal and type the following command (after the $ character). Now press Enter three times to use the default location, and to set no passphrase. E.g. StarCCM+ will not be able to connect to a remote computer if you do set passwords for the key.
    ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/user/.ssh/id_rsa):     press Enter here (only if you don't have a key pair already!)
    Enter passphrase (empty for no passphrase):                       press Enter here
    Enter same passphrase again:                                      press Enter here
    Your identification has been saved in test.
    Your public key has been saved in test.pub.
    The key fingerprint is:
    SHA256:VbvpxtFz3adfafafasdfasdfp4ScZbiBY+jfh84 user@computer
    The key's randomart image is:
    +---[RSA 2048]----+
    |B.+o........oo   |
    |BOo  . =   . .o  |
    |     E     .. . .|
    |             ..o |
    +----[SHA256]-----+
  2. You now have a public key (id_rsa.pub) and a private key (id_rsa) in the hidden .ssh directory in your home directory. Never share the private key.
  3. On your local machine, check if the command ssh-copy-id is available. E.g. run
    which ssh-copy-id

    when a path to the command is shown, then it is available.

  4. If ssh-copy-id is available run the following command to finish quickly. It is a tool which copies your login credentials to a remote machine.
    ssh-copy-id -i ~/.ssh/id_rsa user@141.44.8.59

    After the argument -i the path to the key pair is expected, by default it should be in ~/.ssh/id_rsa. If you have changed the location, or used a different name for the key pair, insert the respective path here. The next argument is the remote computer's IP address and your account on the remote computer, such as user.

  5. If you don't have ssh-copy-id, you have to do a bit more manual work:
    1. Copy the public key (id_rsa.pub) to the remote computer.
      For example:
      scp ~/.ssh/id_rsa.pub user@141.44.8.59:/home/user/ 

      Type your password for the remote computer now, to authorize the file transfer.

    2. Now log into the remote computer via ssh:
      ssh user@141.44.8.59 
    3. On the remote computer, add the public key to the authorized_keys file
      cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
    4. Now refresh the authorized_keys using command
      chmod 600 .ssh/authorized_keys
    5. To efficiently use remote clusters, there is an additional key pair necessary.
      On the remote computer run the following commands again (copy-paste them):
      1. Create a second keypair. Press enter three times after running this command:
        ssh-keygen
      2. Add the new key to authorized_keys:
        cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    6. You can log out the remote computer for now by typing:
      exit

If your private key ever got compromised, repeat these steps to set up a new key. And, delete the old login from authorized_keys file.

Using SSH config

For convenience, it is a good idea to save your login details in the config of ssh of your local computer. That way you don't have to type username@141.44.8.59 every time you want to login. Check if a conifg file is already present

$ ls ~/.ssh/

If it is not there yet, copy the following file to '~/.ssh/config':

config
Host           neumann
HostName       141.44.8.59
IdentityFile   ~/.ssh/id_rsa
User           UsernameOnRemote

Replace UsernameOnRemote with your username on the remote machine. Then make sure the 'IdentityFile' directs to your private key. The location ~/.ssh/id_rsa is the default. When you followed the previous section, then there is no need to change this line. If you delete this line, the remote server will persist to ask for you password (if the default option is not used). In case you already have a config file, append the content above to your own ssh config seperated by a empty line.

With this config in place, you can now log in simply with:

$ ssh neumann
Last login: Wed Sep 27 11:21:56 2017 from  mylocalcomputer

Infos/News: http://www-e.uni-magdeburg.de/urzs/t100/

This will come in handy for the file transfer tools scp, and rsync. Moreover, this step is necessary for StarCCM and possibly other tools to connect to remote computers.

File transfer

SCP

Summary
  • uses syntax of cp
  • good for few small files
  • use rsync for bigger/many files. Scp cannot continue an up/download after losing a connection.
  • scp -rC [source_files] [destination_files]

FIXME explain locations on remote and local, such as:

scp -r ~/dir1 remoteusername@remote_host:/home/remoteuser/

Rsync

Summary
  • keeps directories in sync efficiently
  • can continue up/download after connection lost
  • rsync -az [source_files] [destination_files]

Rsync is a very flexible network-enabled syncing tool between remote and local files. It uses an algorithm that minimizes the amount of data copied by only moving the portions of files that have changed. In this part, we will cover the basic usage of this powerful utility. Due to its ubiquity on Linux and Unix-like systems and its popularity as a tool for system scripts, it is included on most Linux distributions by default.

Basic Syntax

Zum Anzeigen hier klicken ⇲

Zum Verstecken hier klicken ⇱

The basic syntax of rsync is very straight forward, and operates in a way that is similar to ssh, scp, and cp. We will create two test directories and some test files with the following commands:

cd ~
mkdir dir1
mkdir dir2
touch dir1/file{1..100}

We now have a directory called dir1 with 100 empty files in it.

ls dir1
file1    file18  file27  file36  file45  file54  file63  file72  file81  file90
file10   file19  file28  file37  file46  file55  file64  file73  file82  file91
file100  file2   file29  file38  file47  file56  file65  file74  file83  file92
file11   file20  file3   file39  file48  file57  file66  file75  file84  file93
file12   file21  file30  file4   file49  file58  file67  file76  file85  file94
file13   file22  file31  file40  file5   file59  file68  file77  file86  file95
file14   file23  file32  file41  file50  file6   file69  file78  file87  file96
file15   file24  file33  file42  file51  file60  file7   file79  file88  file97
file16   file25  file34  file43  file52  file61  file70  file8   file89  file98
file17   file26  file35  file44  file53  file62  file71  file80  file9   file99

We also have an empty directory called dir2. To sync the contents of dir1 to dir2 on the same system, type:

$ rsync -r dir1/ dir2

The -r option means recursive, which is necessary for directory syncing. We could also use the -a flag instead. The -a option is a combination flag. It stands for „archive“ and syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions.

$ rsync -a dir1/ dir2

-a is more commonly used than -r and is usually what you want to use.

You may have noticed that there is a trailing slash (/) at the end of the first argument in the above commands:
$ rsync -a dir1/ dir2

This is necessary to mean „the contents of dir1“. The alternative, without the trailing slash, would place dir1, including the directory, within dir2. This would create a hierarchy that looks like: '~/dir2/dir1/[files]'. Always double-check your arguments before executing an rsync command. Rsync provides a method for doing this by passing the -n or –dry-run options. The -v flag (for verbose) is also necessary to get the appropriate output

Rsync to remote

Syncing to a remote system is trivial if you have SSH access to the remote machine and rsync installed on both sides. See above for details on SSH.

Once you have SSH access verified on between the two machines, you can sync the dir1 folder from earlier to a remote computer by using this syntax:

$ rsync -a ~/dir1 username@remote_host:destination_directory

This is called a „push“ operation because it pushes a directory from the local system to a remote system.

The opposite operation is „pull“. It is used to sync a remote directory to the local system. If the dir1 were on the remote system instead of our local system, the syntax would be:

$ rsync -a username@remote_host:/home/username/dir1 place_to_sync_on_local_machine

Like „cp“ and similar tools, the source is always the first argument, and the destination is always the second.

If you have set the SSH config then the above command simplifies. Compare this equivalent lines (when config is set as explained above):
$ rsync -a username@remote_host:/home/username/dir1 place_to_sync_on_local_machine
$ rsync -a neumann:/home/username/dir1 place_to_sync_on_local_machine

Useful Options for Rsync

Zum Anzeigen hier klicken ⇲

Zum Verstecken hier klicken ⇱

Rsync provides many options for altering the default behavior of the utility. We have already discussed some of the more necessary flags. If you are transferring files that have not already been compressed, like text files, you can reduce the network transfer by adding compression with the -z option:

$ rsync -az source destination

The -P flag is very helpful. It combines the flags –progress and –partial. The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers:

rsync -azP source destination
sending incremental file list
./
file1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=99/101)
file10
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=98/101)
file100
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=97/101)
file11
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=96/101)
. . .

If we run the command again, we will get a shorter output, because no changes have been made.

This illustrates rsync's ability to use modification times to determine if changes have been made.

rsync -azP source destination
sending incremental file list

sent 818 bytes received 12 bytes 1660.00 bytes/sec
total size is 0 speedup is 0.00

We can update the modification time on some of the files and see that rsync intelligently re-copies only the changed files:

touch dir1/file{1..10}
rsync -azP source destination
sending incremental file list
file1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=99/101)
file10
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=98/101)
file2
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=87/101)
file3
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=76/101)
. . .

In order to keep two directories truly in sync, it is necessary to delete files from the destination directory if they are removed from the source. By default, rsync does not delete anything from the destination directory. We can change this behavior with the –delete option. Before using this option, use the –dry-run option and do testing to prevent data loss:

$ rsync -a --delete source destination

If you wish to exclude certain files or directories located inside a directory you are syncing, you can do so by specifying them in a comma-separated list following the –exclude= option:

$ rsync -a --exclude=pattern_to_exclude source destination

If we have specified a pattern to exclude, we can override that exclusion for files that match a different pattern by using the –include= option.

$ rsync -a --exclude=pattern_to_exclude --include=pattern_to_include source destination

The contents of this section have been slightly adapted from Justin Ellingwood, 2013
licenced under CC BY-NC-SA 4.0

Windows

Setting up the SSH connection in a UNIX environment is relatively easy. On a Windows machine(before v10) there is no SSH program installed by default. This problem can be solved in different ways.

PuTTY

PuTTY

PuTTY is an implementation of an SSH client on windows. To use it follow this short guide:

Setting up the Windows computer

  1. Install PuTTY on your computer. PuTTY contains additional tools such as pageant.exe which holds your key pairs, and puttygen.exe which creates ssl key pairs. Don't skip them.
  2. Add the PuTTY installation directory to your system's environment variable PATH (for example C:\Program Files (x86)\PuTTY).
    This is done automatically if you install PuTTY. Jump here if you don't know how to modify the PATH variable. This step is necessary for WinSCP and other advanced tasks.
    Do not skip this step! It has been one of the main reason for questions on PuTTY in the past…
  3. Create a public/private key pair.
    1. Start PuTTYgen.exe and select the „SSH-2 RSA“ option and set the number of bits in the generated key to be 2048. Click „Generate“ and then follow the prompts.
    2. Don't set a password protection here for the key! Only do this if you know how to deal with it.
    3. Save the private key without password protection, e.g. with the name mySSHkey.ppk. Do not share the private part of your key at any time.
    4. Save the public key, too. Use the same name: mySSHkey.pub
  4. Put the key into your key ring (Pagent)
    1. Open Pagaent (the key manager of PuTTY)
    2. Pageant starts minimized in the system tray. Click on Add key in the context menu:
    3. Locate your private key and press open.
    4. You can now close the window. Check that Pagent remains in the system tray.
  5. You'll have to open pageant after restart and load the key each time!
    Optionally, do the following steps to avoid the maunal key adding at every start:
    1. Go to the directory where pagent is saved, e.g. C:\Program Files (x86)\PuTTY\
    2. Right click on pageant.exe, press Create Shortcut (ger: Verknüpfung erstellen). It might be possible, that you are asked to save the shortcut on the desktop. Confirm in that case.
    3. Now, open a new file explorer and go to the directory C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup
    4. Move the previously created shortcut to the Startup folder.
    5. Rename it back to a sane name, if necessary.
    6. Right click on the shortcut and open properties
    7. In the menu Shortcut, you'll find the field Target. It's marked by default.
    8. Add the end of the file path enclosed in quotes, add the path to your private key. For example:
      "C:\Program Files (x86)\PuTTY\pageant.exe" C:\Users\user\ssh\mySSHkey.ppk
    9. Press accept or Ok to save and close the property window.

Forward the SSH key to the remote computer

In this section we forward you public key to the remote computer, so that the remote computer can recognize your login and doesn't require you to give a pasword.

  1. Start PuTTY, for example via the desktop shortcut.
  2. In the configuration tree, on the left, make sure to select the section Session.
  3. Under connection type, select SSH.
  4. In the right half, fill in the Host Name. The host name is the IP address of the remote system or its url.
    For example, the IP address of Neumann is 141.44.8.59
  5. For convenience add your username as prefix to the IP address, seperated by an @ symbol.
    For example: user@141.44.8.59
  6. Then, go to the Auth subsubsection in the configuration tree which is part of the SSH subsection in the Connection section. \\See below:
  7. At the buttom of this page, you have to set the location of your private (and public key).
    Press the Browse button to locate the private key (.ppk)
  8. Then go to the section Connection/SSHX11 - Make sure, the checkbox Enable X11 forwarding is checked.
    - Optionally, follow here to setup an X-server(Xming) on windows - Now, return to the section
    Session (at the top of the config tree). - In the entry field just below Saved Session fill in a name for the current setup.
    For example, type Neumann. - Then press the
    Save button! This will allow you the load a prepared session without the need to reenter any information for a specific remote computer.

    Test your connection now!
    Select a saved session and press load.
    Make sure once that the settings done above are already set with the session loaded.

    - Then press Open at the buttom of the window to open the connection. - Connecting the first time to a new computer will raise a window. It asks you, wether you trust the remote computer's key. Usually it's fine to press accept. Ideally, you would cross-reference the key before. - When the username has been entered, by prefix or manually, a prompt will ask you for your password. Type your password and press enter.
    You will not see any cursor movement or masking stars when entering the password. It's UNIX default behavior. - Congrats !! You are now connected to the remote computer. The terminal now accepts commands. - For setting a passwordless connection we need to do a few further steps.
    Copy the public part of your ssh key pair (created in the previous section) to the remote server. Use WinSCP, or pscp.
    For simplicity, store the key in your home directory. It's usually the directory which is opened first. - PuTTYgen creates a key pair in a specific format. This has to be converted to an open standard. On the remote UNIX computer run: <code bash>ssh-keygen -i -f mySSHkey.pub > myopenssh_key</code> Replace mySSHkey.pub with the uploaded public key of your local computer. If you have stored the key in a different folder, give the absolute location to the key on the remote machine. - Then, add the public key (myopenssh_key) to the authorized_keys file in the ~/.ssh/ directory of the
    remote server.
    The easiest way: run the following command in the folder where the public key is stored on the remote server. <code bash>cat myopenssh_key » ~/.ssh/authorized_keys</code> It appends your public key to authorized_keys. Keep the number of authorized keys as small as possible and remove old logins. - Now refresh the authorized_keys by using the command
    <code bash>chmod 600 ~/.ssh/authorized_keys</code> - And for good measure, run <code bash>chmod 700 ~/.ssh</code> - To efficiently use remote clusters, there is an additional key pair necessary.
    On the
    remote computer run the following commands again (copy-paste them): - Create a second keypair. Press enter three times after running this command: <code bash>ssh-keygen</code> - Add the new key to authorized_keys: <code bash>cat ~/.ssh/id_rsa.pub » ~/.ssh/authorized_keys</code>

    If your private key ever got compromised, repeat these steps to set up a new key. And, delete the old login from authorized_keys file.

    Note, that SSH was set up only for a single direction of connection, from the Windows machine to the Linux machine. The other way is typically not needed and is cumbersome to set up. Avoid it, if neccessary. Go here if unix→windows is neccessary. ==== WinSCP ==== WinSCP You might find the CLI environment to transfer files inconvenient at the moment. For this purpose there are tools with graphical interface available. For example WinSCP. Advanced users might also try MobaXterm. For WinSCP, you ideally have set up PuTTY already. Especially, the PATH system variable should contain the directory of your PuTTY installation. === Quick start guide === - Start up WinSCP - In the login window * select the transfer protocol SCP * enter the server address (141.44.8.59). Remember our HPC servers are only available within the university network. Use VPN if your are connecting from outside. * enter your account details, either account and password, or account and key file. See attached images. - The first time you have to confirm to add a server to your known hosts. - If everything was entered correctly you should see two folder structures now in two columns side-by-side. - You can work in a windows-like fashion, such as drag-and-drop of files. Use context menu for advanced features. For more information, be refered to the documentation of WinSCP.
    WinSCP also has shortcuts to an own terminal and PuTTY.
    Don't use the WinSCP's own terminal!**

It's plain garbage! Use PuTTY only. When installed as described above, PuTTY can be opened from WinSCP, too (Ctrl + P).

Sources: https://winscp.net/eng/docs/getting_started

File transfer in command line

To transfer files, PuTTY brings the tool pscp which works similar to the linux scp.
pscp runs in the command prompt of windows (DOS prompt). To make pscp available in the prompt, PuTTY needs to be added in the system's PATH variable. See above.

The syntax of pscp is very similar to linux' scp

> pscp [options] [user@]host:source target

An example to copy a file from windows to a remote linux server. The command is run on the windows computer:

> pscp c:/star.sim  username@141.44.8.59:/home/username/myfolder

And the other way; requesting a file from a remote server and save it locally:

> pscp username@141.44.8.59:/home/username/myfolder/star.sim c:/

Windows 10 and Linux Subsystem

Installation Guide for Windows Subsystem for Linux (Use the Guide for For Anniversary Update and Creators Update: Install using lxrun)

The linux subsystem is an interesting feature introduced to windows 10 ( More about here). It can run an ubuntu system natively in windows, and other distributions, such as opensuse. This means you run a bash in windows with (almost) all the features of a native linux system. However, this feature is still beta.

If you installed it, you can follow the instructions for Linux. However, if you want to set up WinSCP too, you might have to setup PuTTY (and it's key pair), too.

Cygwin

Suggested only to advanced users, or users eager to learn how to use it. Don't bother googling it if you „just want to get things done“.

It is simply said a linux environment compiled in the MS windows environment. This has the advantage, you can work in a linux like environment on windows, even with windows programs. However, this comes at a cost, it's a bit crude for the beginner. You sometimes have to fix uncommon issues yourself, or compile rarely used programs yourself. This makes the tool for advanced, or enthusiastic users only. Moreover, cygwin starts to be superceeded by the Windows Subsystem, although there remain tasks which a subsystem cannot do yet in windows.

For a brief overview anyway have a look at this Cygwin introduction.

The environment installer can be downloaded here:
Cygwin

Most commands from linux are available, such as:

ssh
scp
rsync

Others might have to be installed additionally. Use the setup.exe to install openssl, and cygwin/X.

guide/connecttoneumann.txt · Zuletzt geändert: 2019/03/12 12:43 von Sebastian Engel
Nach oben
CC Attribution-Share Alike 3.0 Unported
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0