You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
342 lines
79 KiB
342 lines
79 KiB
import { addBanner, addArticle, addTitle, addHeader, addParagraph, addSubHeader } from '/scripts/article.js'; |
|
import { addInset, addInsetList, addInsetCodeListing, addInsetBulletList } from '/scripts/inset.js'; |
|
import { addImageWithCaption, addButtonGroup } from '/scripts/visuals.js'; |
|
import { addSidebar} from '/scripts/sidebar.js'; |
|
import { addSyntax } from '/scripts/code.js'; |
|
|
|
const heading = document.querySelector(".heading"); |
|
const global = document.querySelector(".global_menu"); |
|
const local = document.querySelector(".local_menu"); |
|
const sidebar = document.querySelector(".sidebar"); |
|
const main = document.querySelector(".main_content"); |
|
|
|
heading.append(addTitle("Learning SSH")); |
|
heading.append(addParagraph("Scott Simpson - LinkedIn Learning - October 2022")); |
|
heading.append(addParagraph("Chapter 3 - Practical Tasks with SSH")); |
|
|
|
main.append(addHeader("Transferring Files with SFTP")) |
|
main.append(addParagraph("The primary purpose of ssh is to connect to another machine, but it can do other things as well. For one thing, it provides a more secure alternative to ftp which once upon a time was the primary method of transferring files between computers and is still used. It is, however, notoriously insecure and definitely not recommended for anything other than trivial data where security is not a concern.")) |
|
main.append(addParagraph("A better option nowadays is SFTP (SSH File Transfer Protocol) which is similar to ftp, it use the same get/put model, but it operates on a secure ssh connection so is, of course, more secure! It transfers files over an ssh connection so it will use the ssh credentials to secure that connection. Let's see how that is done from the command line.")) |
|
main.append(addParagraph("For the purpose of this exercise I am imagining that I am creating a project on my dev machine that I will want to transfer over to my webserver at some point so I have created a projects folder and created some empty project files I can expermiment with. To work with sftp, we need to start it running with a conection which you can do with a command like the following:")) |
|
main.append(addSyntax("sftp philip@192.168.0.20")) |
|
main.append(addParagraph("You should then be prompted for a password and if you enter that successfully, you should see the sftp prompt which indicates that you are now ready to start. However, it is worth pointing out that sftp uses port 22 by default and I assume that is because you are using it via the ssh conection so it would probably be more accurate to say that it uses the same port number you use to connect to via ssh but the method for specifying the port is a little bit different. If you have already setup a host directive in the config file, you can use that for sftp as well, to do that you would type")) |
|
main.append(addSyntax("sftp webserver")) |
|
main.append(addParagraph("or whatever alias you have set up for that host so in that sense, it works in exactly the same way that an ssh command would and would use the port number you specified in config. If you need to specify the port number from the command line, it is similar to the syntax you would use with two small differences. The first is the you use an upper case P rather than a lower case p as you would for an ssh connection. Unlike ssh, the port number must come before the host name or ip address (with ssh, it can come before or after) and also you cannot use (as far as I can tell) an o option with the word Port (ie, oPort).")) |
|
main.append(addParagraph("That was a little long winded but it took me a few minutes to find that because of the fact that I don't use the standard port number for ssh so I thought it was worth mentioning. That all being said, the sftp command with, let's say port number 223344 would be")) |
|
main.append(addSyntax("sftp -P 223344 philip@192.168.0.20")) |
|
main.append(addParagraph("Here you can see both the command to initiate sftp and the sftp prompt.")) |
|
main.append(addImageWithCaption("./images/sftp1.jpg", "The SFTP prompt.")) |
|
main.append(addParagraph("A good place to start is help which displays a list of sftp commands.")) |
|
main.append(addImageWithCaption("./images/sftp2.jpg", "The SFTP help command.")) |
|
main.append(addParagraph("Remember that sftp follows the same get/put model as ftp and this is reflected in the fact that we have both get and put commands to download and upload a file respectively. Note that these commands are run on the client so, for example, download refers to downloading a file from the server to the client and upload refers to uploading a file from the client to the server.")) |
|
main.append(addParagraph("We can also change directory on either the client or the server so we have some control over where the file that we download (or upload) will be stored. I want to save my project files in the projects directory I had created for this purpose on the webserver so I will navigate to it with the command")) |
|
main.append(addSyntax("cd /home/philip/Learning/projects")) |
|
main.append(addParagraph("Next, I want to verify that the remote directory is empty with ls and that the local directory contains the files I want to copy with lls. I can then copy the files using the put command. Note that I have already connected sftp to the server so I don't need to supply the servers name or IP address. I can just use put with the filename and I will start by uploading pf1.css with the command")) |
|
main.append(addSyntax("put pf1.css")) |
|
main.append("This shows a confirmation that the file is being uploaded. In this example, it's an empty file so it doesn't take long. I'm not sure if I can use wildcards here so next I will try to upload all of the files with") |
|
main.append(addSyntax("put *")) |
|
main.append(addParagraph("I see a similar result, this time listing all 5 files and confirming the uploads. When it's done I can use ls again to confirm that all of the files have been uploaded and again, I will run lls to compare this to the files in the local directory to confirm they have all been uploaded. Of course, you can also connect to the machine in a separate terminal, in my case a separate kitty session and navigate to the directory there to confirm that the files have been successfully uploaded. All of this is shown in the image below.")) |
|
main.append(addImageWithCaption("./images/sftp3.jpg", "Transferring files from the client to the server using SFTP.")) |
|
main.append(addParagraph("This is pretty straightforward, but copying single files or even a batch of files using a wildcard is pretty limited. For me, personally, I would like to be able to upload or download an entire directory including its contents and subdirectories etc. That doesn't seem to be covered in the course, at least not here, so before we move on to a GUI equivalent of sftp, I looked this up to find out if it is possible and it is, as I discovered from an main on TecMint dated February 25 2017. The main is entitled <a href='https://www.tecmint.com/sftp-upload-download-directory-in-linux/'>How to Upload Files/Directories Using sFTP in Linux</a>, thanks to Aaron Kili for providing that.")) |
|
main.append(addParagraph("In order to try this out, I've deleted the temp files used in the previous test and created something that looks more like a real-life project although the files are just dummy files so they are all empty, but I have a project folder for an imaginary blog project which contain a file and some folders and those folders contain files or subfolders and so on. The structure is shown in the image here.")) |
|
main.append(addImageWithCaption("./images/tree1.jpg", "The directory structure and files for my imaginary blog project.")) |
|
main.append(addParagraph("We should be able to upload the whole project with the put command just as we did previously, but we will also add the -r option to upload recursively (in other words, to copy directories and their contents as well) so the command would be")) |
|
main.append(addSyntax("put -r blog")) |
|
main.append(addParagraph("If the command is unsuccessful or if it doesn't do what you expect, that should be fairly obvious because you will wee a confirmation of the files that are being uploaded as shown here.")) |
|
main.append(addImageWithCaption("./images/sftp4.jpg", "Confirmation of the files uploaded from the blog folder.")) |
|
main.append(addParagraph("Again, I can check that this has worked as expected by running tree on the web server to show that the structure of the blog project has been completely transferred with all its files.")) |
|
main.append(addImageWithCaption("./images/tree2.jpg", "The directory structure and files for my imaginary blog project on the web server.")) |
|
main.append(addParagraph("So that's how we transfer files from the local machine (client) to the remote machine (server) using sftp. It is fairly straightforward to use and would probably allow me to accomplish the main file transfer task that I have which is copying my websites files which I write on my Windows PC over to the web server which is on a Raspberry Pi. You may be aware of a utility called scp which has largely been replaced by sftp so if you are working at the command line, you would be more likely to use sftp if it is available. If you want to use a graphical interface, which can be a lot easier if you are moving a lot of files, WinSCP is basically a front end for scp and can be really convenient to use. At the moment, I use it for transferring files to my website primarily because it is so easy to use and it allows me to sync the files on my local machine with the server. Let's take a look at that now.")) |
|
main.append(addParagraph("When you start WinSCP, you are presented with something that looks a little like File Explorer in Windows which is what it is.")) |
|
main.append(addImageWithCaption("./images/winscp1.jpg", "The WinSCP interface.")) |
|
main.append(addParagraph("THe files on the local system are shown on the left but the right pane is blank because I am not connected to the server at the moment. To make the conection, I will click on the New Session button near the top left corner which brings up the login window.")) |
|
main.append(addImageWithCaption("./images/winscp2.jpg", "The WinSCP login window.")) |
|
main.append(addParagraph("This is pre-populated with the login details for my server. Notice that WinSCP, in spite of the name, does support the newer sftp protocol but you can select others, includig scp and ftp. I am connecting with the ip address and the port number and user name are pre-populated because this was the last connection I made. If it's not clear, I don't actually use 223344 as a port number, this is effectively hiding the actual number I use. In fact, 223344 is not a port number (which would be in the range from 0 to 65535) so if you are following along, don't use that number! I could also insert the password here but I usually just click login and type the password at the prompt which is more secure.")) |
|
main.append(addParagraph("I haven't set up an ssh key for this connection yet so I will do that now. Actually, I made a bit of a mistake which should server as a warning to be careful using sftp. In this instance, WinSCP would have prevented the error but I copied my public key over to the .ssh folder on the web server, but it had the default key name and it overwrote the public key for the user, philip, on the webserver! I have started again with a more organised approach which will help me to avoid mistakes like that. I have created key pairs for three users, developer and philip on the dev machine and philip on the web server. For the key name I have used the username followed by an underscore and then dev on the dev machine and web on the webserver. For example, the philip account on the web server uses the identity philip_web and the philip account on the dev machine uses the identity philip_dev and for any other users that I might want to grant ssh access to I will use the same naming convention.")) |
|
main.append(addParagraph("This is significant because I have previously just used the default name for the keys and that changes how you connect which I was forgetting and it gave me a few headaches so I am going to start again from scratch by deleting everything in the .ssh folder and regenerating the keys etc. I will go through this step by step on the web server. So the account name is philip and I want to be able to ssh to the dev machine, connecting via the philip account.")) |
|
main.append(addParagraph("I'm on the web server in the .ssh folder (/home/philip/.ssh) so we will start by generating the key pair which we will do with the command")) |
|
main.append(addSyntax("ssh-keygen")) |
|
main.append(addParagraph("When prompted for the file where I want to save the key, I responded with")) |
|
main.append(addSyntax("/home/philip/.ssh/philip_web")) |
|
main.append(addParagraph("The next step is to transfer the public key over to the dev machine using the ssh-copy-id command which I have used previously but without an identity so that will be a little different. However, I will try without the identitiy first to see what happens if it is omitted. I don't know if you would be likely to have different keys in .ssh but bear in mind that if you also have a key with the default id, the command would work but would not pass over the key that you intended to send!")) |
|
main.append(addSyntax("ssh-copy-id -p 223344 philip_web philip@192.168.0.10")) |
|
main.append(addParagraph("I was starting with an empty .ssh folder so I don't have any other keys so this command will fail with the error message")) |
|
main.append(addSyntax("/usr/bin/ssh-copy-id: ERROR: No identities found")) |
|
main.append(addParagraph("Essentially, this error message means that the ssh-copy-id command has looked for an identity and didn't find it because it is looking for the default filename. You would also see this error if make a mistake when specifying the identity so you might have the path or the filename wrong or the key may not have the filename you think it has. If you do see the error, check the name of the key in the file, whether it has the default value etc. If you try running the command again, make sure you know the name of the key you want to install on the server. If it is the default, make sure you don't specify the identity or that it is specified correctly. If you have a different filename, just make sure that it is specified correctly - you can use the tolde to specify the home directory.")) |
|
main.append(addSyntax("ssh-copy-id -p 223344 -i /home/philip/.ssh/philip_web philip@192.168.0.10")) |
|
main.append(addParagraph("The output from this is shown below.")) |
|
main.append(addImageWithCaption("./images/ssh-copy-id1.jpg", "The output we get when installing the public key on the server.")) |
|
main.append(addParagraph("We can see both the output from the original command with the identity missing as well as the command where it was specified and the identity was correctly specified. So we saw the 'No identities found' error and they we and it then gives you the response to the command with the identity and it outputs the figngerpribnt of the server which will be added to the known hosts. It tries to log you in with the key and fails. Why does it do that?")) |
|
main.append(addParagraph("It does output a message saying that this is to filter out any that are already installed. Essentially, if the log in is successful, this means that the key is already installed. If it isn't, the login will fail because you haven't yet installed the key and you are then prompted to prove that you have a valid login - you are asked for the password for the philip account on the server and when you type that in correctly, the process of installing the key is completed and you can then log in with the command that it provides for you so that will let you know if everything is correct!")) |
|
main.append(addParagraph("We can now login with the command")) |
|
main.append(addSyntax("ssh -p 223344 philip@192.168.0.10")) |
|
main.append(addParagraph("Remember that when we do sign in, we are passing over our identity (in effect) and the server will try every key it has installed, if necessary, to work out which public key works so that doesn't have to be specified in the ssh command so this gets us onto the server.")) |
|
main.append(addParagraph("That's good enough to allow us to ssh without a password but we can make life a little easier by creating a nickname for the server by adding it's details to the config file in our client .ssh folder. I will start that with the command")) |
|
main.append(addSyntax("vim config")) |
|
main.append(addParagraph("We do need to specify the identity here so the config file will look something like this.")) |
|
main.append(addImageWithCaption("./images/config1.jpg", "The output we get when installing the public key on the server.")) |
|
main.append(addParagraph("I gave the server(my dev machine) the nickname philip_dev because I will want to be able to connect to different accounts. I can add another host with the name developer_dev with the same details with the exception of the value for User which I will change to developer, but of course I need to also install the key in the .shh file for that user. So I will copy the key with")) |
|
main.append(addSyntax("ssh-copy-id -p 223344 -i ~/.ssh/philip_web developer@192.168.0.10")) |
|
main.append(addParagraph("and I will copy the details for the philip_dev host, change the nickname to developer_dev and the username to developer and save. I can then log in to the dev machine as developer with")) |
|
main.append(addSyntax("ssh developer_dev")) |
|
main.append(addParagraph("I can repeat the process on the dev machine to allow me to ssh from there to the web server from either account, although in both cases, I will be logging in as philip on the web server. This has been a bit of a digression but I do think that it is useful for a couple of reasons. First of all is the fact that it acts as a really useful recap because although there is a lot more to ssh than just allowing you to connect to a server without a password, it tends to be central to everything you do with ssh because you are always going to need to connecto to the remote machine.")) |
|
main.append(addParagraph("I think that the process of going over this has also illustrated a number of useful points about troubleshooting in general. It demonstrates the fact that on order to get from a point where you are signing in with a password to where you are signing in with an ssh key only can sometimes be quite tricky, there are a number of errors you might encounter so it is important to work in a way that is logical and that you are able to troubleshoot any errors you might have. That means, for example, making sure that you have the proper credentials to allow you to sign in and then you can start setting up your ssh key. You also want to make sure that you can sign in with the ssh key before you think about setting the config to allow you to connect with just the nickname you set up for the server in your config file. Of course, you also want to make sure that you are able to connect to the server without a password before you consider disabling password authentication in the sshd_config file.")) |
|
main.append(addParagraph("It is also important, of course, to test the connection at every step to make sure that the security measures you are applying are applied in the way that you think that they are! At the same time, it is important to remember that you may very well want to make sure that we can also connect via any other tools we might be using including kitty (or putty) ad WinSCP.")) |
|
main.append(addParagraph("I have used kitty to generate a key pair on my desktop machine. Copying the public key to the servers can be a bit tricky but WinSCP helps with that so I will describe the process there first. Let's start with the private key. I already have a session saved for my connection to the web server because I use it to transfer files to the server on a daily basis so I will start by loading that up, clicking Edit and then clicking on the Advaced button.")) |
|
main.append(addImageWithCaption("./images/winscp3.jpg", "The WinSCP Advanced settings.")) |
|
main.append(addParagraph("You can generate your key pair here by clicking on the Tools menu and selecting Generate New Key Pair with PuTTYgen and you can save these in your local .ssh folder. You can then point the Private key at the saved key. There are a couple of options for helping you to install the public key on the server and as before, your choices here may be determined by what sort of access you have to the server. For my web server, I have clicked on the Display Public Key button and then the Copy key button. I then signed into the web server via Kitty and pasted it into the Authorizes keys file in the .ssh folder. I can then connect by clicking the New Session tab and double clicking on the Web Server connection and I am connected to the web server without having to type the password.")) |
|
main.append(addParagraph("A nice little bonus here is that I only need to open up Kitty, load the web server connection and expanding the SSH options in the left panel, clicking on Auth and then using the Browse button to locate the same private key and I can then connect to the web server via Kitty without a password. I don't need to worry about the public key because I am using the same key pair that WinSCP uses so the public key is already on the server.")) |
|
main.append(addParagraph("For the dev server, I want to setup the connection on WinSCP to allow me to login as developer with they key so I have browsed for the private key as before, but I have then selected the option to copy the key to the server which initiates a login and prompts me for the password. When I provide that, they key is installed and I get a box popping up to confirm that.")) |
|
main.append(addImageWithCaption("./images/winscp4.jpg", "Confirmation that WinSCP has copied the public key over to the server.")) |
|
main.append(addParagraph("I can then click on OK and OK again which takes me back to the edit page for that connection so I can save it and again, I can just point the logins for the dev machine on Kitty to point to that same private key and I can then use it to log in.")) |
|
main.append(addParagraph("As I mentioned, testing your connections is critical because you don't want to switch off password authentication on the server and then find that you are unable to connect, especially via Kitty! I have made sure that I can connect via Kitty to both the web and dev machines and although it's not as critical, I did test this for both the developer and philip accounts on the dec machine. I do have other accounts on at least one of the machines but as long as you can connect via at least one user account, that allows you to at least connect to set up logins for other users which is really straightforward once you have set up one account. Also, don't forget that as perhaps a last resort, you can always switch password authentication back on as long as you have one account you can log in with.")) |
|
main.append(addParagraph("So do make sure that you always have one account that you can connect with to allow you to troubleshoot if needed. Having ensured that I can ssh between servers and that I can connect to the servers via either kitty or WinSCP using my ssh key, I can now switch off password authentication on both servers. Before I do that, I have backed up the sshd_config file before making any changes (it's always a good idea to do that so you can always recover to a working version of the file if you need to. I then edited the live config file and enabled the PasswordAuthentication and set it to no.)")) |
|
main.append(addParagraph("The easiest way to test this is by opening a Kitty session and trying to login with the IP address and the port number (so we are ignoring the saved logins. You can put the account name in front of the IP address or provide it when prompted and also provide the password when prompted. In theory, you shouln't be able to log in but you might if you have forgotten to restart the sshd service, in which case, the update to the sshd_config file won't have taken effect so if you are able to login (as I was!!!), try restarting the sshd service with)")) |
|
main.append(addSyntax("sudo systemctl restart sshd")) |
|
main.append(addParagraph("And then try logging in again. If you can still login, you should check your config files to see if there is a setting overriding the sshd_config file or a setting earlier in the config file which invalidates the setting you used. Of course, you also want to ensure that the PasswordAuthenticate directive has been set to no.")) |
|
main.append(addParagraph("If you have successfully disabled password authentication, you should see this error.")) |
|
main.append(addImageWithCaption("./images/winscp5.jpg", "The error message you should see if you are logging in to a server with a password in order to confirm that the PasswordAuthoenticate directive has been set correctly (and that you remembered to restart the sshd service).")) |
|
main.append(addParagraph("So that's a brief overview of sftp. To recap, it is a secure replacment for ftp and it is used in a variety of applications in order to allow files to be trasferred to and form a server. We saw how it was possible to do that from the command line or using a GIU such as WinSCP, but there are lots of other applications that also use it. Many IDEs do and VS Code is an example of that. How you would set that up can vary, which is something we have seen here, so I won't go into any details for specific apps, but be aware that it is possible. On a personal note, it might be handy for me if I could do that with Dreamwwaver which might allow me to sync files with my wev server without having to use ftp which kind of underlines the fact that although I do use Drewamweaver every day, the truth is that I have never really learned to use it properly so it might be worth going through one of the DW courses on LinkedIn (or perhaps just reading the expensive Dreamweaver Clasrtoom in a Book that I bought in February 2021).")) |
|
main.append(addParagraph("That leads me to close off this section with a couple of thoughts. Thought number one. The sftp protocol is extremely useful and versatile so learning how it works in general or in terms of a specific application can be extremely beneficial. Thought number two is that the tools we use can be quite powerful but sometimes we use them on a quite superficial level without knowing all of the things that they can do so learning those tools can also be hugely beneficial!")) |
|
main.append(addHeader("Transferring Files with SCP")) |
|
main.append(addParagraph("SCP is short for secure copy and it is basically the Linux command, cp, but it copies files between systems. Because of that, it is a convenient command for use in scripts or for a quick command to copy a single file and it can also be used in a reverse tunnel to allow you to copy files back from the remote to the local server.")) |
|
main.append(addParagraph("To copy a file over to the remote server, you use the scp command followed by the filename and then the destination (username and hostname and a :). Again, I had problems with this because of the port number, although I was able to use the server's nickname so the standard format would be something like")) |
|
main.append(addSyntax("scp file1 philip@192.168.0.10:")) |
|
main.append(addParagraph("Using the server's nickname, that would be")) |
|
main.append(addSyntax("scp file1 philip_web:")) |
|
main.append(addParagraph("To copy a file from the server to the client, we switch these around a bit and add a filename to copy the file to so")) |
|
main.append(addSyntax("scp philip@192.168.0.20:file1 file3")) |
|
main.append(addParagraph("or again, using the nickname")) |
|
main.append(addSyntax("scp developer_dev:file1 file2")) |
|
main.append(addImageWithCaption("./images/scp1.jpg", "Example of the SCP command and the directory listing to confirm the file was downloaded.")) |
|
main.append(addParagraph("You may have noticed that we added a colon (:) at the end of the remote host and this is important because it allows us to specify the path of the file we want to save to or copy from on the remote machine. In these examples, we used the : on its own which means that if we pass a file over to the remote system, the file will go into the home directory of the specified user and if we download a file, it will be downloaded from the user's home direstory. If we take that last example, we were connecting to the server with the user credentials for developer on the dev machine, so we downloaded the file from that user's home directory.")) |
|
main.append(addParagraph("We specified the filename only for the last parameter and so the file was saved in our home directory. It was actually saved to the directory we were in when we executed that scp command which happened to be our home directory. Let's say that we wanted to download it to our Downloads folder instead, there are two ways we could do that. We could specify that directory name in the command, so")) |
|
main.append(addParagraph("scp developer_dev:file4 ~/Downloads/file4")) |
|
main.append(addParagraph("which will download file4 and place it in the downloads folder. Alternatively, we can specify the filename on its own if we navigate to the Downloads folder first so if I")) |
|
main.append(addSyntax("cd ~/Downloads")) |
|
main.append(addParagraph("and then run the command as")) |
|
main.append(addParagraph("scp developer_dev:file4 file4")) |
|
main.append(addParagraph("again, that will download the file to that Downloads folder.")) |
|
main.append(addParagraph("If we want to download a file from a directory other than the home directory on the remote system, I can specify the path after the colon. For example, let's say that I want to copy a file from the Downloads folder in the developer home sirectory on the dev machine. I can do that with")) |
|
main.append(addSyntax("scp devloper_dev:Downloads/file5 file5")) |
|
main.append(addParagraph("Here, I am using a relative path starting from the home directory and the file is being downloaded to the current directory because I didn't specify a path for the file location. If I had wanted to, I could use a path for the location to save the file as well.")) |
|
main.append(addParagraph("Now, consider this example. Let's say that I want to copy a file that isn't in the user's home directory. As an example, let's assume that I want to download bootstrap.log from the /var/log directory on the dev machine, if that is even possible! Let's assume that it is and run the command")) |
|
main.append(addSyntax("scp developer_dev:/var/log/bootstrap.log bootstrap.log")) |
|
main.append(addParagraph("I expected that to fail because I don't have permissions for that file as developer on the dev machine, although I do have read only access so let's try again with boot.log which I have no permissions for.")) |
|
main.append(addSyntax("scp developer_dev:/var/log/boot.log boot.log")) |
|
main.append(addParagraph("I do get a permissions error this time. I suspect that running the command as sudo wouldn't give us the appropriate permissions because that would only give root access to the user on the client system, but in any case, it fails with the nickname because that was set up for the user philip and so is not accessible to root. From the dev system, I have tried to upload that file to the web server with")) |
|
main.append(addSyntax("scp philip_web:/var/log/boot.log boot.log")) |
|
main.append(addParagraph("and again I get a permissions error. I would expect to get an error if I run this as sudo for the reason mentioned above, but probably will not see a permissions error. The error I do get is")) |
|
main.append(addSyntax("ssh: Could not resolve hostname philip_web: Name or service not known")) |
|
main.append(addParagraph("This has illustrated two important points about scp. Notice that we specified the path as an absolute path - /var/log/ - so that didn't look in the home directory and this allows us to download a file from anywhere on the remote system. However, we do need to have permissions to access the file in order to copy it and read is sufficient for that.")) |
|
main.append(addParagraph("You can, in theory, omit the colon but if you do that, the string you used to specify the host will be treated as a filename and can give you unexpected results. For example, let's say that I want to copy a filename called rename from my dev machine to the web server and that to do that, I am logged in as developer on the dev machine. The correct way to do that is with the command")) |
|
main.append(addSyntax("scp rename philip_web:")) |
|
main.append(addParagraph("which will upload to the home directory for philip on the web server. If I try that command again but without the colon, it does seem to work, but nothing happens. You might say that without the colon, the results are undefined so we might have nothing happen, as it did here, or we might see some unexpected result such as the file being renamed as philip_web rather than being copied over to the web server.")) |
|
main.append(addParagraph("As we said before, the scp command is probably not the best way to transfer files, particularly if you are copying large numbers of files. It does more or less the same things as sftp or even a tool like WinSCP, but it does it in a slightly different way and so the tool you choose for a particular task will be detmerined by what the task is and which tool is better suited to it. In some cases, for example a one-off file copy, scp can be the best tool to choose so it is useful to know about it and to be able to use it.")) |
|
main.append(addHeader("Multi-Step SSH Connections")) |
|
main.append(addParagraph("Most of the time, when you connect to an ssh server, this is a direct connection from the client machine to the server. In some cases, however, an origanisation might want to control access to machines in the network by using a server that acts as a gateway.")) |
|
main.append(addParagraph("This is known as a jump host or a bastion host and is often used in large organisations because it allows you to employ heavy security on one machine which users will have to connect to before connecting to other machines. It also means, of course, that you don't need the other machines to be as secure. It can be on the same network as the machine that you want to connect to, but it doesn't have to be.")) |
|
main.append(addParagraph("Let's say that we are on a client machine, we want to connect to a server but we don't have access to it directly, we have to connect via an intermediary.")) |
|
main.append(addParagraph("There two ways to do that and the simplest is just the same way in which we have been connected so far, just repeated.")) |
|
main.append(addParagraph("We would do that in two steps. First, we connect to the intermdeiary via ssh. We may not have access to do very much on that intermediary machine, but from there, we can ssh to the server.")) |
|
main.append(addParagraph("Alternatively, we can connect from the client to the server in one step by sing the -J option which allows us to specify one or more intermediate hosts so the syntax for that ould be:")) |
|
main.append(addParagraph("ssh intermediary1,intermdiary2 server")) |
|
main.append(addParagraph("For example, let's say that I want to connect from my web server to my dev machine via a copuple of other hosts, the command might look like this:")) |
|
main.append(addSyntax("ssh -J philip@192.168.0.5,philip@192.168.0.15 philip@192.168.0.20")) |
|
main.append(addParagraph("Notice that there is a clear distinction between the intermediary hosts and the taget server. The intermdiary hosts are separated by a comma and there are no spaces between them. The target server is separated from the list of intermediaries by a space. Depending on how the hosts are configured for authentication, we might have to authenticae at each step so, for example, if the hosts allow password authentication, we might be prompted for a password at each step.")) |
|
main.append(addParagraph("One advatage of this approach is that we only need to store our private key on our own server, it doesn't need to be stored on each intermediate. Consider this, if we want to connect to each separately, we would do it like this. First, we would ssh to 192.168.0.5 with the command")) |
|
main.append(addSyntax("ssh philip@192.168.0.5")) |
|
main.append(addSyntax("If we are connecting via an ssh key, we are using the private key on our own server and we connect using the public key stored in the first intermediary. For the second step, we will use a similar command.")) |
|
main.append(addSyntax("ssh philip@192.168.0.15")) |
|
main.append(addParagraph("But this won't work if our private key is not also stored on this server or we have another private key saved there, which would mean a different public key stored on the second intermediary. For the last step")) |
|
main.append(addSyntax("ssh philip@192.168.0.20")) |
|
main.append(addParagraph("we again need to have a private key on the intermediary and a public key on the target server. Essentially, each connection will need a key pair so we might use the same key pair for every connection and we might not, but we do have to have a private key stored on them. If we use the -J option, we would need our private key stored on the original server and a public key stored on each of the intermediaries and the target server. In this case, we are using the same key pair in each step, but the private key is being passed along so we only need to store it on the web server and we can then install the public key on the machines we will connect to along the way including the dev machine - the target server.")) |
|
main.append(addParagraph("Whichever method we use, the overall result will be the same. We will be connected to the target server as though we had connected to it directly via ssh. We will be able to access whatever resoources on the machine we have permissions for but we will not have any access to the intermediary hosts but it is wirth remembering that a host can act as both an intermediate and a target machine. For instance, one of the hosts in my example might be a media server so I may be able to connect to it, do some work on the machine and then ssh over to the dev machine and do some work there. However, once I have connected to the dev machine, that is the only machine I have access to.")) |
|
main.append(addParagraph("We can set up a connection via one or more intermediates in our config file within the .ssh folder which you might want to do if you have a complicate route to follow. To give you an example of this. If I sign in to my dev machine with the username, philip, this is config file in the .ssh directory for that account.")) |
|
main.append(addImageWithCaption("./images/proxy1.jpg", "The config file in the .ssh folder for user philip on my dev machine.")) |
|
main.append(addParagraph("Let's say that I want to setup another connection to a machine I don't have access to, but that the web machine shown in this config file does. I can set up a connection to it using the ProxyJump directive so the config file would look like this,")) |
|
main.append(addImageWithCaption("./images/proxy2.jpg", "The config file set up to allow me to connect to another machine via the web server.")) |
|
main.append(addParagraph("I can then connect to server 2 via the command")) |
|
main.append(addParagraph("ssh server2")) |
|
main.append(addParagraph("Again, I would have to authenticate on the web server (with a password if that is how it accepts connections but I am using an ssh key) and would then have to authenticate on server 2 using an ssh key (if that is set up) or a password pr whatever form of authentication that is required to connect to server 2.")) |
|
main.append(addParagraph("For the sort of setup I have at home, it's unlikely that I would want to use a bastion host but if I had more servers, I could set that up if I wanted to. Ishave encountered them in a work situation where I might be required to connect to a couple of different hosts set up for different purposes. In my experience, it is more common to be able to connect to whichever server I need to access but if a client hosts their own servers, it is probably more common to only be able to connect directly to one host and I can then connect from that to other hosts on the client's network as required.")) |
|
main.append(addHeader("Port Forwarding with SSH")) |
|
main.append(addParagraph("One of the faclilities offered by OpenSSH is port forwarding which allows us to send different types of data such as web pages from one system to another. A typical use case would be in a scenario where you are hosting a web server on a home network. The web server typically will have a p private IP address but it is made available to the public via your public IP address. In my case, the router, which handles all incoming data is set up to forward any requests on port 80 to the web server so this works as if the public IP address was the address of the system. If I open a web browser and type my public IP address, I get the web page that is being server up by the web server.")) |
|
main.append(addParagraph("Just a reminder, there are a number of ports which are reserved as the default values for certain services such as port 22 for SSH (as we have seen earlier), ports 80 for HTTP or port 443 for HTTPS. Any service that runs over the web will use a port in order to communicate over the network.")) |
|
main.append(addParagraph("We can configure port forwarding to have data sent from our local system to the remote system or from the renote system to our local system so essentially we can set up a port on one system to accept data sent to the other and this data is sent securely using ssh. There are 3 ways to do this.")) |
|
main.append(addSyntax("ssh -L [bind_addr:]port:host:port")) |
|
main.append(addParagraph("This is local port forwarding so this would send data that comes into the specified port to a port on the remote system.")) |
|
main.append(addSyntax("ssh -R [bind_addr:]port:host:port")) |
|
main.append(addParagraph("This is remote port forwarding so the remote system would send any data that it receives on the specified port to our local system.")) |
|
main.append(addSyntax("ssh -L [bind_addr:]port:host:port")) |
|
main.append(addParagraph("Dynamic port forward is a little bit more complicated but we are going to look at each in turn so I will leave that for now. One interesting point is that the IP address is optional and this is the IP address of the local machine. We might have to specify an IP address if we have more than network interface, this option tells the system which IP address we want to bind the tunnel to. If it is omitted, ssh will be able to use any available address. We might also specify an IP address for the remote host but we can also use an alias or some other form of hostname.")) |
|
main.append(addHeader("Local Port Forwarding")) |
|
main.append(addParagraph("With local port forwarding, we are specifying a port on our local host that sends traffic to a port on the remote system and this allows us to use some service on the remote system as if we were using the server that actually provides that service. An example of that might be where we need to access a database on server that is running on a private network. Let's say we have a host called server1 with a database running on port 3306 and we want to make this available on our local server via port 3333. We could set that up with the command:")) |
|
main.append(addSyntax("ssh -L 3333:localhost:3306 user@server1")) |
|
main.append(addParagraph("We can then use port 3333 to access the service although I experimented with this on my hoem network where I tried to access mysql which I have running on my webserver on my dev machine (which doesn't have mysql). I first ran the command")) |
|
main.append(addSyntax("ssh -L 3333:localhost:3306 philip_web")) |
|
main.append(addParagraph("but this just had the effect of ssh'ing to the web server. I also tried it in a format more like that in the course video")) |
|
main.append(addSyntax("ssh -L 3333:localhost:3306 philip@192.168.0.20")) |
|
main.append(addParagraph("but this time, I got an error due to the fact that it was trying to connect me via the default ssh port. It's possible that this works differently on a Raspberry Pi but it looks as though the -L option is being ignored and I am simply being connected via ssh. However, I checked the man page for ssh and the -L option is shown there with the following entry.")) |
|
main.append(addImageWithCaption("./images/proxy3.jpg", "The -L option in the man page for ssh.")) |
|
main.append(addParagraph("Connecting to a database isn't really the topic of this course but it would be an interesting way to demonstrate that it works so hopefully this will become clearer before the end of the course but if not, it will become a topic to investigate later.")) |
|
main.append(addParagraph("In any case, the effect of this should be that we can access the database as if it were running on our local machine, in my case, the dev machine. Note that the localhost in the command refers to the remote server and as I mentioned earlier, we can bind this to a specific network interface if there were more than one available.")) |
|
main.append(addParagraph("In our example, we were forwarding a port from our dev machine which acted as the ssh server and is forwarding the service, so to speak, on to the web machine which suggests that I was running this command the wrong way round and should actually have run it from the web machine but when I tried that, the effect was the same!")) |
|
main.append(addParagraph("It might be helpful to work through the course example and see what is happening there so we will start with the command")) |
|
main.append(addSyntax("ssh -L 8080:localhost:80 scott@10.0.1.110")) |
|
main.append(addParagraph("The interesting part here is probably")) |
|
main.append(addSyntax("8080:localhost:80")) |
|
main.append(addParagraph("and understanding this is key to understanding the concept of port forwarding so let's break it down into its three parts.")) |
|
main.append(addSyntax("8080")) |
|
main.append(addParagraph("This is the port I want to open up on my local machine.")) |
|
main.append(addSyntax("localhost")) |
|
main.append(addParagraph("This is from the perspective of the server so essentially it is localhost on the server.")) |
|
main.append(addSyntax("80")) |
|
main.append(addParagraph("This is the port we want to open on the server and remember, the server is a web server so it means that we want traffic going to the web server to be forward to port 8080 on the local machine.")) |
|
main.append(addParagraph("Now, if you go yo the local machine, you should be able to type localhost:8080 in a browser URL and the website being server up by the browser should be available there. Following that logic, I should be able to make my website available on my dev machine in the same way so to do that, I will log into the dev machine and run the command")) |
|
main.append(addSyntax("ssh -L 8080:localhost:80 philip_web")) |
|
main.append(addParagraph("The result of doing that is shown in the image below.")) |
|
main.append(addImageWithCaption("./images/local_port_forwarding1.jpg", "RDP to my dev machine and viewing the default page from my web server at localhost:8080")) |
|
main.append(addParagraph("That seems to work perfectly well which demonstrates the fact that web traffic is being forwarded from port 80 on my web server to port 8080 on my dev machine. I guess that what's happening is that when I go to localhost 8080, since we have the port forwarding in place, the browser is going to port 80 on the web server to get the page.")) |
|
main.append(addParagraph("To me, this sounds like remote port forwarding since traffic is being forwarded by th web server to the local machine but this is probably because of a misunderstanding of what was meant be local port forwarding. Based on this, my interpretation of the command would be that we are, from the dev machine, asking the web server to forward the traffic on so we are actually setting up local port forwarding up on the web server. In other words, the command isn't to allow us to forward traffic onto another machine, it is to allow another machine to forward traffic on to us. I guess that if we wanted to configure the remote machine to accept traffic from our local machine, this would be remote port forwarding which we will see in the next section.")) |
|
main.append(addParagraph("In essence, what we are doing with local port forwarding is to open up an ssh connection to the machine whilest at the same time, connecting the port (in this example 80) on the machine we ssh'ed to over to port 8080 on the originating machine so that is probably why we consider that to be local port forwarding. The port forwarding will remain active as long as that ssh connection remains open so if we exit to go back to the original machine, traffic will no longer be forwarded and we will no longer be able to view the web page from the web server using local host.")) |
|
main.append(addParagraph("Remember that when we ssh'ed from one machine to another, we were able to do that either directly or via an intermediary host, and with port forwarding, we can use a similar method. Before I do that, I want to be able to check that it works correctly. I have added a third Raspberry Pi to my active network, it doesn't have any specific purpose at the moment so I just refer to it as my Pi3 since it is the only Raspberry Pi 3 I currently have active. I will start by ssh'ing over to the Pi3 and using the command")) |
|
main.append(addSyntax("ssh -L 8080:localhost:80 philip_web")) |
|
main.append(addParagraph("to set up port forwarding from port 80 on the web server to port 8080 on the Pi3, just as I did previously on the dev machine and the result is the same. When I type in localhoust:80 in a browser, I get the webpage in a browser on my Pi3. I'm now going to amend the command so that it routes the web traffic through the development machine so the command becomes")) |
|
main.append(addSyntax("ssh -L 8080:localhost:80 philip_web")) |
|
main.append(addParagraph("Again, I will confirm this worked by showing a screenshot of the desktop on the Pi3 with the browser pointing to localhost:80.")) |
|
main.append(addImageWithCaption("./images/local_port_forwarding2.jpg", "RDP to my Pi3 and viewing the default page from my web server at localhost:8080 with the web traffic being routed via the dev machine.")) |
|
main.append(addHeader("Remote Port Forwarding ")) |
|
main.append(addParagraph("As we mentioned earlier, remote port forwarding is essentially the same as local port forwarding but in reverse and it can be useful when we have already established an ssh connection to allow, for example, copying files. This is also sometimes referred to as a reverse tunnel since it creates a tunnel from the remote system to the local system. Let's look at an example of that.")) |
|
main.append(addSyntax("ssh -R 5432:localhost:22 user@server1")) |
|
main.append(addParagraph("This has the effect of opening up port 5432 on the remote system and forwarding it to port 22 on the local system. To experiment with this, I am going to try forwarding web traffic from my web server to my dev server. Just as a reminder, I previously did that by going to the dev server and running the command")) |
|
main.append(addSyntax("ssh -L 8080:localhost:80 philip_web")) |
|
main.append(addParagraph("The effect of this was that I had connected to the remote server, the web server, which therefore became the local server and this began to forward traffic from the web server to the remote server (which is mow the dev machine) and so in a remote desktop connected to the dev machine, I was able to access the web site server up by my web server by going to the URL")) |
|
main.append(addSyntax("localost:8080")) |
|
main.append(addParagraph("The goal here is to forward traffic from the remote server to the local machine, so we want to be ssh'ed into the local machine and then have the web traffic follow - that is, the web traffic should also be forwarded from the web server to the local machine so this time, we will start off on the web server and run the command")) |
|
main.append(addSyntax("ssh -R 8080:localhost:80 philip_dev")) |
|
main.append(addParagraph("To confirm whether that worked or not, here is the browser on my dev machine pointing to localhost:8080.")) |
|
main.append(addImageWithCaption("./images/remote_port_forwarding1.jpg", "RDP to my dev machine and viewing the default page from my web server at localhost:8080")) |
|
main.append(addParagraph("So that did work. The only point I find a little confusing is that according to the course video, the ssh command should forward traffic on to port 80 from port 8080 on the local machine and if that were the case, I should be able to view the web site with just localhost or localhost:80 but I have to specify port 8080. Also, the traffic we want to forward should be coming from port 80 and this suggests that this has been described the wrong way round in the video.")) |
|
main.append(addParagraph("In any case, you might think that this is largely pointless since much of what we have seen in these examples is really resundant. For example, we can use local port forwarding to send the web traffic from our web server to the dev machine so you might question whether there is any benefit to trying to figure out how remote port forwarding works and yes, it might be advantageous to set up the reverse tunnel so we can work on our dev machine while forwarding the traffic. However, we could also do that using local port forwarding and then opening a second session to work on the dev machine.")) |
|
main.append(addParagraph("Maybe even more relevant is the fact that we can access the webpage on our dev system without forwarding the web traffic just by using the correct url - in this case <a href'osztromok.com'>osztromok.com</a> in our browser. I think that the important thing to remember here is that there may be some instances where it is useful to be able to forward traffic on to a port on your local machine and you do have an ssh connection available, so these tools are really just taking advantage of the fact that the ssh connection is available. At this point, I'm not really all that sure about how this might be useful to me but all of the machines I need to work on are easily accessible, but I can imagine that there may be some situations where these tools may be useful.")) |
|
main.append(addHeader("Dynamic Port Forwarding")) |
|
main.append(addParagraph("Dynamic port forwarding works by creating a SOCKS proxy on the local system which we can use to send traffic out via a port on the local system to an ssh server and that traffic can then be forwarded on to any machine that can access the ssh server. Essentially, it allows you to use a port on your own system to act as a proxy. This is often useful for web browsing when you don't want your local system to be exposed to the internet.")) |
|
main.append(addParagraph("As an example, I will set up a dynamic proxy between my dev machine and my web server so that traffic is forwarded from my webserver to the dev machine. The syntax is pretty similar to that for local port forwarding so I start on the web server and execute the command")) |
|
main.append(addSyntax("ssh -D 3000 philip_dev")) |
|
main.append(addParagraph("In order to use this, we need to configure the system or the browser to do that. For example, to do that in Firefox, we want to go to the settings and look for Network Settings. From there, we will switch over to Manual proxy configuration, insert localhost as the SOCKS Host and 3000 as the Port as shown below.")) |
|
main.append(addImageWithCaption("./images/dynamic_port_forwarding1.jpg", "Configuring the network settings in Firefox to work with the SOCKS proxy.")) |
|
main.append(addParagraph("We can then browse the internet using the proxy as shown below.")) |
|
main.append(addImageWithCaption("./images/dynamic_port_forwarding2.jpg", "Browsing the internet via the SOCKS proxy.")) |
|
main.append(addParagraph("To confirm that we are browsing the internet via the proxy, I will close the ssh session to stop the port forwarding and we should then see that the LinkedIn homepage is no longer reachable and the result is shown below.")) |
|
main.append(addImageWithCaption("./images/dynamic_port_forwarding3.jpg", "Browsing the internet via the SOCKS proxy after closing the ssh session which stops port forwarding.")) |
|
main.append(addParagraph("Once the ssh session has been closed, if Firefox is still configured to use the proxy, we won't be able to browse the internet unless we resinstate port forwarding or we set the proxy connection settings in Firefox back to no proxy.")) |
|
main.append(addHeader("Options to Use with Port Forwarding")) |
|
main.append(addParagraph("In the examples we have seen so far, when we set up port forwarding using ssh, we opened a new session on the remote server. For example, the command")) |
|
main.append(addSyntax("ssh -L 8080:localhost:80 philip_web")) |
|
main.append(addParagraph("not only set up the port forwarding from the web server to my local machine, it also connected us to the web server. This is because the process, as is normal with the ssh command, opens up a new shell session. However, we don't actually have to do that. There are some useful options we can use with port forwarding and these are")) |
|
main.append(addInsetBulletList(["-f - this forks the ssh process into the background.", "-n - this prevents reading from stdin", "-N - this prevents remote commands from running", "-T - this prevents a TTY from being allocated."])) |
|
main.append(addParagraph("The options, -nNT, work together to prevent a new shell from being opened and the -f option puts the ssh process into the background so that we can continute to work in the existing shell. To demonstrate that, we will use the command again to set up local port forwarding from the web server to the dev machine. The screenshot below shows the ip address, 192.168.0.10, which is the address of my dev machine, the ssh command that sets up the local port forwarding (note that I am forwarding to port 80801 this time) with these options and the ip address again which is still 192.168.0.10 showing that we haven't opened up a new shell process.")) |
|
main.append(addImageWithCaption("./images/local_port_forwarding4.jpg", "Setting up local port forwarding using the options -fnNT.")) |
|
main.append(addParagraph("Back on the dev machine, I can then put localhost:8081 into the browser's address bar and the result is shown below.")) |
|
main.append(addImageWithCaption("./images/local_port_forwarding5.jpg", "Browsing the internet via local port forwarding on port 8081.")) |
|
main.append(addParagraph("Since the ssh process has gone into the background, closing the ssh connection will not stop the port forwarding since we put that process into the background which means that it will continue to run even after the ssh session we ran it from is closed. We can check that by closing the terminal and then going back to the browser on the dev machine to check if we can still see the web page with localhost:8081 and we can.")) |
|
main.append(addParagraph("We can also confirm the process is still running by opening a terminal on the dev machine and using the ps command like this")) |
|
main.append(addSyntax("ps x | grep ssh")) |
|
main.append(addParagraph("The output from this is shown below.")) |
|
main.append(addImageWithCaption("./images/local_port_forwarding6.jpg", "The output from running 'ps x | grep ssh' on the dev machine.")) |
|
main.append(addParagraph("This shows that traffic is still being forwarded from port 80 on the web server to port 8081 on the dev machine and it also gives us the process id which is 63853 in this case. To close the connection, we will need to kill that process with the command")) |
|
main.append(addSyntax("kill 63853")) |
|
main.append(addParagraph("and we can then refresh the browser on the dev machine to confirm that the web traffic is no longer being forwarded to port 8081.")) |
|
main.append(addParagraph("We can add these port forwarding options to our config file in the .ssh folder. As an example, one of the entries in my config file for user philip on the dev machine specifies a host called philip_web which allows me to easily connecting to the web server using an ssh key and it looks like this.")) |
|
main.append(addInsetList(["Host philip_web", " Hostname 192.168.0.20, ", " User philip", " Port 22", " IdentityFile /home/philip/.ssh/philip_dev"])) |
|
main.append(addParagraph("I will add another host with the name local-port-forward which I will use to set up local port forwarding from the web server to the dev machine so it will look like this.")) |
|
main.append(addInsetList(["Host local-port-forward", " Hostname 192.168.0.20, ", " User philip", " Port 22", " LocalForward localhost:8080 localhost:80", " IdentityFile /home/philip/.ssh/philip_dev"])) |
|
main.append(addParagraph("This is different to the syntax given in the course video so I am amending the LocalForward directive as shown and I will also add a couple of other entries for remote forwarding to demonstrate the syntax for those.")) |
|
main.append(addParagraph("One thing I discovered when I added the entry in the config file as shown in the course video is that if there is an error in the config file, none of the entries will work so I got an error when I tried to connect with local forwarding which pointed to the line that I had to change. However, I also noticed that when I tried to connect to any other host via the configuration in the config file, I got the same error so an error in the config file (at least this error) will affect all of the configured connections. The fact that open ssh gives you a nice and clear error message is really helpful and it made it very easy for me to find a solution on the internet.")) |
|
main.append(addParagraph("I also added options in the config file for remote and dynamic port forwarding and these are the same as the config section for with the exceptions of the connections nickname and the actual line that configres the forwarding so for remote port forwarding, it looks like this.")) |
|
main.append(addInsetList(["Host remote-port-forward", " Hostname 192.168.0.20, ", " User philip", " Port 22", " RemoteForward localhost:8080 localhost:80", " IdentityFile /home/philip/.ssh/philip_dev"])) |
|
main.append(addParagraph("and for dynamic port forwarding it looks like this")) |
|
main.append(addInsetList(["Host dynamic-port-forward", " Hostname 192.168.0.20, ", " User philip", " Port 22", " DynamicForward 3000", " IdentityFile /home/philip/.ssh/philip_dev"])) |
|
main.append(addHeader("Tools That Use and Extend SSH")) |
|
main.append(addParagraph("For some tools, being able to use a secure ssh connection can be very useful since it allows them to offer features such as file copy with the protection offered by your ssh connection. A good example of this is WinSCP which essentially provides you with a graphical interface for manipulating files on a server via an ssh connection.")) |
|
main.append(addSubHeader("mosh")) |
|
main.append(addParagraph("mosh is an example of a tool that extends ssh. It works pretty much like and uses ssh but it gives you a fault-tolerant connection. It has to be installed on both the client and the server and you can do that on most Linux systems with a command like")) |
|
main.append(addSyntax("sudo apt install mosh")) |
|
main.append(addParagraph("We also need to open a port up for mosh to use - it uses UDP ports in the 60,000 range and we can open one port or a ranger of ports. To open one, we would use a command like")) |
|
main.append(addSyntax("sudo ufw allow 60001/udp")) |
|
main.append(addParagraph("The mosh command is essentially the same command as ssh but unlike ssh, if you are running a command in the shell you have connected to and the connection goes down, the command will keep running. When you mkae the connection using mosh, you might notice something unusual compared to using ssh. Consider the following image.")) |
|
main.append(addImageWithCaption("./images/mosh1.jpg", "Connecting to the web server with the ssh command.")) |
|
main.append(addParagraph("In this case, I have opened a kitty session with a connection to my dev machine and I have then used ssh to connect to the web server. I have then connected to the web server via ssh. This looks fairly normal so I won't comment on it until we see what happens when I do the same thing but using the mosh command.")) |
|
main.append(addImageWithCaption("./images/mosh2.jpg", "Connecting to the web server with the mosh command.")) |
|
main.append(addParagraph("As you can see, this looks exactly pretty much what you see when you just open up a kitty session so I am using ssh to connect via kitty to the dev machine but not connecting to any other server. For reference, this looks like")) |
|
main.append(addImageWithCaption("./images/mosh3.jpg", "Connecting to the dev machine with kitty - for comparison.")) |
|
main.append(addParagraph("There doesn't seem to be much difference between connecting directly to a server via kitty or connecting via mosh aside from a couple of lines generated by kitty. Perhaps more significantly, it shows [mosh] in the title bar and this is similar to what we see when opening a screen. Given that mosh preforms a similar function to screen, I suspect that it is essentially opening a screen when you make the connection so it is kind of equivalent to connecting via ssh and then opening a screen.")) |
|
main.append(addParagraph("It doesn't actually use or require the screen command because you can use mosh even of you don't have screen installed. Nevertheless, mosh is still effectively using a screen so I guess it has that functionality built in and I think that it does make sense to think of mosh as being a combination of ssh and screen.")) |
|
main.append(addSubHeader("blink")) |
|
main.append(addParagraph("blink is an ssh client for iPad and it uses mosh so it gives you not only a secure connection, but also will allow you to reconnect if you do have a dropped connection without stoppoing whatever processes were running. I do, on occassion, connect to my web server from my iPad but only for small and quick tasks such as quickly editing a file or checking its status so that's not really critical for me and I use a simple client called Raspberry Pi Helper which is perfectly sufficuent for me. I believe blink offers free and subscription options but I don't have any reason to consider subscribing. For reference, you can click <a href'https://blink.sh/'>here</a> for more information on blink and there are a bumver of YouTube videos relating to Raspberry Pi Helper.")) |
|
main.append(addParagraph("However, if you do need to connect to a server via ssh, tools kike mosh and blink can be very useful to have and this is particularly true if you are working with an unreliable internet connection.")) |
|
main.append(addSubHeader("rsync")) |
|
main.append(addParagraph("rsync can be used to either copy or sync files both remotely and locally. It has it's own security mechanisms but it can use ssh which might be useful in simplifying the admin on a system. The course video only mentions it briefly, but it is interesting for me because it may be simpler for me to be able to sync the web site files I copy to the server with the live files so it is worth digging a little bit more into it.")) |
|
main.append(addParagraph("There is a useful main on this topic on TecMint entitled <a href='https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/#5_Rsync_Over_SSH'>16 Rsync Command Examples for Efficient File Synchronization</a>. This is by Tarunika Shrivastava and it was updated as recently as July 18 2023 which at time of writing this is around 6 weeks ago. This provides some useful background to the command and covers different options. As the title suggests, it does also provide 16 different examples of using the command and the first one I am interested in is a slight digression from the topic of this course because it doesn't involve ssh. It shows how to sync two directories on one machine. The general format for the syntax for that is")) |
|
main.append(addSyntax("rsync -azvh /source_folder /destination_folder")) |
|
main.append(addParagraph("For reference, the command I usually use to copy the files either manually or via cron is")) |
|
main.append(addSyntax("logger Starting to copy files for website && cp -Rf /home/philip/FTP/website/* /var/www/html/ && logger Finished copying the files for website")) |
|
main.append(addParagraph("In my case, the command I need to run rsync would be")) |
|
main.append(addSyntax("rsync -azvh /home/philip/FTP/website/ /var/www/html/")) |
|
main.append(addParagraph("To test it, I will save this file in Dreamweaver. I have disabled the copy command in cron by commenting it out and checked in the logs - /var/log/sysfile to ensure that it is't already running. Now, if I save this file, WinSCP will copy the file over to the source directory and rsync should copy the file over to the live site so I should be able to refresh the page and see the changes. This is what the website looks like (this is the bottom of the file before I run rsync)")) |
|
main.append(addImageWithCaption("./images/rsync1.jpg", "The bottom of the web page before running rsync - this will be the last line of the page at that point.")) |
|
main.append(addParagraph("I am now adding this paragraph and will run rsync and refresh the page. Notivr that the previous image shows a missing image file so you can only see the caption and this is because I took the screenshot and then saved the file. As a result, when I run rsync, I am expecting that it will update this file and also copy across the image. If it works, the following image will look similar but the image will no be missing and you will also see this paragraph.")) |
|
main.append(addImageWithCaption("./images/rsync2.jpg", "The bottom of the web page after running rsync - this will be the last line of the page at that point.")) |
|
main.append(addHeader("Securing an SSH Server")) |
|
main.append(addParagraph("Obviously, security is very important on an ssh server, so there are a few things you should consider, most of which are configurable in /etc/ssh/sshd_config.")) |
|
main.append(addSyntax("Permit Root Login")) |
|
main.append(addParagraph("The ability to login as root is a vulnerability because if an attacker was able to do that, they would have full access to your server, so it's probably a good idea to disable it. You would do that by setting the value of PermitRootLogin to no. In most cases, there will be no need to login as root, but if there is, you could also set this to prohibit-password which would allow you to set up a login for root with an ssh key. On both my web server and dev machines, this directive was commented out so I have enabled them and set the value to no.")) |
|
main.append(addSyntax("Password Authentication")) |
|
main.append(addParagraph("If possible, you should also prevent other users from logging in with a password. This is done with the PasswordAuthentication directive which I have already set to no on both my machines, so it is no longer possible to log into a user account with a password. It should be quite obvious, but it is very important to ensure that you can log in without a password, in my case I have set up ssh keys, before you set this to no. You should also ensure that PubkeyAuthentication is set to yes.")) |
|
main.append(addSyntax("Port")) |
|
main.append(addParagraph("Changing the port number used to ssh to a server can make it a little bit harder for someone to hack into it, but it's not as effective as it used to be.")) |
|
main.append(addSyntax("Encryption Method")) |
|
main.append(addParagraph("In the sshd_config file, you can configure the server's encrtpytion method for greater security but that is not covered in this course.")) |
|
main.append(addSyntax("User Control")) |
|
main.append(addParagraph("This is not really an issue for me since I am the sole user on my servers but you can use this to control who can or cannot access a server using DenyUser, AllowUser, DenyGroup and AllowGroup directives.")) |
|
main.append(addSyntax("Fail2Ban")) |
|
main.append(addParagraph("Fail2ban can help to protect you against brute-force attacks so you may want to consider installing it if you know that your server is constantly seeing attempted logins from an unknown source. It can ban connections from an ip address that makes too many failed login attempts.")) |
|
main.append(addSyntax("Bastion Host")) |
|
main.append(addParagraph("If you have a number of servers you want to protect, you might want to consider setting up a bastion host. This doesn't necessarily make your servers more secure, but it simplifies security management by ensuring that you only have to safeguard one system and you can then prevent logins from outside your network on other servers.")) |
|
main.append(addSyntax("VPN")) |
|
main.append(addParagraph("You can put your server behind a VPN if you want to ensure only authorised users can access it. This has a number of advantages, particularly in that it will prevent things like brute-force attacks. Note the wording here, it doesn't protect against brute-force attacks, it prevents them altogether. While this type of attack will almost always fail if your server is secured with, for example, ssh key authentication with password logins disabled, it still generates both traffic and logs so there can be overheads even if an attack doesn't succeed.")) |
|
main.append(addSyntax("Fingerprint")) |
|
main.append(addParagraph("Strictly speaking, this is not something that makes your server more secure, but if you provide users with the servers fingerprint, tihs can help them to avoid inadvertently logging into a machine that is impersonating your server. You my be able to get the key by using the command, ssh-keyscan with the servers ip-address but I got a connection refused message when I tried it. This isn't important for me since I only login to my servers from within the network, but if you provide the key to a user, they can add it to their known-hosts file so they don't need to rely on getting that from the server when they log in for the first time.")) |
|
main.append(addSyntax("Organizational Requirements")) |
|
main.append(addParagraph("Again, this is a non-issue for me since my servers are owned by me, but if you are securing your ssh servers on behalf of an organization, you may have to take account of that organizations policy with regard to security. For example, there may be a policy that prohibits the use of passwords for logging in to some or all servers.")) |
|
main.append(addHeader("Troubleshooting SSH")) |
|
main.append(addParagraph("If you can't ssh to a server, the first thing you will want to check is whether the ssh server software is running on that machine and you can do that with systemctl. For example, we can check the status with")) |
|
main.append(addSyntax("systemctl status sshd")) |
|
main.append(addParagraph("With luck, the output should look something like this, showing that the serviuce is active and running.")) |
|
main.append(addImageWithCaption("./images/systemctl.jpg", "The output from checking the status of the ssh server software with systemctl.")) |
|
main.append(addParagraph("If necessary, we can also use the systemctl command to start or restart the ssh server.")) |
|
main.append(addParagraph("If the service is being prevented from starting, we can use the journalctl command to check that and we can also searxh specifically for issues with ssh with the command")) |
|
main.append(addSyntax("journalctl -u ssh")) |
|
main.append(addParagraph("The journalctl using something like less to provide output so you can navigate through it with the arrow or page up/down keys and quit with q. You can also pipe the output to grep or redirect it to a file if you need to do more analysis. For example, let's say we want to filter for a specific ip address, say 192.168.0.5 (which for me is my Windows machine) and put the output into a file called journalctl.txt, the command for that would be")) |
|
main.append(addSyntax("journalctl -u ssh | grep 192.168.0.5 > ~/journalctl.txt")) |
|
main.append(addParagraph("A common source of errors with ssh is typos in the shhd_config file. When you make changes to this file, you need to restart the ssh server for the changes to take effect so it is a good idea to check the status after doing that in case of any problems. Personally, I also like to take a backup of any file I make changes to so I can easily return to the previous version of the file if needed.")) |
|
main.append(addParagraph("If you are seeing issues with users connecting with ssh, but you can see that the service is running, you should check the firewall to make sure traffic on that part is permitted and you can do that with the command")) |
|
main.append(addSyntax("sudo ufw status")) |
|
main.append(addParagraph("Depending on which Linux OS you are using and which firewall software is running, this command may change. It does work on the Raspberry Pi but isn't installed by default. As far as I know, there is no firewall. I just installed ufw on my web server but I don't have it configured yet so if I run the status command I just get back status: inactive. If you are using it, you should see something like this in the output.")) |
|
main.append(addImageWithCaption("./images/ufw.jpg", "Sample output from the command - sudo ufw status")) |
|
main.append(addParagraph("If the port is blocked for some reason, you can open it using the appropriate commands for that firewall software. For ufw, you can check the man page for more information or you can find plenty of info with a Google search!")) |
|
main.append(addParagraph("You might also want to check the user's account if a specific individual is having problems connecting. for exapmle, does the user account exist, is it blocked by a block user or block group directive or is it locked. To check if a user account is locked, you can a command like")) |
|
main.append(addSyntax("sudo cat /etc/shadow | grep philip")) |
|
main.append(addParagraph("If the account exists, you should see something like this.")) |
|
main.append(addSyntax("philip:$5$bzMIb5soxP$e.sT/3BKVaIS/kasnxrqq/VrlYrt6EV2kORWUhjkX/2:19257:0:99999:7:::")) |
|
main.append(addParagraph("The long string of characters after the first colon starts with a $ sign and this is the user's hashed password. If the account is locked, this will start with an exclamation mark")) |
|
main.append(addParagraph("If you know that the user's account should not be locked, you can unlock it with")) |
|
main.append(addSyntax("sudo usermod -U philip")) |
|
main.append(addParagraph("If a user account is locked, it's probably been locked for a good reason so you should be careful about using the unlock command if you are not sure whether or not that account should be locked.")) |
|
main.append(addParagraph("You may also want to check the users authorisation method. How are they trying to login, is that authentication method permitted and are the credentials correct. That means checking their password, perhaps updating it and checking whether the correct key has been stored for that user and is still in place if the user is signing in with an ssh key.")) |
|
main.append(addParagraph("A user will see a warning message if they try to sign in to a server whose fingerprint is not stored in the kmown hosts file. This is something that should always be invesigated. It can be a result of a server rebuild or it could be an indication of a man-in-the-middle attack where another machine is intercepting traffic intended for your server (hence the fingerprint not being recognised).")) |
|
main.append(addParagraph("Other things you can check when there is a problem are the clients ssh configuration, either at machine level in config_ssh or at an individual level in the config file in the users .ssh directory.")) |
|
main.append(addParagraph("A connection can also go down while a client is connected and this will usually lead to the client seeing a broken pipe error. As well as the obvious problems (server shotdown,restarted or crashed) this can also happen if the server goes to 'sleep' so if you want a client to have continual access, you may need to configure power settings on the machine. From the clients point of view, if this happens regularly, it might be worth considering something like mosh for a fault-tolerant connection that can be resumed when a connection is available again.")) |
|
|
|
addSidebar("linux"); |