More Work to be Done - Ubuntu Server Hardening

Brian Fajardo -

In the conclusion of the last blog post, we ended up telling ourselves that more work needed to be done to further harden our server. This is exactly what this blog post will talk about. How can we improve the security measures that have already been taken? Let's briefly list those security measures that were taken in the last blog post.

  1. Changes in /etc/ssh/sshd_config file where in line PasswordAuthentication the value was set to no. In line PubkeyAuthentication the value was set to yes. We made these changes because we do not want to use password authentication as it's less secure than using public key authentication as not all passwords are created equal when it comes to their complexity or strength. Public key authentication adds a keypair consisting of public key and private key and also the option to add a passphrase to the private key file allowing an extra layer of security. Lastly the SSH port was changed from 22 to 2222 in the same /etc/ssh/sshd_config file. This was done to obscure SSH scans with port 22 minimizing the potential of fingerprinting or banner grabbing of our host. To further add we locked down our system by ensuring that only SSH connections from our IP address are accepted by editing the security group of our EC2 instance to reflect the source IP address to be ours.
  2. Installing Cockpit Control Panel Access. Although we did not go over much of it's other useful features, in this blog post we will go over more of those features! To summarize, Cockpit is a web-based graphical interface for servers. It helps those new to Linux like Windows admins and those already familiar with Linux, to administer a server without the need to do it from the command line.
  3. Using Let's Encrypt client certbot for generating an SSL certificate for our registered domain used for our UniFi controller allowing https communication with our server and removing the pesky not secure warning.

Now let's dive deeper into the additional features Cockpit offers that will help us plan other security measures that can be taken to further harden our server!

Fortifying the Fort

To start fortifying the fort, we should consider looking at logs to see what activity we are seeing from our server. We had asked ourselves whether in the timeframe our instance was opened up before we made changes to our firewall, could there have been any successful SSH bruteforce attack where a SSH session was opened successfully from a malicious IP? Let's review!

Our instance was created back on July 11th, 2021. The next day there was activity of SSH bruteforce attack attempts. Some pieces of information of interest are the user, IP address, and port from which the authentication attempts are being made.We'll use these pieces of information to further obscure our system should we ever change the port number for SSH connections or create another user on our system. We can also blacklist the malicious IP using an IPS.

By analyzing the logs for the months of July to January we concluded the following:

  1. The users root, admin, oracle, usuario, test, user, ftpuser, test1, and test2 are known users being used for SSH bruteforce attack attempts.
  2. The following IP addresses were linked to the above users and should be blacklisted:
  • 60.8.87.190 (China)
  • 27.110.250.34 (Philippines)
  • 117.248.249.70 (India)
  • 180.103.49.63 (China)
  • 87.241.1.186 (Italy)
  • 189.113.131.44 (Brazil)
  • 36.112.171.51 (China)
  • 103.131.52.4 (New Zealand)
  • 131.1.252.111 (Italy)
  • 222.168.30.19 (China)
  • 131.1.245.34 (Italy)
  • 41.226.25.4 (Tunisia)
  • 212.103.119.94 (Russia)
  • 62.14.232.3 (Spain)
  • 62.32.86.238 (Russia)
  • 189.113.131.44 (Brazil)
  • 177.52.28.116 (Brazil)
  • 201.179.17.181 (Argentina)
  • 185.15.150.12 (Germany)
  • 172.248.49.109 (United States)
  • 187.72.6.225 (Brazil)
  • 204.210.110.164 (United States)
  • 61.1.252.104 (India)
  • 120.224.50.233 (China)
  • 189.162.126.198 (Mexico)
  • 184.82.72.123 (Thailand)
  • 179.39.131.231 (Argentina)
  • 169.197.141.98 (United States)
  • 124.79.244.108 (China)
  • 50.70.207.244 (Canada)
  • 202.29.214.13 (Thailand)
  • 185.238.72.164 (Poland)
  • 14.102.30.33 (India)
  • 45.238.165.32 (Brazil)
  • 185.124.87.159 (Turkey)
  • 185.213.167.35 (Iran)
  • 114.92.193.67 (China)
  • 13.72.72.237 (United States)
  • 95.172.47.98 (Russia)
  • 150.107.92.222 (India)
  • 164.177.78.187 (France)
  • 60.170.247.162 (China)
  • 91.137.125.250 (Germany)
  • 170.244.192.190 (Brazil)
  • 203.253.68.244 (South Korea)
  • 201.249.146.101 (Venezuela)
  • 182.78.137.222 (India)
  • 36.110.228.254 (China)
  • 203.253.66.253 (South Korea)
  • 185.110.184.127 (Switzerland)
  • 85.95.191.186 (Russia)
  • 181.204.99.26 (Colombia)
  • 185.238.36.24 (France)
  • 143.198.120.20 (United States)
  • 124.79.246.228 (China)
  • 70.49.126.223 (Canada)
  • 45.155.173.143 (Germany)
  • 109.230.253.146 (Germany)
  • 45.138.48.211 (Germany)
  • 90.84.198.49 (France)
  • 124.79.244.250 (China)
  • 65.255.182.223 (Canada)
  • 97.71.110.39 (United States)
  • 186.46.86.218 (Ecuador)
  • 2.50.12.216 (United Arab Emirates)

As you can see these are a lot of IP addresses. Cockpit offers the option to filter the timeframe, severity, and service for which you want logs for. For example in the screenshot above, the filters are set to any date, severity error and above, and service all. This is not ideal when we want less noise from our data. The right approach I should have taken from the beginning would be to keep everything as is but for service set to sshd as this is the service for which we are seeing suspicious activity which we proved to be malicious when we referenced VirusTotal and from our own knowledge of our system.

3. Since none of the sshd entries for the months of July to January show "session opened for user any of the known malicious users" this concludes that there was no successful SSH bruteforce attack where a SSH session was opened successfully from a known malicious IP.

Taking the Next Steps

Even though we extensively reviewed the logs to see if there was any successful SSH session opened from a known malicious user, this is also not ideal especially if one were to discover many more malicious users. In this situation we need to now focus on the network layer side of things. What do I mean by this? Aren't we already dealing with this since we are talking about SSH? Yes! However there needs to be more emphasis on what exactly more we can do on the network layer. Robert David Graham a well known InfoSec expert amongst the InfoSec community once said the following at his DEFCON 22 "Mass Scanning the Internet" talk in 2014, "Sadly todays methodology for monitoring the network tends to monitor the inbound stuff." Robert later tells a joke in the form of a story, "Guy is looking on the ground, and is looking for something. And his friend asks him what are you looking for? Well I dropped my keys over there and I can't find them. And his friend asks but why are you looking over here? And the guy says, well the light is better over here." Robert explains the story by saying, "They monitor inbound attacks, because the lights better inside because you put an IDS on the wire, and you immediately see alerts coming up. The detection you really want is on the outbound side." This methodology for monitoring the network that Robert proposes is something that caught my attention when I watched his presentation three years ago on YouTube. Not many people would think to monitor the "outbound side". However just as it's important to monitor the inbound stuff of a network, it's equally just as important to also monitor the outbound side. Although there is more light as the story says on the inbound side, the outbound side has light too if we put in the extra work to make this side brighter!

Making the Light Brighter All Around - Using Wazuh as our SIEM Solution

Wazuh is an open source security platform that I was introduced at my former company. I got to learn some of the basics as to what the tool accomplishes and even troubleshooted some issues with the agent on a few domain controllers. Now I want to leverage this tool again so I can not only learn more of it's features, but also further harden our server and the Windows servers I also created a while back. The installation guide for Wazuh can be found here. In my deployment I selected the all-in-one option. The difference comes down to being whether you want the Wazuh server and Elastic Stack installed on the same host directly on your system, or you prefer a distributed deployment option where each component is installed on a separate host as a single-node or multi-node cluster. Wazuh describes this deployment option as offering high availability and scalability and is convenient for large working environments. Since we do not have such environment, we went with the all-in-one deployment option. If we take a look at the requirements listed by Wazuh for the all-in-one deployment option, Ubuntu Server is not a supported operating system. I have already launched an EC2 instance of Amazon Linux 2 which is supported along with a few other operating systems. We will use this new instance to strictly host our Wazuh server and Elastic Stack. Wazuh mentions that a typical use case for this kind of environment supports around 100 agents. Since the requirements are pretty hefty where minimum is listed at 4 GB of RAM and 2 CPU cores, and the recommended 16 GB of RAM and 8 CPU cores 64-bit operating system, having a dedicated system that strictly hosts our Wazuh server and Elastic Stack components is crucial to ensure resource consumption is kept properly maintained. In my deployment I selected a t2.large instance type which comes with 2 vCPUs and 8 GB of RAM. This configuration is somewhere in between the listed minimum and recommended requirements from Wazuh. This will allow me to test my environment somewhat better because I am not short on resources, and I am also not anywhere near the listed minimum or recommended requirements. A sweet spot perfect for testing resource consumption should I ever have to downscale my instance type. Wazuh provides two installation methods one is unattended installation which is an automated way of installing Wazuh using a script, and the other is step-by-step installation. I went with step-by-step just so I could see what exactly is being installed. Once you decide on the installation method and complete the steps, you should be able to login to the web interface of Kibana by entering in the URL: https://<wazuh_server_ip>.

Kibana Web Interface

Once you log in with the default credentials you will be at this page the Wazuh manager.

Right now there are no agents in our environment. The agent is what will be used to monitor our endpoints. It will monitor and enhance the security of our system by providing agent modules each with different purposes and settings. To install an agent on an endpoint follow the instructions here choosing your operating system. I chose Linux as I want to install the agent on my Ubuntu Server. If your system is in AWS like mine, you will need to add another rule in your security group so the endpoint with the Wazuh agent can communicate with the Wazuh manager.

On the Wazuh server instance, add a custom TCP rule that reflects both port 1514, which is the default port listed in the /var/ossec/etc/ossec.conf file for the Wazuh agent, and the public IP address of the endpoint with the agent. I would allocate an elastic IP address and associate it with the endpoint with the agent to make things simple. 

Upon completing the steps required to install the agent, you should now see both total agents 1 and active agents 1.

If we click on the number one under active agents, you will be taken to the next screen. In this next screen click on the "open summary panel for this agent" under actions.

You will see a new screen with a summary of your system. Taking a further look at SCA: Last scan reveals how the security posture of our system is currently doing. There is certainly a lot we can do based on the results of the scan. I won't show the entire scan of what failed and what passed, but this is certainly an excellent starting point to further plan out other security measures. We will review additional features of Wazuh like using a NIDS to work in tandem with Wazuh in a future blog post.

Future Plans

In a future blog post we will cover more of Wazuh by leveraging this tool even more. We learned that security is a multi-faceted challenge so it deserves to have multi-faceted solutions. One thing for certain is that persistence is your best friend in this game. What else could we do better in? Is there anything that we are already doing good in but can just make a small tweak to and improve our security posture significantly? These are just some questions we can ask ourselves. I hope you enjoyed this post. Please reach out to me and share your thoughts on what you think is interesting or what we can do better in.

Stay secure. :)