↓Skip to main content
Tim Luu

Getting into linux administration

·2334 words·11 mins
IT Linux Cloud
Table of Contents

Synthesised from Reddit

Phase 1: Learn
#

  1. Install Fedora Workstation

  2. Learn to navigate and use the Terminal. The Terminal is how you will perform around 90% of sysadmin tasks. Your life is text now.

  3. Read O’Reilly’s “Learning the UNIX Operating System, Fifth Edition” to master filesystem navigation, searching files with grep and find, etc. Plan 15 days to read.

  4. Read O’Reilly’s “Learning the bash Shell: Unix Shell Programming” to learn bash scripting. Another 15 days to read.

  5. Learn the basics of systemd like starting and stopping services.

  6. Learn to use journalctl to access system error logs and text output. The majority of Linux errors are found in text logs. So, you need to know how to access that error text:

    journalctl -u <name of service>

    gets you just the service.

    journalctl -u <name of service> -f

    gets you a “wall scroll” of the service: it keeps updating till you hit control+c

    The important skill you need from journalctl is to just be able to see why a service fails and then copy/paste the suspected error into google to get suggestions.

  7. You will fuck up. Get used to it. There are no helper training wheels, you are 5 years old and you were given an off-road motorcycle. You will fuck up. You may break something. Learn from the fuck up how to NOT fuck up.

Phase 2: Practice
#

Then practice what you’ve learned here:

Phase 3: Homelab
#

Inspired by this post on reddit with only editing is to host a wiki first for documentation purpose (so task 23 becomes task 0)

This is what I tell people to do, who ask me “how do I learn to be a Linux sysadmin?”:

  1. Provision a VM, make that server host a wiki software of your choice and document everything up to this step in there.
  2. Set up a hypervisor (KVM).
  3. Create two VMs on the Hypervisor, these will be used for hosting freeIPA as authentication services for your environment. Remember to set them up with static IP addresses.
  4. Inside of that hypervisor, install a Foreman server. Also set that host up with a static IP address.
  5. Configure the foreman host according to your environment.
    - create organizations in foreman
    - create locations in foreman
    - create compute resources that foreman should deploy your VMs to
    - create compute profiles for your VMs
    - create domains to associate to your hosts
    - create environments for puppet (deprecated)
    - create realms that foreman should enroll your hosts to
    - update settings that you want to adjust
    - create subnets that hosts can be provisioned to
    - create content credentials allowing to be associated with repos
    - create products that will contain sets of software repositories
    - create repositories for the products you created
    - create lifecycle environment(s) to manage content rollouts to your hosts
    - create content views that will contain the required products and repositories
    - create activation keys so that hosts can be automatically subscribed to content
    - create installation media to pxeboot vms that will be provisioned by foreman
    - create operatingsystem definitions (will be utilized in the provisioning process)
    - create hostgroups that combine predefined settings

Use RHEL-based as the distro for all work below.

  1. Assuming you’ve done all of #4, register your freeIPA hosts to the foreman server. Make sure they are getting repo information as expected.
  2. Set up a dhcp-server which will hand out PXE instructions for your environment. This one can be hosted on your foreman server, IPA server or on a separate VM you create manually again.
  3. From here on out, you should be able to provision your first VM unattended. Install Ansible on this host and leave any configuration aside for now.
  4. Provision another VM using foreman. Give this one an extra disk and make it an ISCSI target and install NFS server on it. Use the second disk for nothing OS related, only for NFS and ISCSI storage
  5. Provision another VM. This one will handle email for your environment. Use whatever you see fit for a setup. Here are some options:
    • zimbra
    • iredmail
    • mailinabox
    • mailcow
    • postfix+dovecot+roundcube
  6. Reconfigure all VMs to use that email server you just provisioned for sending mails.
  7. Provision 3 more servers, setup a kubernetes cluster on them. Integrate metallb, nginx-ingress and metrics-server.
  8. Provision another VM. This host will be your internal registry for containers you build yourself. You can use the community edition of nexus or the default docker registry container image. It doesn’t really matter here.
  9. Reconfigure the container runtime on your Kubernetes nodes to utilize the internal registry as well for pulling images.
  10. Deploy 3 more VMs, these will host your Elasticsearch cluster
  11. Provision another VM. Remember the Elasticsearch cluster mentioned in the task before? We need to get logs, so setup Logstash on this host. No pipelines need to be defined yet.
  12. Provision another VM. This one will host Kibana so you have a graphical interface for managing and looking at data stored in Elasticsearch
  13. Set up Metricbeat and Journalbeat on every host you have and configure them to push their data to your Elasticsearch cluster.
  14. Deploy another VM. This host will be responsible for monitoring your environment. Use whatever monitoring system you think has the most beautiful web interface. Some examples here are:
    • Icinga2
    • Nagios
    • CheckMK
    • Zabbix
    • Grafana
    • Prometheus
  15. Provision another VM. This one will host a webserver of your choice with some static web content.
  16. Deploy another VM. This time, it will be a Nginx reverse proxy for the webpage you created in the step before. Configure this host only as a reverse proxy, nothing else.
  17. Remember that internal registry you set up? It’s time to put it to good use and build a container that contains the webpage you created earlier. Build it and push it to the local registry.
  18. At this point, you should have published a container image to your local registry and have your Kubernetes nodes configured to be able to pull container images from that registry. Create a new deployment with your website along that current VM for your website. Add this new endpoint to the reverse proxy.
  19. (moved to task 0)
  20. Recreate all the above mentioned steps using Ansible playbooks. Do not destroy the current environment but create the new environment in parallel.

BONUS:

  • Set up an internal git server, bonus for setting up gitea or gitlab-ce
  • Set up AWX instead of just using plain Ansible (14.1.0 is quite stable and works with docker flawlessly)
  • Set up a backup job(script) for your IPA environment. Make sure to use systemd-timers

Phase 4: Cloud
#

  • Once done with all the tasks, migrate the servers group by group to CentOS 8
  • Migrate your homelab to cloud. Do all your work in the cloud. Pick a cloud provider from AWS, Google Cloud, or Azure, & go to town. Look into CI/CD pipelines, infrastructure as code (IaC). For example
    • For step 6, 7, 17, 18, 20 - - Use Terraform & Ansible. You can use this to either craft a Kickstart or learn Ansible/Terraform. Setup a Server that hosts AWX so you can have Tower-like functionality with ansible
    • For step 16, automate and setup a Git repository to hold your Ansible playbooks and Terraform plans (see Phase 4 below). The documentation can then detail what the playbooks/plans do, how to run then, how to maintain them, etc..
    • Get familiar with containers. Start with just plain Docker at first, then work your way up to learning Kubernetes.
    • For monitoring (observability), Prometheus is king now. Even Cloudwatch & DataDog are implementing similar concepts to what Prometheus does. Replace Nagios with Prometheus

Do these things and you will be fully exposed to every aspect of Linux Enterprise systems administration. Do them well and you will have the technical expertise required to seek “Senior” roles.

Some other sample projects:

  • Setup a couple LAMP or LEMP servers + PHP-FPM + Laravel and Jetstream installed
  • Setup an NFS server for these servers to store your files on
  • Setup replication between mariadb instances
  • Setup Redis server to store your #PHP sessions
  • Setup high availability for the 2 LAMP/LEMP servers with HAProxy load balancer
  • Setup monitoring of those servers using Prometheus and Grafana
  • Finally, try doing a load test with Siege.

Do you want to get into automation and DevOps/SRE? Focus heavily on on technologies like Ansible, Puppet, Git, and Kubernetes. Perhaps you write your infrastructure as pure Ansible roles and craft a one-touch deploy. Also learn cloud, AWS and GCloud both have robust free offerings.

Do you want to dig into pure Linux Administration and Engineering? Do the LFS project ( https://www.linuxfromscratch.org/).

I’ll try to keep the “explanation” part to a bare minimum, since as a Linux administrator you should be able to look up and understand complicated stuff within seconds (as your whole enterprise is burning and you’re the guy who’s supposed to put the fire out).

  1. Install Arch Linux. You’ll learn a lot in the process. But if you’re a complete beginner, just take an extra year or so using Manjaro or PopOS first as your daily driver - just as a regular desktop user who wants to watch videos and play video games - so you won’t loose your motivation (and to get a better familiarity with some Linux system overall). But no dual-booting or virtual machines. Just a Linux OS on bare metal. Also, you should try out Debian or CentOS in your spare time, since the systems derived from Debian or Red Hat are what you’ll most probably find in your work environment. You should be able to install packages on them with blindfolds on.

Got Arch up and running? Make yourself a CLI-driven desktop. You should be able to access the internet, send and receive email, check out weather, play music, maybe play videos in a separate X server, and so on. No GUI. In your admin job you’ll be sitting in front of a terminal for the whole day - every day - so you need to be very, very comfortable with the shell. Plus side: you’ll quickly realize that the text driven computing is actually very good and you’ll miss it very much in GUI-driven environments. After you’re done with the “console desktop”, play around with tiling window managers like DWM, AwesomeWM or i3-gaps.

  1. Learn the shell [bash/zsh] (aliases, functions, loops, subshells, pipes, keyboard shortcuts + grep/tail/head/less/cut/awk/sed/curl/xargs/ps/find/kill) by automating as much of your everyday stuff as you can. If you did something twice, create a function for it. Learn Vim (neovim + plugins) by just using it; also, there are excellent interactive tutorials available. Also, learn touch typing. It’ll get handy very soon. Take out all of the keys from your keyboard with a fork and put them back in random places, so you won’t be tempted to look at it when you type. Getting a mechanical keyboard is also a plus.
  2. Learn enough C to be dangerous in it. If you know a little bit of C, quirks and bugs in other programming languages (interpreters/compilers are usually written in C) are far easier to understand and - eventually - fix. Also, you’ll be able to write simple code in most programming languages you don’t even know yet (just with some documentation reference available). And believe me, during the course of your career you’ll frequently come into contact with foreign programming languages ranging from weird JavaScript dialects through Lua up to Scheme. Most important, however, is the fact that you’ll be able to debug the system errors much quicker and to understand the overall system much better.
  3. Learn Python. Advanced intermediate is a must.
  4. Learn some Functional Programming. Just a little bit. It’ll do some good to your coding habits.
  5. Also, you should know some bits about OOP. Get some Java or C++ basics. (C++ will be faster to learn since you’re already dangerous in C, but understanding Java stack traces will be very helpful in an enterprise environment which is probably drowning in the stuff).
  6. Also, admins are usually terrible at software engineering - you, by contrast, should know the best coding practices (refactoring, code smells). We’re living in the times where most of the infrastructure is a bunch of code (and it should be tidy).
  7. Get an SSH account somewhere. Maybe try out some “free” AWS instance. Bonus: Get an old-school SSH account on some community server with screen/tmux and IRC channel. Mega boost!
  8. Create an Apache (with Nginx reverse-proxy) server hosting a simple website. Does it work? Cool! Now automate the process via Ansible and do the same thing on another, fresh system, but this time you have only 30 seconds to do it. While we’re at it, fiddle around with virtual hosts (and, possibly, NodeJS)
  9. Create a floating IP between your remote systems. Take one of the systems down, see how the High Availability software kicks in.
  10. Learn how to create RAID arrays with mdadm. Plug your filesystems into LVM. Destroy one drive, replace it with a fresh one, watch how the system is self-healing. Extend the partition, shrink the partition, do some other weird stuff with it. You may use files as block devices for the exercise.
  11. Learn Git.
  12. Create a Jenkins instance (and some freestyle projects) for your code in a repository. Make it build and deploy stuff.
  13. Learn Docker/Kubernetes and LXC containers
  14. Learn Grafana/Kibana and Nagios
  15. Learn some database engines (mariadb, postgres), create some kind of Master/Master replication on them.
  16. Schedule database backup jobs using rsync with cron (or a systemd unit)
  17. Recreate exercises #10, #11, #12, #14, #15, #16, #17 and #18 using your very own OpenStack cloud.
  18. Get the basics of networking. Try setting up your own DNS server (might even be useful for cache’ing!) and an email server, redirected by your exercise DNS using proper record. Also, learn what BGP is and how routing works .
  19. Get familiar with system internals and debugging tools: dmesg, strace, /proc/, and so on…

This should be the basics I’d require from an admin for any sensible job position.

Related

Subnet mask and subnetting
·1275 words·6 mins
IT Study-Note Networking
First of all wth is /26 or /24 mask? Then I realised I didn’t understand anything (or at least not deep enough) about subnetting