Joe P has MASTERED Nebula and shares some tips, tricks, pros, and cons...

Hey there!

I heard Chris mention he would like people using Nebula to reach out about their use of it.  While I don't really have a "production" setup, I do use it daily and have been happy with it.  In fact, just yesterday it came in really handy.  I received a text about a power outage from my iSocket3G that I have set up at the one location, and after waiting 10 minutes or so without getting a notification that power was back, I simply logged into the one machine via NoMachine from my home laptop and shut it down.  I also then logged into the Synology NAS there with a browser on my home laptop and shut it down cleanly as well.  Both were running on UPS power at the time, naturally, but since I am going to that location today anyway, I figured I would just shut everything down instead of monitor the status and hope the power came back on.  (It seems it didn't yet.)


That said, first let's talk the cons of Nebula:

My main concern with it is what feels like the snail pace that development has proceeded at after Nebula has become a Defined Networking project.  There's incidents and pull requests piling up that get pretty much no attention.  At least it feels that way to me, and looking at the Git commit history, you can see that not too much is happening.  I'm fine with a project not seeing much work being done on it when it is pretty much "finished," but there's still quite a bit of work to do on Nebula, even if it's just to address some of the issues.

The second issue is the documentation isn't all that good.  It took me a solid three day weekend with a ton of trial-and-error to get my setup just how I wanted it, iterating over and over again after getting a good idea how it works.

The third issue is it's slower than something like WireGuard (which I also use for communication between various sites).  To me this isn't much of a minus to me though, but it could be if you are moving a ton of data around.  I did see you guys covered Headscale lately, which I admit I don't know much about (or Tailscale for that matter).  Even if I ended up using something like Headscale long term, I'd probably still keep Nebula around just to have an alternate connectivity option.


Okay, that out of the way, let's go into some ways that I use it...

1.) All of the VPSs I have deployed have Nebula on them.  While I allow sshd to bind on 0.0.0.0/0, I use UFW on those Ubuntu 20.04 servers to only allow inbound traffic on the Nebula interface.  This is a nice, secure way to make sure people aren't trying to SSH into machines out on the internet, since they'll never be able to find a port where they can establish an SSH connection.  I also have a Nebula ACL just for SSH connectivity, so that is an additional layer of security as well.
2.) I have all of my traffic for Salt run over Nebula.  All of my VPSs and other servers (and even workstations) all have Salt minions running on them.  I tell them to connect to the Salt master via its Nebula IP, and I have the Salt master bind to its Nebula interface.  I do still keep my Salt master on the LAN at my home, but using this setup, it would probably be just as secure throwing it out on a VPS.  I have a dedicated Nebula ACL for configuration management for Salt as well.
3.) All of my Synology NAS devices run Nebula on them.  While I'm not using this for data replication purposes yet, that will probably be a future use case, just to provide an additional location to have backup data.


Some tips:

1.) Use more than one lighthouse for Nebula.  I use three lighthouses in three different geographical locations with three different providers.  At a minimum I would set up two lighthouses though, if only to make maintenance be non-disruptive.

2.) When you create your certificates, don't just choose a short name for a given machine.  For instance, choose something like nayru.nebula instead of nayru.  In this case, it lets you take advantage of the DNS functionality that Nebula lighthouses can provide, and avoids conflict with things like .local name resolution.  Once set up, you can then just do something like "ssh farore.nebula" without any headaches compared to if you didn't put .nebula (or .jb or what have you) when you created the certificates.  Even if you just go with editing hosts files on machines and not using the Nebula DNS functionality, it is still cleaner/better to be able to utilize names that are essentially "fully qualified".  Do keep in mind that Nebula lighthouses do not actually create DNS entries for any of the lighthouses, so if you need to connect to a lighthouse you named lighthouse1.nebula, for instance, you'll actually have to add that to a hosts file, or just refer to it by IP.  It's not a big deal, but something I thought I would point out.  I'd suggest if you don't have a ton of Nebula machines to track, just add entries in your hosts file.  You can get DNS working, but if you are using something like systemd-resolved, it can cause a lot of headaches when it blows away your custom DNS settings for Nebula every single time there is any sort of network change.  I had to resort to a systemd timer and unit just to fix this on a regular basis for me on devices like laptops.

3.) Utilize the ACLs to your advantage.  I don't do anything crazy with the ACLs, but I do use the groups.  I use groups like the following:

configuration_management_agents
nas_clients
remote_desktop_clients
self_hosted_software_clients
ssh_clients

For example, if I want nayru.nebula to ssh into din.nebula, I make sure that nayru belongs to the ssh_clients group when generating its certificate.  Then in din.nebula's config.yml file, I set up an inbound rule for the ssh_clients group.  I can provide exact examples if needed.

4.) Don't just rely on Nebula.  I also use WireGuard where I can, particularly on machines I really need to ensure are reachable at all times. It also protects you in case you make a change to a config.yml file and fat finger something, for instance.  On a VPS you can work around this with the provider's web interface to a terminal, but if you have a physical machine a long distance away, it's better to be safe than sorry and offer more than one way to connect in.


Okay, I hope this helps, and again, if you need any assistance setting anything up, or want some suggestions, let me know.  I'd be happy to help.

- Joe P