[{"content":"Just a day ago I launched this blog, with a first post about how my idea to self-host Immich turned into an entire homelab stack. But apparently I didn\u0026rsquo;t complicate it enough yet\u0026hellip;\nSoon after publishing the post, I got a message from a friend. He complimented my blog post, but also sent a screenshot of how a simple ping command now exposed my home IP, which could potentially reveal an approximate location of the server.\nI had thought about this risk initially while setting everything up. However, I didn\u0026rsquo;t see it as too big of a problem at first, as I was just hosting a personal Immich instance for myself and maybe some people I know, so no one would accidentally stumble across it.\nA public blog changed that - After all, random strangers might now access a service hosted on my server, and I wouldn\u0026rsquo;t want my residential IP to be publicly discoverable to any of them. So, of course, I decided to overengineer a solution.\nHow To Hide Your Homelab IP This issue can be solved in a few ways. The most straightforward way would have been to delegate the hosting of any publicly accessible services to a VPS. So the DNS would simply point there, and it would expose the public IP of the VPS instead of my home IP.\nHowever, I didn\u0026rsquo;t want to limit my homelab to just private projects. I wanted to have the flexibility to decide where each new service should live - my homelab or a VPS, regardless of whether it would be a public or a private service.\nI found cloudflare to be a commonly used solution for this. Cloudflare can act as a reverse proxy for HTTPS traffic. Instead of someone looking up duckylu.uk and getting the server\u0026rsquo;s IP, they would get a Cloudflare IP address. Cloudflare receives the request and forwards it to the server.\nThis has a few nice advantages. Cloudflare handles things like TLS certificates and offers DDoS protection with relatively little to set up and configure. It\u0026rsquo;s a convenient solution for a homelab - but convenience doesn\u0026rsquo;t really seem to be my thing.\nUsing Your Own Reverse Proxy Instead, I looked for an option that could cut out Cloudflare as a dependency. And a good option turned out to be to \u0026ldquo;simply\u0026rdquo; create my own reverse proxy. I could host this on a small VPS, where the DNS records point to the VPS\u0026rsquo;s public IP, which then forwards it to the homeserver.\nI already had a small VPS running with a few services. Conveniently, it was also already running Caddy as a reverse proxy to expose these services. So ultimately I just needed to modify the Caddyfile to forward requests for my blog domain to the homeserver IP:\nduckylu.uk { reverse_proxy https://\u0026lt;MY_PUBLIC_IP\u0026gt; { header_up Host duckylu.uk transport http { tls_server_name duckylu.uk } } } The Host header is important here, as the Caddy configuration on the homeserver uses the domain to decide which service should handle the request. Adding the header makes sure that this information is preserved between Caddy on the VPS and Caddy on my homeserver.\nThen I just needed to update my DNS record to point to my VPS IP instead of my home IP, and everything worked as expected - duckylu.uk now resolved to the VPS\u0026rsquo;s public IP.\nMaking It Slightly More Secure While my current solution did its job, I decided that I wanted to flesh it out a bit more. Currently, my router still had to forward port 80 and 443 to my homeserver so the Caddy proxy on the VPS could reach it. Finding a way to close these ports while still allowing the VPS to direct to them would make my network inaccessible from the public internet, except through the VPS.\nI decided to use Tailscale for this. Tailscale can be used to create a private network between the homeserver and the VPS, allowing them to communicate securely without directly exposing them to the internet.\nTailscale is built on WireGuard. For this use case, Tailscale does much more than we need and WireGuard itself would have been more than enough. However, as I was already using Tailscale for private remote access to my homelab, Tailscale didn\u0026rsquo;t really add much complexity.\nTailscale provides their own control service which you can use to orchestrate the networks and devices. To avoid introducing this as another dependency, I have been hosting Headscale on my VPS as an open-source alternative for the control server.\nI added the VPS to my existing Headscale network, so it could communicate privately with the homeserver (which was already on the network), without requiring anything to be publicly exposed on my home network.\nTo avoid giving the VPS more access than it needs, I configured an additional grant in the Headscale configuration, next to the existing grants for letting my devices access the homelab and VPS. This grant makes sure that the VPS can only connect to the homeserver on TCP port 80 and 443.\n{ \u0026#34;grants\u0026#34;: [ { \u0026#34;src\u0026#34;: [\u0026#34;web-proxy@\u0026#34;], \u0026#34;dst\u0026#34;: [\u0026#34;homelab-pi@\u0026#34;], \u0026#34;ip\u0026#34;: [\u0026#34;tcp:80\u0026#34;, \u0026#34;tcp:443\u0026#34;] } ] } Then I just modified the Caddy configuration on the VPS to point to the internal Tailscale IP instead of the homeserver\u0026rsquo;s IP:\nduckylu.uk { reverse_proxy https://\u0026lt;INTERNAL_TAILSCALE_IP\u0026gt; { header_up Host duckylu.uk transport http { tls_server_name duckylu.uk } } } After this, I could close the forwarded ports on my router. The homeserver was no longer publicly reachable, while the domain continued to work normally through the VPS.\nEvery Solution Has Its Downsides This extra bit of setup helped me to hide my home IP address while still running this blog (and other services) from the server running inside my home.\nThis is what a request looked like before: And here\u0026rsquo;s what it looks like now: Unfortunately, my current setup still has one major downside: There are more potential points of failure. The VPS needs to be running for services on my homeserver to be reachable, and both its Caddy and the Tailscale connection need to be working, on top of all the potential things that could break initially already. Now something could fail on the VPS, causing the services to become unreachable even if the homeserver itself is perfectly healthy. The VPS is probably less likely to fail than my homeserver, but it\u0026rsquo;s still worth acknowledging the additional risk. For me, that trade-off is worth it. I can keep my home network private while still exposing services from home.\n","permalink":"https://duckylu.uk/posts/exposing-my-homelab-less/","summary":"Hiding my home IP while self-hosting public services from home, using a VPS, Caddy and Tailscale","title":"How I Stopped Exposing My Home IP for My Homelab"},{"content":"Until a few months ago, my photos had been backed up and synced into Google Photos. It worked perfectly well - I could easily navigate through my pictures, share them with friends, and they were backed up on something other than my physical devices.\nThere was only one small problem with this: all my photos were with Google\u0026hellip;\nFrom Google to Proton Over the past few months, I\u0026rsquo;ve slowly been trying to reduce my reliance on American Big Tech companies. The reasons behind that, and the alternatives I\u0026rsquo;ve switched to, will probably make for an interesting future blog post. But the important part for this story is that it also meant I didn\u0026rsquo;t want to store my personal photos with Google (even though they pinkie-promise their AI won\u0026rsquo;t scan them!)\nAs I had chosen to move to Proton as my email service, I decided to try out Proton Drive\u0026rsquo;s Photos feature. It did its job for the basics - I could store and scroll through images and videos, and even automatically back up photos from my phone. I could make albums and share media with people, and look through my old photos from anywhere.\nSo what\u0026rsquo;s the problem with it? Well, for me at least, there were quite a few. The options for searching through photos are quite limited. At best, I could filter by date, filename or type, or scroll through the timeline. It also took quite a while to load photos when scrolling through the timeline. It worked well as a private photo backup, and solved the privacy concerns I had with Google, but didn\u0026rsquo;t really feel like a photo library to actually interact with.\nHow Immich Won Me Over I looked around at some other options, and quickly started considering self-hosting something. I had been looking at setting up a homeserver for a while, and even bought two large HDDs some time ago to get myself started, so this seemed like the perfect way to actually put them to use.\nI looked at a few options for a self-hosted photo library and soon came across Immich. Immich immediately seemed to fit my needs perfectly - it is self-hosted, has an interface that\u0026rsquo;s easy to use on both mobile and desktop (and looks nearly identical to Google Photos), has great options for searching and organising pictures, and even supports sharing photos or albums with friends.\nSo I took out my Raspberry Pi that I hadn\u0026rsquo;t used in a while, did a fresh install on it, and mounted my hard drives. I followed Immich\u0026rsquo;s own tutorial for setting it up through Docker Compose, configured Immich\u0026rsquo;s upload/library location on the external drive, created a user, and soon after I could upload and view my photos over the local network. I uploaded a batch of images to test with, downloaded the mobile app, and everything was working as expected.\nAnother huge bonus was Immich\u0026rsquo;s machine-learning features: facial recognition, OCR, and Smart Search, which help a lot with finding specific images. These work through locally running models, meaning none of my pictures are sent to the cloud for processing. On the Pi, it worked surprisingly well, even on the relatively limited hardware. At that point, I was convinced I\u0026rsquo;d found what I was looking for - Immich felt like the right Google Photos alternative.\nSo\u0026hellip; Why Not Stop There? Even though the setup was working as I\u0026rsquo;d hoped, I wasn\u0026rsquo;t fully satisfied with it yet. The two biggest issues I had with this setup were data resilience and accessibility. The images were still only stored on my own hardware. The second issue was that Immich was still only available from my local network, meaning that I couldn\u0026rsquo;t easily use it remotely, and it also wasn\u0026rsquo;t possible to share my photos with others.\nBacking Up Immich Immich itself didn\u0026rsquo;t really end up being a photo backup for me - after setting it up, I quickly realised that it was how I would be interacting with my media from here on out. And of course, only storing the data there would be quite risky. From a corrupted disk to ransomware to me knocking a glass of water over my home server, there were plenty of ways to lose data, and I didn\u0026rsquo;t want to risk it.\nAfter looking around a bit, I settled on Hetzner and their Storage Box option to create cloud backups. Using Restic, I could create versioned, encrypted backups, so that I can restore old versions of files if needed and so that even the cloud provider doesn\u0026rsquo;t have access to the unencrypted media. I decided to back up the full media drive, as well as the most important configuration on the Pi so I could easily restore the entire setup if needed. I configured the backup to run every night while I was asleep.\nI also have two HDDs connected to my Pi, and up to this point, my data didn\u0026rsquo;t even get close to filling up one of them - so I decided to use the second disk for redundancy for now. I figured that the most likely source of data loss would be a drive becoming corrupted or failing, so just having the data on a second local drive would make it significantly easier to restore a backup, as restoring the full backup over the network would take a lot longer than over a direct connection. So I set up a second Restic repository, also backing up the image storage and Pi configuration every night. This wasn\u0026rsquo;t a replacement for the off-site backup, but just a much faster way to recover from a potential drive failure.\nThis gave me a much safer feeling about my data - if one disk failed, I could restore it from the second disk, and even if the entire home server died, I could restore everything from the Hetzner Storage Box. It would take longer, but at least the data would still be there.\nMaking Immich\u0026rsquo;s Machine Learning Faster While Immich\u0026rsquo;s Machine Learning features worked perfectly fine on my Raspberry Pi, processing an initial library of thousands of images took hours to process. Which isn\u0026rsquo;t necessarily an issue, but it still seemed like something I could improve.\nLuckily, Immich supports using multiple machine-learning endpoints. So I decided to set up the same immich-machine-learning Docker container on my personal laptop, and give the Pi access to it over the local network. I set the container to automatically start when I start my laptop, so Immich starts using it whenever it\u0026rsquo;s available. When it\u0026rsquo;s not available, it\u0026rsquo;ll simply fall back to the internal machine learning on the Pi itself.\nThe Real Rabbit Hole: Opening Up to the Internet The Immich setup worked great from my home network. But since it was still running on the local network, it would be inaccessible as soon as I left home. I thought about fixing this through VPN access, but since I also wanted to be able to share links to images, videos or albums, I decided that it would be more worthwhile to just expose Immich to the internet.\nHowever, since the server contains thousands of personal photos, it was important to set this up in a safe way. I could have just port-forwarded Immich\u0026rsquo;s port on the Pi, configured my subdomain to point to my public IP, and called it a day. But even with my limited networking knowledge, I knew that this wasn\u0026rsquo;t the safest option.\nI started looking into reverse proxies, and that seemed like a better approach. A reverse proxy is essentially a service between the incoming request and the other services on the Pi. This meant I only needed to expose Caddy to the internet, while Immich could remain accessible only internally on the Pi.\nI chose to use Caddy for this, as it seemed lightweight and easy to use. It also has the benefit that it handles HTTPS certificates automatically, so I don\u0026rsquo;t need to worry about that anymore. I created a DNS record from a new subdomain to my public IP, set up my router to port-forward it to Caddy, which could then forward requests to Immich.\nAfter just a few seconds, I could reach the new subdomain, Caddy had generated the necessary certificates, and I could use Immich from outside my network through the mobile app and browser. I also tried to share a link to an album with some friends, and they could indeed see the shared content without getting access to the rest of my library.\nAnd Then Things Got Complicated There was just a small issue with my current setup\u0026hellip; My internet service provider doesn\u0026rsquo;t provide a static IP, meaning that my IP address could change if, for example, the router shuts off for some time. Luckily, my domain provider supports Dynamic DNS, meaning that it can automatically update the DNS record to point to my IP even if it changes. I configured this by setting up a small script, running every few minutes in the background. The script makes a call to the dynamic DNS service, which detects the public IP the request came from and updates the DNS record if it has changed.\nWith that issue resolved, I still didn\u0026rsquo;t really feel like my services were protected enough. Yes, it worked, but that\u0026rsquo;s not really good enough for a service exposing thousands of public images to the internet\u0026hellip; I looked into some options to further protect my internet-facing services to prevent suspicious behaviour, and that\u0026rsquo;s where I came across CrowdSec.\nCrowdSec can be used to detect suspicious behaviour and identify malicious IP addresses by inspecting logs as well as crowdsourced block lists. I set it up in Docker, as I had been doing with all other services, gave it access to Caddy\u0026rsquo;s logs, and integrated the caddy-crowdsec-browser plugin so that CrowdSec\u0026rsquo;s decisions could be enforced by the reverse proxy. After a few hours, I could verify that it actually blocked some HTTP probing requests!\nOf course, this doesn\u0026rsquo;t magically secure the entire server, but it does make things like repeated login attempts harder. It was just an extra layer of security that made me feel a bit better about exposing my service. However, this did go from wanting to run a simple photo library to having a reverse proxy, HTTPS certificates, Dynamic DNS and intrusion detection\u0026hellip; I started to see where this was going.\nTwo-Factor Authentication for Immich I also wanted to enable two-factor authentication for my Immich server, just to make sign-in more secure than a password alone. I assumed this would just be a toggle in the admin settings to enable it, as many other open-source services support this. But of course it wouldn\u0026rsquo;t be that simple\u0026hellip;\nUnfortunately, Immich currently does not natively support 2FA. The development team\u0026rsquo;s reasoning is that \u0026ldquo;Implementing a good authentication system is hard\u0026rdquo; and that it\u0026rsquo;s better to delegate it to people who know what they\u0026rsquo;re doing by supporting OAuth - which, I guess is a valid point. Immich doesn\u0026rsquo;t dedicate its resources to building an authentication system and instead focuses on their core product, the image library. And since they do support OAuth, that means external providers can be used to sign in instead of the native username/password system.\nBut with how deep I had gotten with self-hosting this setup, I didn\u0026rsquo;t want to just connect an external OAuth provider to sign in with - so I started looking at other options, and quickly came across Authentik. Authentik is an open-source identity provider that supports OAuth, meaning that I could host my own OAuth provider to authenticate through.\nSo I followed the documentation for setting up Authentik, exposed it through Caddy, set up the same Dynamic DNS logic as for Immich, and soon I had another working service running. It was easy enough to configure Immich to use Authentik for signing in and disable the native username/password authentication following the Official Documentation. I configured a minimum password strength and enforced 2FA, both for my own account and any potential other users I might add in the future.\nMonitoring the Pi I now had quite a few services running on my Pi, with no good way to see the load on it other than through console commands, which wasn\u0026rsquo;t the most convenient. How much CPU and RAM was Immich\u0026rsquo;s machine learning using? How hot is the Pi while running? Did any of my services stop working? Of course, setting up just another service for this felt like the perfect solution!\nI knew monitoring tools like Grafana and Prometheus are commonly used in larger environments for monitoring, but they felt a bit overkill for my needs, and I didn\u0026rsquo;t want to add anything too heavy to my server just to see some statistics. So I looked for some alternatives and came across Beszel.\nSo back I went, configuring another service just like the others, following Beszel\u0026rsquo;s setup documentation, exposing it through Caddy, setting up my DNS to point another subdomain to it, and deciding to also use the Authentik provider I had already set up for authentication. The result was exactly what I needed - a nice dashboard showing how my Pi was performing.\nI set it up to monitor the Pi itself as well as the disks, so it could see the SMART status of the disks and notify me early of potential disk issues. I then configured it to send email notifications when a service goes down, a disk has issues or almost fills up, or the Pi runs under extremely high load, so I would know if I needed to take action in the future.\nBut What If Everything Goes Down? While Beszel could notify me of a lot of potential issues, there are a few cases where it can\u0026rsquo;t help - if Beszel itself goes down, or worse, the entire home server, obviously it has no way to notify me. I decided the best way to solve this was with an external uptime monitor.\nI already had a small VPS running with Hetzner, on which most services weren\u0026rsquo;t really being used anymore. So I decided to clean up the VPS, removing all unused services and updating its OS and packages. Then I set up Uptime Kuma as a Docker container, exposed it behind its own Caddy proxy, set up DNS for another subdomain pointing to the VPS\u0026hellip; You know the drill by now.\nAfter setting it up, I did the initial configuration, set up an account, and added a monitor for each of my services. I decided to use Kuma\u0026rsquo;s own authentication instead of my Authentik provider, as that wouldn\u0026rsquo;t work anymore if my Pi were down.\nI configured Kuma to ping each service every minute, and if it doesn\u0026rsquo;t get a response, it knows the service is down. If a service is down for two pings in a row, I configured it to send me an email notification. This way, potential network issues or tiny restarts don\u0026rsquo;t immediately flag the service as broken, but I still get notified quickly enough if something does break.\nFrom One Service to a Whole Stack At this point, I decided I was finally happy with my setup, at least for now. So, let\u0026rsquo;s take a step back and look at what it actually looks like:\nIt ended up being quite a lot of architecture for what was supposed to be a simple service. But now it\u0026rsquo;s more resilient and (hopefully) secure, and I learned a lot while setting it up.\nConclusion I started out just experimenting with self-hosting an application to conveniently store and access my personal photos. But somehow it turned into a whole personal infrastructure stack made up of backups, authentication, monitoring, security and an external VPS. While not all of these are necessary, it was a fun journey to set them up one by one, solving each new issue that came to mind.\nBecause why would you keep things simple if you can overcomplicate them?\nAnd lastly, to round things up, here is the list of the ten things I ended up running as a result of self-hosting Immich (as the title promises!)\nImmich itself, of course Remote machine learning on my laptop Restic backups to an extra drive and the cloud Dynamic DNS Caddy as a reverse proxy CrowdSec for network protection Authentik for OIDC Beszel for local monitoring Kuma for uptime monitoring If you\u0026rsquo;ve been counting, you might notice that that\u0026rsquo;s only nine things\u0026hellip; So you might ask where the tenth one is. Well, of course it\u0026rsquo;s this blog that you are reading right now ;)\n","permalink":"https://duckylu.uk/posts/immich-self-hosting-journey/","summary":"A journey of how setting up an Immich photo library turned into a full self-hosted stack with backups, remote access, authentication, monitoring and more.","title":"Immich: How Self-Hosting A Photo Library Became Ten Services"},{"content":"This is my blog where I write about things I did, things I built, things I broke, or anything else I find interesting.\nA lot of what ends up here will probably revolve around self-hosting, privacy, programming and just figuring things out, but I also have plenty of other interests I might dive into. You\u0026rsquo;ll find a mix of project write-ups, tutorials, experiments, and just my thoughts and experiences. I don\u0026rsquo;t plan to limit myself to any particular topics - if I feel like writing a post about ducks tomorrow, I just might do so.\nThanks for reading my stuff!\n","permalink":"https://duckylu.uk/about/","summary":"\u003cp\u003eThis is my blog where I write about things I did, things I built, things I broke, or anything else I find interesting.\u003c/p\u003e\n\u003cp\u003eA lot of what ends up here will probably revolve around self-hosting, privacy, programming and just figuring things out, but I also have plenty of other interests I might dive into. You\u0026rsquo;ll find a mix of project write-ups, tutorials, experiments, and just my thoughts and experiences. I don\u0026rsquo;t plan to limit myself to any particular topics - if I feel like writing a post about ducks tomorrow, I just might do so.\u003c/p\u003e","title":"Hey, I'm Duckyluuk :)"}]