You Don't Love Systemd Timers Enough

(blog.tjll.net)

135 points | by yacin 5 hours ago

21 comments

  • thomashabets2 55 minutes ago
    I haven't used systemd timers enough to disagree, but

    > Ambiguous $PATH settings make cron script execution difficult to predict.

    What makes you say that? You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?

    > You might feel cool knowing the scheduling grammar by heart

    I've used Linux since 1994 and I don't know it by heart. But luckily it's pre-printed in the crontab as comments:

        # For more information see the manual pages of crontab(5) and cron(8)
        # 
        # m h  dom mon dow   command
    
    You just put numbers aligned with the titles.

    The rest of the complaints, sure. Next time I need a cronjob, I'll try it out.

    • jerf 4 minutes ago
      "You just put numbers aligned with the titles."

      That is not a fair summarization of their point because that is not the grammar. There's commas, slashes, asterisks, combinations, and then if you want randomization you need to put it in the command itself because cron can't do it. (Some crons can, but it's not a general capability of cron.) Writing a non-trivial cron spec is not easy.

    • jchw 19 minutes ago
      The main nice thing about the environment in systemd is that it is standard and mostly a blank slate, whereas at least for me I was always getting bit by the fact that the environment in Crontab was completely different from say, the environment inherited by supervisord or sysvinit scripts. In systemd the actual unit that gets executed is the same regardless of what triggers it, so there is no gap.

      That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.

      I'd have to concur that I agree this is an advantage of systemd.

  • NikhilVerma 1 hour ago
    I have a Canon printer, I actually can't trust that their print nozzle won't get jammed up after sitting idle for a while. So I had claude setup a systemd script to print a picture of my dog every week, I ensure it has enough CMYK spectrum to stress the printer. Its a nice surprise every monday as I sit on my desk to see a sudden picture pop up from the printer :)
    • tdeck 1 hour ago
      I wish printers could have a mode like this to print random images from an album, or a calendar, rather than wastefully draining ink into a sponge every few days.

      If nothing else, maybe it could be some kid's high school science fair project idea.

      • sunrunner 1 hour ago
        How about printing a QR code for a randomly generated private key for Satoshi Nakamoto's Bitcoin wallet, then every few days you get a tiny moment of excitement, hope, and then disappointment. It's still wasteful, but it could pay off big time?
        • NSUserDefaults 1 hour ago
          Or if you have a printer/scanner combo, you can turn it into a pen pal!
    • magicalhippo 1 hour ago
      Dad had an Deskjet 720 or something like that.

      It sat unused and powered off for a couple of years after he passed, until I needed a color print.

      Didn't do anything but hook it up to power and print. Took about 1/5 of a page until all colors were back in action, after that it printed about 20 pages flawlessly.

    • ThePowerOfFuet 1 hour ago
      Laser printers are your friend. The savings on consumables alone will make it pay for itself.
      • tobyhinloopen 1 hour ago
        Epson Ecotank. I’ve been using mine for years and I only had to buy new ink once.

        And I printed a lot of photos, notes, documents, etc

        • Blackthorn 54 minutes ago
          Those will still get their nozzles clogged if you don't run them.
      • hmng 1 hour ago
        Not to mention more water resistant, when printing things like envelopes.
    • RicoElectrico 1 hour ago
      I was about to recommend a cheap OKI LED color printer (I think C322dn); alas they withdrew from consumer market :/ The colors are super nice and uniform, even if the maximum resolution is only 600 dpi - and the toner won't dry out, which was my brother's crucial purchase criterion; we had HP inkjet clogged more than once.
  • gchamonlive 2 hours ago
    Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixed time. With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.

    Btw this is my repo for the backup automation: https://github.com/gchamon/borg-automated-backups

    • mid-kid 1 hour ago
      Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules.

      To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this.

      Many cron implementations have a similar mechanism.

      • gchamonlive 38 minutes ago
        EDITED

        This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.

        • newsoftheday 17 minutes ago
          > fallback to running when the system comes online.

          That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.

          I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then.

          > With @hourly I lose this control and multiple machines could potentially trigger backups at the same time

          Then don't use @hourly, use staggered times, it's very easy.

          • gchamonlive 11 minutes ago
            > That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.

            This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.

        • layer8 11 minutes ago
          Cron also has @reboot. Not exactly the same, but has been sufficient for me so far.
    • newsoftheday 22 minutes ago
      I'm sorry, I tried Googling the word "tolates" but I can't find any definition that makes sense?

      > runs the service as soons as the system is available.

      cron has the @reboot option which I use for a few scripts and works great.

      • gchamonlive 20 minutes ago
        Typo, I meant tolerates. Fixed it.

        Not an option either, because if I reboot two machines and the backup starts in both of them it'll cripple my NAS

      • regularfry 20 minutes ago
        "tolerates".
  • kayson 41 minutes ago
    I love systemd timers! I've slowly moved all of my ansible-deployed cron jobs to timers (now just an ansible copy!). The integration with journalctl, especially in a newer OS like Debian 13 where syslog is gone, is really nice. It's also really nice to be able to start the service manually for debug. Having a cron job that didn't work was an annoying exercise in copy/pasting or writing an extra shell script. Don't even get me started on the black hole of cron job stdout. I can monitor systemd services like I already do and get a notification on failure.

    I've noticed more and more open source projects recommending timers as a deployment method and I think that's great!

  • mindcrime 16 minutes ago
    I haven't always been the biggest fan of systemd in some regards, but I will say that I mostly agree with this sentiment. I've almost completely quit using cron, and now favor systemd timers for scheduled jobs - at the "system" level anyway. I might still embed Quartz for scheduling that's scoped to a particular application or something.

    Why? It's one of those fuzzy and somewhat hard to explain things. The systemd approach just maps more cleanly to my mental model of "how things should work" I guess. And maybe some of it is that I did indeed experience plenty of " Ambiguous $PATH settings make cron script execution difficult to predict" in the past, although it's not just that.

    I won't sit here and claim that systemd timers are necessarily better than cron in any universal / objective sense. But they've won me over, for what it's worth.

  • stryan 2 hours ago
    Timers can work with arbitrary units (not just a similarly-named service unit) so they can be surprisingly flexible. I have a timer on my servers that starts a backup.target that fires off a full "restic backup","restic prune", "restic forget" backup cycle each morning with randomized start times and notifications. The actual restic-* units are Podman Quadlets so the whole setup runs agnosticaly of what's on the server, just as long as it has Podman and Systemd installed.

    I will admit thought, timers are up there in terms of being the clunkiest systemd unit type to use on a regular basis. I get why they're split up into two files and require different start vs enable syntax's, but man sometimes I just want to create a file that runs a script and be done with it.

    • esperent 1 hour ago
      Why do you randomize your backup times?
      • stryan 1 hour ago
        Should have been more clear: I use RandomizedOffsetSec= to add a random offset to a set start time (usually 4am), to prevent overloading the backup server, not truly random start times.
  • hombre_fatal 2 hours ago
    NixOS comes with systemd, so I've been using it as a first-class part of managing stuff. It's great, especially coming from macOS' launchd.

    Which makes it nice to distribute a tool for NixOS so that it can lean into systemd instead of as some bolted-on afterthought.

    Makes me wonder what you'd do if you were distributing a lifecycle-heavy tool for Linux users in general since systemd isn't ubiquitous.

    I use a systemd timer to run a monthly scrub for my btrfs pool. Kinda cool how you can do increasingly useful things like skip the next scheduled event if the user initiates a scrub, do or don't accumulate tasks if you have a monthly task but the machine was offline for 6 months -- or fold them into a single task, etc.

    • Cyph0n 2 hours ago
      +1, NixOS makes working with systemd a breeze. Defining units in Nix beats wrangling INI files.

        systemd.services.sync-recyclarr = {
          serviceConfig.Type = "oneshot";
          path = [ pkgs.podman ];
          script = ''
            podman exec -it recyclarr recyclarr sync radarr
            podman exec -it recyclarr recyclarr sync sonarr
          '';
        };
        systemd.timers.sync-recyclarr = {
          timerConfig = {
            OnCalendar = "daily";
            Persistent = true;
            Unit = "sync-recyclarr.service";
          };
          partOf = [ "sync-recyclarr.service" ];
          requires = [ "podman-recyclarr.service" ];
          wantedBy = [ "timers.target" ];
        };
    • drunner 2 hours ago
      Have you been defining them directly in your flake.nix file? I too am on nixos but I keep all my configurations in their native format and symlink them with nix, that way I can take and reuse that config on a non nixos system easily.

      The problem I have found is that nixos doesn't seem to pickup and run systemd timers and services placed into the ~/.config/systems/user folder and additionally things like WantedBy=default.target have no effect.

      So after I restart all my services manually on reboot I agree, systems timers are cool.

      • Cyph0n 2 hours ago
        I define all units in Nix because:

        a) It is way nicer and you get decent validation at build time

        b) A LLM can port units over if the need arises; it’s a very light abstraction around systemd syntax

        c) I personally don’t see how I would ever move to another distro :)

  • frays 1 hour ago
    I've been using Linux for over 20 years, systemd for over 10.

    Yet there's always something new to learn and actually consider as another useful tool.

  • sammy2255 1 hour ago
    I've converted all my crons to systemd timers+services over the past year but cant help but think it's sort of.. less tangible than cron

    Like imagine trying to explain systemd timers and services and unit files to a beginner.

    • darkwater 43 minutes ago
      > Like imagine trying to explain systemd timers and services and unit files to a beginner.

      I think it's... easier? Like "systemd is the place where your system manages all the processes it needs to run. Part of those processes can be run on a schedule, or on a timer, and you define them using this simple text file".

  • lanycrost 1 hour ago
    systemd is complex on first view, but after using it you didn't want to use anything else. It's handy to manage everything using systemctl
    • alyandon 1 hour ago
      That and systemd having actually useful man pages.
    • the_real_cher 56 minutes ago
      This is such a modern view. People used to HATE systemd when it first came out, but I always liked it and knew people would eventually come around and its nice to see they finally did!
  • ktm5j 2 hours ago
    Oh I love them quite a lot! I use them to run all of our backup jobs, easy to set up and have never had an issue.
  • mkesper 24 minutes ago
    There's another big feature: You're not relying on the time zone to which the server was set (like with cron) but can explicitly specify a time zone:

    https://www.freedesktop.org/software/systemd/man/latest/syst...

  • cmsj 52 minutes ago
    Is there a way yet to force-trigger a timer? There wasn't the last time I used them, which I found to be super annoying for testing them.
    • Biganon 46 minutes ago
      It's covered in the article, you can simply start the unit that would be started by the timer.

      Oh but it won't appear in the timer-specific logs, I guess...

  • jjgreen 5 hours ago
    I've been almost convinced by systemd (and have switched to using it), but God the syntax of those service files is so ugly ...
    • zamadatix 2 hours ago
      Never thought I'd see hackers saying INI format looked ugly of all things. It's basic, sure, but that's a good thing for something meant to be easily editable by hand from any editor. Otherwise, it's just key value pairs in named sections, how ugly can it be about that?
      • jjgreen 2 hours ago
        key-value pairs where the = cannot be surrounded by spaces, so I have to write

          [Service]
          Type=oneshot
          WorkingDirectory={{ home }}/current/
          Environment=RAILS_ENV=production
          ExecStart=/bin/sh -lc "bin/db-backup --verbose"
        
        which fills me with sadness
        • voxic11 2 hours ago
          Whitespace immediately before or after the equals sign is completely ignored by the parser. Its the standard INI format.
        • yjftsjthsd-h 2 hours ago
          What? You absolutely can have spaces; most of mine look more like

            [Service]
            Type             = oneshot
            WorkingDirectory = %h/current/
            Environment      = RAILS_ENV=production
            ExecStart        = /bin/sh -lc "bin/db-backup --verbose"
          • jjgreen 1 hour ago
            Friend, you have changed my life
            • weavie 1 hour ago
              Is this one of those cases where at one point you had an error in the file and you figured it was down to spaces? You fixed that issue, it still didn't work but from that point you never thought to question the assumption.

              I find myself doing this sort of thing all the time..

              • troyvit 43 minutes ago
                That was (cough still is) ddclient for me.
      • ramon156 2 hours ago
        TOML would look a lot more quiet, but I'm not sure if TOML would be a good fit
        • kevinmgranger 1 hour ago
          unit files barely have any nesting, so the INI-like format is already 90% of the way towards TOML, no?
    • mrweasel 2 hours ago
      There's definitely some weirdness to certain parts of systemd service files, but was a huge improvement over Upstart and the old SysV-style init scripts.

      Over all I think Systemd get way to much criticism. You don't have to use all the parts, but if you care to go through the documentation you'll find interesting features such as journald log-shipping and systemd-machined which can manage containers and VMs.

    • linsomniac 1 hour ago
      Hard disagree. Compared to an init script, with all its boilerplate, I'd take a systemd unit file.
    • SEJeff 2 hours ago
      Oh yes, because the well documented clean syntax of sys v init shell scripts was so nice.

      If I never recall hacking in ulimit calls in the top of buggy shell scripts for crappy old services that done respect pam_limits it won’t be soon enough.

    • WesolyKubeczek 2 hours ago
      Could have been worse.

      Could have been YAML.

      Could have been XML.

      • silvestrov 2 hours ago
        XML would have the advantage of having a grammar so we could validate the config files.

        It would also make it much simpler to make good GUI editors for the files instead of the Notepad approach most unix config files take.

        • pwdisswordfishq 2 hours ago
          The systemd dialect of INI is actually pretty well-defined though.

          https://www.freedesktop.org/software/systemd/man/latest/syst...

        • WesolyKubeczek 2 hours ago
          Since systemd is successfully parsing its INI files, and barks at you when you put weird shit into them, a grammar for them does exist as well.

          XML is that wonderful format that gave us vulnerabilities like death by million laughs, up to a certain moment, you could MitM DTDs, and a whole slew of everything-XML stuff back when XML was like AI is today, none of which I miss today.

          Oh, and remember times when programmers would argue whether argument order in XML files should be significant or not?

          But XML books with their idealized XML future description did give me the same warm fuzzies as some intricate clockwork mechanism to a Victorian geek.

        • Juliate 2 hours ago
          There are good GUI editors for XML?
      • Tsiklon 27 minutes ago
        XML - I see you’ve used macOS’ LaunchD, the system that inspired Systemd
      • wpm 1 hour ago
        Could have been better.

        Could have been XML Property Lists.

        ducks

      • jjgreen 2 hours ago
        To be honest, I think either of those would have been better ...
    • whateveracct 2 hours ago
      This is why I like NixOS. Defining systemd services in it is very neat.
  • andrewstuart 2 hours ago
    Even better is systemd socket activation.
    • wmanley 1 hour ago
      I design all my services expecting to receive sockets this way. It makes sandboxing easy as the service itself doesn't need network access to have a listening socket.

      It's a shame docker never supported it. I feel like if they had got on board all those years ago there would be broad support across the software ecosystem for it and we wouldn't need half of these complicated iptables rules and proxies and service mesh. It would be a step towards a capability based system.

    • interf4ce 1 hour ago
      This is very interesting. I'm not sure what I'd use it for yet, but I imagine it could be useful for triggering ad hoc jobs over the network. Maybe have Home Assistant make a network call to kick off a daily back up when I leave the office at the end of a work day.
      • kevinmgranger 1 hour ago
        I believe its original motivation was just speeding up boot times by starting fewer services, even if you'd eventually want the service running. This was achieved in the past with xinetd, but systemd made the approach more popular for the masses.
        • roryirvine 29 minutes ago
          inetd began to fall out of favour in the mid-late 90s as services became more heavyweight and startup times became longer (think of the initial crypto setup needed by sshd vs rsh/telnetd)

          CPU speeds have increased & and i/o latency has decreased so much since then that startup times are generally imperceptible, so the pendulum has swung back to favouring socket activation.

          The anti-systemd "traditionalists" never seem to acknowledge that history, though!

  • baggy_trough 51 minutes ago
    In decades of trying, I do not believe there was one time that I ever got a cron job to work properly in the first attempt. Systemd timers are a godsend.
  • sunshine-o 39 minutes ago
    This is actually something that I like in systemd.

    I am dealing with mostly non systemd system: BSD, Alpine, termux On BSD anacron works well, but I do not why I am always running into problems with the cronie anacron implementation. And it is very hard to debug.

    I would really like a simple modern cron/anacron alternative.

    Cronicle looked cool but it is node.js, a bit heavy and being replace now by their new product called xyOps anyway.

  • iso1631 2 hours ago
    > humble systemd
    • pwdisswordfishq 2 hours ago
      That the same cannot be said of its maintainer is another matter.
      • the_real_cher 55 minutes ago
        The main person in charge of Linux itself isn't considered the most humble but they make amazing products.
  • iluvcommunism 2 hours ago
    [dead]