Several hours went into hypotheses that turned out to be wrong:

• The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
• PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
• Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
• Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
• Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

• Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
• New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

At the time, I replied, “A Software Engineer, because they produce measurable output.”

Now, years later, I reflect on that conversation with a deeper understanding. Since then, I’ve had the privilege of working alongside exceptional Agile Coaches, Scrum Masters, and Product Owners. Yet, let’s be honest:

The majority of Scrum Masters and Agile Coaches lack a clear understanding of the how and why behind their roles. Many find themselves stuck in these positions, not out of passion or conviction, but because they see no alternatives. Instead of pursuing what they believe is right, they focus on maintaining their roles, often to the detriment of their teams and organizations.

The market is saturated with so-called Scrum Masters and Agile Coaches. For every open position, there are often over 200 applicants. Since these roles are unregulated, anyone can claim the title. Unfortunately, this influx of underqualified individuals has created challenges for the Agile community.

The Problem with Unqualified Agile Practitioners

Agile practitioners are perceived as experts, wielding authority. However, when they lack the necessary experience or knowledge, they themselves become the biggest impediments. This issue is compounded when they fail to listen to those they aim to serve, addressing superficial concerns rather than solving real problems. Instead of driving transformation, they retreat into an “Agile kindergarten,” disconnected from reality.

Moreover, the oversupply of candidates drives salaries down, making these roles less attractive to true experts. As a result, the market loses talented professionals who could excel in these positions, creating a vicious cycle.

What Makes a Great Agile Expert?

A true Agile expert must navigate a complex and diverse set of responsibilities, including but not limited to:

1. Team and Dependency Mapping
   Utilizing frameworks like Team Topologies to visualize and optimize team structures.
2. Implementing the Right Workflows
   Choosing and establishing methodologies such as Scrum or Kanban based on team needs.
3. Scaling Agile
   Adopting frameworks like Nexus, Spotify, Scrum@Scale, or SAFe to scale Agile practices effectively.
4. Ensuring Transparency
   Leveraging tools such as Azure DevOps, Jira, and Confluence to provide visibility into workflows.
5. Metrics and Retrospectives
   Establishing metrics and cascading retrospectives to identify and address impediments.
6. Teaching Best Practices
   Guiding teams on concepts like Definition of Done (DoD), Definition of Ready (DoR), OKRs, and Acceptance Criteria (AC).
7. Driving DevOps Adoption
   Helping developers embrace DevOps practices, if necessary.
8. Facilitating Vision and Strategy
   Supporting the creation of visions, missions, and strategies at both the organizational and team levels.
9. Effort and Value Estimation
   Introducing estimation techniques like T-shirt sizes or story points, and prioritization methods like WSJF or Cost of Delay.
10. Conflict Resolution and Coaching
    Combining skills in mediation and coaching to empower individuals, resolve conflicts, and harmonize Agile methods with existing structures.

This list only scratches the surface of what a skilled Agile practitioner must handle. Organizations seeking to hire an Agile Coach should ask candidates to explain these methodologies. Doing so can reveal whether they possess the expertise needed for the role.

Reflecting on the Industry Today

There was a time when companies hired almost anyone with a self-declared title of Scrum Master or Agile Coach. That window gave me the chance to enter this profession. However, times have changed, and so has my perspective.

If my Chief Agile Coach asked me the same question today, I’d answer differently:
“I would hire a qualified Scrum Master—someone with both experience and a strong theoretical foundation. This is a critical role, and you can’t entrust it to just anyone with a certification.”

Agility works. It offers practical solutions to structural challenges that companies face. While traditional waterfall project management isn’t a viable alternative, the challenge lies in finding the right people to implement Agile effectively. Companies must embrace agility to adapt and compete, but doing so successfully requires truly qualified Agile Coaches and Scrum Masters.