2025年12月18日 星期四

Making Jetson Nano Filesystem Tolerant to Power Loss with a Read-Only Rootfs


Jetson Nano boots and runs its root filesystem directly from an SD card, which makes the system fragile under real-world power conditions. Sudden power loss can corrupt the filesystem, and in many cases the entire SD card becomes unbootable and must be fully re-imaged. After this happens a few times, the card itself may no longer be reliable at all.

This post records a set of system-level measures used to make Jetson Nano more tolerant to power cuts and unstable power sources. The focus is simple: minimize runtime writes to the SD card, reduce the chance of filesystem corruption, and—when appropriate—avoid writing to the SD card altogether.
The approaches covered include:

  • SWAP tuning for system stability : reducing swap-induced SD card writes
  • Journald to RAM : keeping system logs off the SD card
  • TMPFS for /tmp : avoiding temporary file writes to disk
  • Disable access-time logging : reducing filesystem metadata writes on read
  • Dirty page flush tuning : adjusting kernel write-back timing
  • Enable hardware watchdog auto-reboot : automatic recovery from hard freezes
  • Root filesystem overlay via initramfs switch : eliminating runtime writes to the system partition

※  The last one, read-only mode is the ultimate protection against SD card corruption. However, it is not suitable during active development, where a writable root filesystem is often required. Running in writable mode restores flexibility, but also removes this layer of protection.

Not all methods need to be applied together. Some are suitable for normal writable development systems, while others are intended for production or appliance-style deployments where power loss is expected and data persistence is not critical. Each technique is described in detail in the sections that follow. 

一. SWAP tuning for system stability


Swap allows the kernel to move inactive memory pages from RAM to secondary storage when RAM is under pressure. On Jetson Nano, swap typically resides on the SD card, which effectively turns the SD card into very slow and wear-prone RAM.

The swappiness parameter controls how aggressively the kernel starts using swap, expressed as a percentage threshold. The default value is 60, meaning the kernel will begin swapping relatively early even when a significant amount of RAM is still available. Lowering this value reduces SD card writes and improves tolerance to abrupt power loss, at the cost of potential lag when RAM is fully exhausted.

※ This tuning is useful in normal (non-overlay) systems. It is not needed when running with a fully read-only root filesystem and RAM overlay.

Check current swappiness:

cat /proc/sys/vm/swappiness

(Default is 60 unless previously modified.)

Set swappiness to 10 via sysctl: 

sudo vi /etc/sysctl.d/99-swappiness.conf

Add the line:

vm.swappiness=10

Apply and verify:

sudo sysctl -p /etc/sysctl.d/99-swappiness.conf
cat /proc/sys/vm/swappiness

Or simply reboot:

sudo reboot

Inspect current swap usage:

cat /proc/swaps
swapon --show
free -h

(Optional) Recreate swap file

Only needed if your current /swapfile is too small or has been removed.

Disable and remove the old swap file:

sudo swapoff /swapfile
sudo rm /swapfile

Create a new 1 GB swap file:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Enable it at boot:

sudo vi /etc/fstab

Add the line:

/swapfile swap swap defaults 0 0

Confirm:

swapon --show


二. Journald to RAM

By default, systemd-journald can store logs on disk under /var/log/journal, which creates continuous writes on an SD card. Switching journald to volatile mode keeps logs in RAM (/run/log/journal), so they disappear after reboot and no longer wear the SD card during normal operation.

※ Not needed in full read-only + RAM overlay mode.

Switch journald to RAM-only by modify the file journald.conf:

# sudo vi /etc/systemd/journald.conf

# Change or add journal storage setting
Storage=volatile       # Always store logs in RAM (/run/log/journal)
#Storage=auto          # Use RAM if /var/log/journal does not exist, otherwise use disk
#Storage=persistent    # Always store logs on disk (/var/log/journal)

SystemMaxFileSize=32M  # Limit individual journal file size

Restart journald or reboot:

# Restart journald service
sudo systemctl restart systemd-journald

# Or reboot to apply
sudo reboot

Verify usage and location:

# Check disk usage of journal
journalctl --disk-usage

# Check actual journal storage location
journalctl --verify

# → Shows /run/log/journal if stored in RAM (volatile mode)
# → Shows /var/log/journal if stored on disk (persistent mode)

三. TMPFS for /tmp

/tmp is used for short-lived temporary files. Mounting it as tmpfs keeps these writes in RAM instead of on the SD card, reducing wear at the cost of some RAM usage.

※ Not needed if a read-only root filesystem with overlay is already enabled, but useful in normal writable mode.

Configure /tmp as tmpfs

# Mount /tmp in RAM (512MB)
# sudo vi /etc/fstab

# Add line:
/tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev,size=512M,mode=1777 0 0

Apply without reboot (or reboot)

# Mount now without reboot
sudo mount -a
mount | grep /tmp

# Or reboot to apply
sudo reboot

Confirm usage

# Confirm /tmp is mounted as tmpfs
findmnt /tmp

# Show how much space is used in /tmp
sudo du -sh /tmp

四. Disable access-time logging on root filesystem

By default, every file or directory read can update its access time (atime) on disk. Disabling access-time logging stops these updates on read operations, while file modification times are still recorded as usual.

※ Useful in normal writable mode to reduce unnecessary SD card writes. Has no effect when a read-only overlay is active.

Edit root filesystem mount options

# Remove access-time logging on root filesystem
# sudo vi /etc/fstab

# Replace:
/dev/root / ext4 defaults 0 1

# With:
/dev/root / ext4 defaults,noatime,nodiratime,errors=remount-ro 0 1

Reboot and verify

# Reboot to apply
sudo reboot

# Output should show "noatime,nodiratime" in mount options
mount | grep 'on / '

五. Dirty page flush tuning 

When an application writes data, the kernel does not immediately write it to disk. Instead, the data is temporarily kept in RAM as dirty pages and written back to disk later in batches. This improves performance, but increases the risk of data loss if power is suddenly cut.

By flushing dirty pages to disk earlier and more frequently, the time window during which written data exists only in RAM is reduced, improving resilience against power loss.

※ Useful in normal writable mode when minimizing data loss on power failure. Has no effect in read-only or overlay modes.

Create a dirty-page tuning config

sudo vi /etc/sysctl.d/99-dirty-flush.conf

Add:

# Start background writeback at 5% dirty memory (default: 10%)
vm.dirty_background_ratio=5

# Block new writes at 10% dirty memory (default: 20%)
vm.dirty_ratio=10

# Force writeback after 5 seconds (default: 30 seconds)
vm.dirty_expire_centisecs=500

# Check for dirty pages every 1 second (default: 5 seconds)
vm.dirty_writeback_centisecs=100

Reboot and verify

sudo reboot

# Verify dirty page parameters
sysctl -a | grep dirty

六. Enable hardware watchdog auto-reboot (Optional)

If the system hard-freezes, a hardware watchdog can automatically reboot the device when it stops responding. Jetson Nano provides a built-in watchdog that can be managed by systemd.

In practice, this mainly acts as a last-resort recovery mechanism for complete system hangs, and does not prevent data corruption by itself.

※ Optional. Mostly relevant for unattended or appliance-style systems; limited benefit during active development.

Check watchdog support

zcat /proc/config.gz | grep -Ei 'dog|tegra_wdt'

You should see entries indicating that the Tegra watchdog is enabled in the kernel.

Test the watchdog directly (forces reboot)

sudo sh -c 'echo V > /dev/watchdog'

If working, the system should reboot after roughly one minute.
Do NOT run this on a system with unsaved work.

Configure systemd to use the watchdog

sudo vi /etc/systemd/system.conf

Set:

RuntimeWatchdogSec=30

Reboot and confirm

sudo reboot

# Verify watchdog timeout
systemctl show --property=RuntimeWatchdogUSec

The reported value should correspond to 30 seconds.


七. Root filesystem overlay via initramfs switch

All previous sections aim to make a writable, SD-card–based system less fragile. However, for real deployments where power loss is expected, the only robust approach is to stop writing to the system partition entirely.

An initramfs-based root filesystem overlay achieves this by mounting the base root filesystem as read-only and redirecting all runtime writes to a RAM-backed overlay. The system runs normally, but any changes made at runtime are discarded on reboot unless explicitly committed in writable mode.

※ Ideal for deployment, but largely unusable during active development, where a writable root filesystem is required.
※ Any data that must persist at runtime must be stored on a separate, non-system partition; this typically requires explicit disk partitioning and mount configuration.

Clone the overlay setup tool

git clone https://github.com/lehni/root-ro.git
cd root-ro

Install the Jetson Nano preset

sudo ./install-nano.sh

This installs a custom initramfs and helper tools that allow booting into either:

  • Read-only root filesystem with a tmpfs overlay (normal safe mode)

  • Fully writable root filesystem (maintenance / upgrade mode)

Reboot into readonly or writable mode

sudo reboot-ro   # Boot with overlay (rootfs effectively read-only)
sudo reboot-rw   # Boot with normal writable rootfs

Switch mode temporarily without full reboot (for testing)

mount-ro   # Remount with overlay (read-only)
mount-rw   # Remount writable (with chroot)

The exact behavior of these commands depends on the root-ro tool’s implementation. The intended workflow is to perform development, debugging, and system updates in writable mode, then return to read-only mode for normal operation.



沒有留言:

張貼留言