Why/when/what does a running LiveUSB session write data onto the USB disk?

[Member rayluo]

My usage pattern has been creating one antiX liveUSB disk, and then use it exclusively, while hopping among my several computers, instead of installing antiX (frugal or not) on each of them. (Besides, some of them do not even equip with a hard disk.)

I understand that, reading data from a USB device is generally slower than reading it from a hard disk, as described in the “Introduction section of the antiX Frugal Install” page. In practice, my liveUSB’s presumably slower app starting time (than if I was running a hard disk based installation) is intangible, because, after an app – such as a browser – starts, it stays in memory and my opening-a-new-tab operation is not slow at all.

But, after the browser has been run for a longer time, or when a tab with a long video is being played, and then I try to open a new tab, my computer sometimes becomes sluggish. At that time, the CPU Load monitor on SysTray shows half of more red color, indicating that the CPU was busy on “sys” mode, I believe. And, my liveUSB flash disk’s I/O light is blinking. It can recover, if I close that busy tab.

My question is, why/when/what does a running LiveUSB session write data onto the USB disk? If it is some buffer data, can I somehow setup a ram disk for it?

Note:

1. antiX is supposed to be able to be run on RAM.
2. When my computer has a hard disk, I already create a large enough swap partition on it. And it seems the swap was not exhausted during the above situation. Even if it was exhausted, I did not anticipate a liveUSB session would use the USB disk itself as buffer, or, should I?
3. I could try the “toram” option when booting from a liveUSB, so that I can eject the USB disk entirely. But not all of my computers have enough memory for that. Besides, if the write-to-USB-disk shouldn’t happen in the first place, the “toram” option should not be necessary.

Is there any tweak options that I need to do, to avoid the I/O on my usb disk (other than those on ~/Live-usb-storage)?

[Moderator christophe]

Helpful

Up

0

::

I can’t address your situation in full, but I can share some tips on running live. Things I always do:

1. NO swap FILE usage (whenever you remaster or add persistence files is when the system asks this, I think).

2. use ZRAM for swap — https://mxlinux.org/wiki/other/zram/

Hope this helps some. 🙂

confirmed antiX frugaler, since 2019

[Member Xecure]

Helpful

Up

0

::

My post got sent to spam again (I edit too much).

I was explaining why I choose dynamic persistence.

That is one of the reasons I use (dynamic) root persistence

Root Persistence

Root persistence allows you to save changes to the system in addition to saving personal settings. It has two significant limitations. These limitations are due to the fact that as the system is running the changes are stored in RAM and will be lost unless they are specifically copied back to the rootfs file. If your computer crashes before the changes are copied back then your changes will be lost. The other limitation is that the amount of new information you can store is limited by how much free RAM you have.
Dynamic root persistence saves file system changes in RAM (just like a normal LiveCD/USB). The persist-save program must be run to copy these changes to the rootfs file so they can be remembered across reboots. Use persist-config to have the program run automatically on shutdown/reboot.

On every boot, all (persistent) changes are loaded to RAM, so it doesn’t write anything on the USB device, only loads stuff from it (only reading, no writing).

If I install things just to test out stuff, or make a mistake and want to return to how it was before, I simply don’t save the persistence changes.
Changes can be saved at any moment (using the persist-save program) or when I am going to shutdown (I am asked if I want to save the changes).
The only disadvantage this persistence method has is losing all new changes if the system breaks suddenly, or the laptop loses power. The best practice for me is to sync my most important folder to the hard-drive (in my case, an encrypted folder that I manually mount) so the worst case scenario is me copying the changes back to the USB (I will be testing out syncing that folder with a cloud service in the future, so I can be device independent).

When changes are too big for RAM, I remaster and start a new rootfs file (automatic after using antiX’s fantastic remastering program).

antiX Live system enthusiast.
General Live Boot Parameters for antiX.

[Moderator christophe]

Helpful

Up

0

::

On the subject of dynamic persistence — I sometimes use NO PERSISTENCE — once I get the system setup as I like it. If I want to change something, or perform updates, then I can still do it, but I have to immediately remaster.

Or we can use home persistence — I use it the same way, but tiny cosmetic changes are easier.

But mostly, I use static. The speed is good (to me, in my experience), and it keeps my memory available for normal memory tasks. (But no-persistence works like this, too, you just must remaster if you want to save changes.)

Just more thoughts…

confirmed antiX frugaler, since 2019

[Member rayluo]

Helpful

Up

0

::

Thanks for the valuable inputs so far. I want to clarify that this topic is not directly relevant to persistence mode, although a (dynamic) root persistence will likely expose that underlying problem more easily, because the computer has less available RAM to be used as RAM, thus the swapping (a.k.a. paging) kicks in earlier. Personally, I’ve used (dynamic) root persistence for quite a while, and recently switch to NO PERSISTENCE (because I already “get my system setup as I like it”). In either case, I observe the same “CPU load icon in systray is largely red” symptom.

christophe wrote:

I can’t address your situation in full, but I can share some tips on running live. Things I always do:

1. NO swap FILE usage (whenever you remaster or add persistence files is when the system asks this, I think).

To be clear, I did NOT setup my swap file on my LiveUSB disk. FYI: The LiveUSB maker did offer that option (perhaps even as the default option), but I said “hell no!” I setup my swap partition and swap files on hard disk.

So, the question remains “why/when/what the liveUSB (no persistence) session would write something to liveUSB disk, presumably when under memory starvation situation”.

christophe wrote:

2. use ZRAM for swap — https://mxlinux.org/wiki/other/zram/

Hope this helps some.

This might still not answer that question. But it seems an interesting idea in itself. I’ll check that out. Thanks!

[Forum Admin dolphin_oracle]

Helpful

Up

0

::

I would hazard that you are running shy on ram and the live system is using a swap file (create by default on persistence is enabled, although you can say no) on the usb device. modern browser’s are ridiculously ram hungry. so multiple tabs with a video running well may have your machine going to swap, even if you haven’t maxed your ram quite yet.

that is a guess, since I don’t know exactly how your live-usb is set up.

[Member rayluo]

Helpful

Up

0

::

— Duplicated post deleted —

[Member rayluo]

Helpful

Up

0

::

— Duplicated post deleted —

[Member rayluo]

Helpful

Up

0

::

My following reply was posted earlier today, but it was not shown up. I tried re-post immediately, it said “duplicated post was detected”. – SMH – One more try here. – Two more.

Thanks for the valuable inputs so far. I want to clarify that this topic is not relevant to persistence mode, although a (dynamic) root persistence will likely expose that underlying problem more easily, because then the computer would have less available RAM to be used as … RAM, thus the swapping (a.k.a. paging) kicks in earlier. Personally, I’ve recently switched from (dynamic) root persistence to NO PERSISTENCE (because I already “get my system setup as I like it”). In either case, I observe the same “CPU load icon in systray is largely red” symptom.

christophe wrote:

I can’t address your situation in full, but I can share some tips on running live. Things I always do:

1. NO swap FILE usage (whenever you remaster or add persistence files is when the system asks this, I think).

To be clear, I did NOT setup my swap file on my LiveUSB disk. FYI: The LiveUSB maker did offer that option (perhaps even as the default option), but I said “hell no!” I setup my swap partition and swap files on hard disk.

So, the question remains “why/when/what the liveUSB (no persistence) session would write something to liveUSB disk, presumably when under memory starvation situation”.

christophe wrote:

2. use ZRAM for swap — https://mxlinux.org/wiki/other/zram/

Hope this helps some.

This might still not answer that question. But it seems an interesting idea in itself. I’ll check that out. Thanks!

dolphin_oracle wrote:

I would hazard that you are running shy on ram and the live system is using a swap file (create by default on persistence is enabled, although you can say no) on the usb device. modern browser’s are ridiculously ram hungry. so multiple tabs with a video running well may have your machine going to swap, even if you haven’t maxed your ram quite yet.

that is a guess, since I don’t know exactly how your live-usb is set up.

Thanks for your reply, Dolphin! As mentioned earlier in this post, I did say “no” when I created persistence on usb device. And I did not even use persistence mode recently.

I have a follow-up question. Does that OverlayFS on “/” build (partially) on top of the usb device file system? I do observe its size in my system is close to my usb device’s remaining disk space.

demo@antix1:~ (master *)$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         1.6G  469M  1.1G  30% /
/dev/sdb2       3.8G  1.4G  2.3G  38% /live/boot-dev
/dev/loop0     1013M 1013M     0 100% /live/linux
tmpfs           1.6G  469M  1.1G  30% /live/aufs-ram
tmpfs            10M     0   10M   0% /media
tmpfs           201M  1.1M  200M   1% /run
tmpfs            10M  6.3M  3.8M  63% /live
tmpfs          1003M   15M  989M   2% /tmp

And then, indeed browsers are memory hungry. Besides, now I notice that Firefox would write and maintain rougly 300MB of cache and data into ~/.mozilla and ~/.cache.

demo@antix1:~ (master *)$ du -h -s ~/.cache/ ~/.mozilla/
246M    /home/demo/.cache/
56M     /home/demo/.mozilla/

Perhaps I can try to “ln -s” them to /tmp and see how it pan out.

[Member ModdIt]

Helpful

Up

0

::

Hi Rayluo, I sometimes get upset about firefox, prefer palemoon as it is more efficient and
user settings are not remotely changed as is my experience with mozilla. You can do some things
to reduce what is saved. Some of my sticks are pretty small so I do a lot to keep live trimmed.

56M /home/demo/.mozilla/
You can delete files in crashes, I never saw a crash but data was written, make folder read only, same goes for minidumps and saved telemetry pings. crash reports and telemetry are disabled with a policy file, enterprise policy generator extension is good for that as policy is .js.

Open 264 will be automagicaly downloaded and put in folder gmp-gmpopenh264.
I deleted and made folder read only, if it does not stay that way will make tiny dummy codec.

246M /home/demo/.cache/
Some of that will be the safebrowsing lists in cache, maybe as in my case old data too, interesting as I have removed safebrowsing setting urls to 0.0.0.0 seems a fallback may be compiled in.

[Forum Admin dolphin_oracle]

Helpful

Up

0

::

if you aren’t running with persistence, nothing is ever written to the usb except on shutdown/reboot, when certain configuration files are saved.

however, even if your swap is on your harddisk (that’s where mine is too), you could still see a pretty big slowdown. the live-usb WILL use the hard disk swap if its present.

any changes made to the running filesystem would be stored in RAM, which would eventually max out ram. I’ve done this while making videos…my videos were saving to ram…

to answer your question, the overlayfs IS the running filesystem. the /dev/loop0 is the mount back to the linuxfs/squashfs device. when running persistence, there are a couple more layers, but even just running without persistence uses overlayfs to mount make the read-only linuxfs squashfs filesystem on the live-usb accessible to the end user.

we’ve talked a lot about USB writes, but I think you may want to check how much RAM you are actually using when things slow down. look at “htop” for a easy view, or “free” to see a numerical chart.

[Member olsztyn]

Helpful

Up

0

::

dolphin_oracle wrote:

f you aren’t running with persistence, nothing is ever written to the usb except on shutdown/reboot, when certain configuration files are saved.

Just to add, IMHO this also applies (of course) when running with persistence (root) but not saving changes, such as running with option 3 (Manual), when changes are saved when you explicitly request.
Assuming Live USB antiX instance is on flash drive then minimizing writes to Live USB stick is of serious importance as frequent writes do deteriorate flash in the course of time to make them eventually unreliable.

Live antiX Boot Options (Previously posted by Xecure):
https://antixlinuxfan.miraheze.org/wiki/Table_of_antiX_Boot_Parameters

[Member rayluo]

Helpful

Up

0

::

olsztyn wrote:

dolphin_oracle wrote:

If you aren’t running with persistence, nothing is ever written to the usb except on shutdown/reboot, when certain configuration files are saved.

Just to add, IMHO this also applies (of course) when running with persistence (root) but not saving changes, such as running with option 3 (Manual), when changes are saved when you explicitly request.

Yes that is also my understanding too. Thanks for confirmation.

dolphin_oracle wrote:

however, even if your swap is on your harddisk (that’s where mine is too), you could still see a pretty big slowdown. the live-usb WILL use the hard disk swap if its present.

any changes made to the running filesystem would be stored in RAM, which would eventually max out ram. I’ve done this while making videos…my videos were saving to ram…

we’ve talked a lot about USB writes, but I think you may want to check how much RAM you are actually using when things slow down. look at “htop” for a easy view, or “free” to see a numerical chart.

Yes I know some of my machines are not necessarily a workhorse with tons of memory, and I know that when paging happens, a machine will be slowed down, regardless of where that swap sits.

But this question thread was NOT about “why my machine slows down when running out of memory”. The slow down was just a symptom typically happens when I CTRL+T to open a new tab, and such symptom led me to notice my USB thumbdrive’s light started flashing.

So, to rephrase my question: Given that “if you aren’t running with (static) persistence, nothing is ever written to the usb except on shutdown/reboot”, why the USB disk light was flashing, which indicates some I/O was happening on the USB drive? (Precisely speaking, a flashing light on USB disk does not necessarily mean it was being written, it could be just being read, however I was just opening a new tab from an already-running Firefox instance, so presumably nothing needs to be read, thus it must be writing something.)

FWIW, back in the days when I was using Windows XP on low RAM machines AND purposely disable its pagefile (the equivalent of Linux’s swap), the OS predictably emitted an “Out of memory” error when I tried to start a new application when the machine was already approaching its memory limit. But then, before/during/after that error message, the machine was performing normally without slowing down. (This actually gives me an idea of completely disable my swap right now, and see how it goes.)

olsztyn wrote:

Assuming Live USB antiX instance is on flash drive then minimizing writes to Live USB stick is of serious importance as frequent writes do deteriorate flash in the course of time to make them eventually unreliable.

Good point. In my case, I do not necessarily care about the longevity of my USB disk. They are cheap anyway. But a premature failure could cost theoretically lost of data. Yet my major concern is simply that, if such USB disk write was not intended in the first place, it might be a bug somewhere that antiX core developer(s) might want to hunt it down.

[Member rayluo]

Helpful

Up

0

::

— Duplicate post deleted —

[Author]

Posts