.dotfiles management, backup, deployment strategies…

Forum Forums General Software .dotfiles management, backup, deployment strategies…

This topic contains 43 replies, has 6 voices, and was last updated by BobC Feb 19-11:24 pm.

Viewing 15 posts - 1 through 15 (of 44 total)
  • Author
    Posts
  • #15189
    Member
    Avatar
    skidoo

    https://www.antixforum.com/forums/topic/improvements-and-looks/page/5/#post-15161
    If anyone is doing dotfile management, could you fill me in please on how you are doing it? I need to be able to save and reimplement my changes easily, somehow. Most of the time all my machines can be configured the same (assuming I freeze or stop working with the 486-686 ones). Git has at least a partial solution it seems, but I don’t have the big picture on it, because I think I need a 3 step process, where I install default, then add packages, then fix or overlay the default configuration files.

    some debian packages for keyword “dotfile”: restic, homesick, yadm
    These typically utilize rsync (and/or git) as the backup/restore mechanisms. restic (to me) seems meritworthy interesting, but I’m not tempted to wade through its documentation.
    man rsync
    Because rsync is ubiquitous, that has been my preferred tool. I’m not keen on the prospect of wrestling with git + proprietary layers ~~ overkill “solutions” to my tiny backup jobs.

    .
    These (dotfiles management utilities) are also available as debian packages:
    stow and xstow
    xstow claims more features (do I need ’em) but provides a comparatively terse, and sucky, manpage.
    stow, in contrast, its package installs both a multipage html manual file:///usr/share/doc/stow/manual-split/Index.html and a pdf version of its manual.
    -=-
    stow is often mentioned in linux subreddit discussions. A few weeks ago I bookmarked an explanatory article which provides a good overview of what stow is/does: managing-dotfiles-with-gnu-stow

    ^—- If I need to “list”, create a manifest, of stuff to be backed up… seems like I might as well just maintain rsync include/exclude lists

    Tip: You can search the linux subreddit to find a treasure trove of recommendations (and custom scripts) regarding .dotfiles management, backup, deployment strategies…

    .
    some debian packages for “deploy”: ansible, boxer, fabric, rex
    Post-installation, “apropos rex” shows 60 (sixty!) separate manpages are available.
    Suggests to me that the learning curve + time invested would be more than the time saved via automation!
    [edit: removed other cloud//deployment//orchestration tools I had name-dropped here]

    .
    debian-packaged “backup, with a GUI” tools that I’ve used and recommend: ukopp and backintime-qt

    I think I need a 3 step process, where I install default, then add packages, then fix or overlay the default configuration files

    What exactly is the goal? Need to hear details of that in order to sort out the best tools for the task.
    (edit: you did state a goal. I just didn’t “hear” it clearly.)

    some “backitup and go” cookie-cutter tools: systemback, timeshift, aptik
    http://forum.mxlinux.org/viewtopic.php?f=104&t=46289
    FWIW, aptik was buggy when I tested it on MX, and again when tested on antiX17
    .

    • This topic was modified 6 months, 1 week ago by skidoo.
    #15195
    Forum Admin
    dolphin_oracle
    dolphin_oracle

    I keep a backup of the base configs of any application I care about, in my case my gpg keys, git, pbuilder, filezilla, geany, and a few other apps configs. These are stored is a separate location and I have a script to copy in the configs into my current home folder.

    I have heard of some folks even using git to store this type of thing.

    #15279
    Member
    Avatar
    BobC

    Thanks for looking at it and posting your ideas, skidoo and DO.

    Aptik looks like at least its possibly able to do some of it.

    I have been using a changes folder off my home folder to keep copies of all changes that I make outside of my home folder, and then use homegrown rsync script to backup the home folder excluding things like cache and trash etc, but last time it took a weekend to redeploy in a newer version due to all the changes.

    Maybe I could install frugal with persistence, then use a script to install missing packages and copy in my software changes, then a git branch solution to be able to bring in my hardware config files for that branch, and then bring in my software config files, and copy all of them into place? Once it is known good, I could then copy the persistence changes to files for that OS version, to save a view of all changes combined for that system.

    Would it make sense to also create an ISO for it at that point and install, or just run it from frugal?

    #15287
    Forum Admin
    dolphin_oracle
    dolphin_oracle

    I’m fairly certain you can make a snapshot from frugal (I know you can from a live-USB).

    if all that was done on a live-USB, you wouldn’t even need snapshot, just install directly, perserving home folder changes.

    #15576
    Member
    Avatar
    skidoo

    @BobC, lookit: https://www.antixforum.com/forums/topic/synch-between-live-usb-and-installed-system/
    Coincidence? Serendipity?

    Betcha we can cobble an “injector” tool.
    Feed it an ongoing list of files which have been added/updated…

    …click a button and it alerts you to any files (or paths) that match patterns in the exclude(s) files (so will wind up being skipped during persist-save), and packs, then copies into place an xtra.tgz file.

    #15637
    Member
    Avatar
    BobC

    I got Aptik-Gtk and Aptik working with version 18.8. I did the backup side successfully.

    I have all kinds of questions in my mind as to the problems a full restore would cause, but at least I got it installed and run. I will have to think thru the restore and try it on a test machine.

    BTW, in case anyone else is trying this, the way to install it seems to be to

    install gdebi,
    download the deb for aptik and aptik-gtk from the releases tab of the git sites,
    use gdebi to install each, and after each run apt-get -f install packagename.
    Afterwards, you need to go find the aptik-gtk desktop file and change the exec to gksu aptik-gtk so it runs as root
    go to control centre in maintenance tab and add it to the menu.

    I’m fairly certain you can make a snapshot from frugal (I know you can from a live-USB).

    if all that was done on a live-USB, you wouldn’t even need snapshot, just install directly, perserving home folder changes.

    DO, yes I use the snapshot to make full backups, but end up with a weekend to a week of effort if I install a new OS version and want to get all my additions and tweaks installed and running.

    skidoo, yes, there looks to be a bunch of what I am asking for there, ie the idea of the frugal install and overlay of changes that could be versioned and saved, and therefore easy to get back to a save point from those persistence files if a calamity occurred, and only needing to save changes from base, not everything. I need to reread all that about 6 times and test, test, test

    • This reply was modified 6 months ago by BobC.
    #15788
    Forum Admin
    Dave
    Dave

    DO, yes I use the snapshot to make full backups, but end up with a weekend to a week of effort if I install a new OS version and want to get all my additions and tweaks installed and running.

    Hmm I used to do the same and remember noticing this “problem”. At that time (which was several releases ago) I took the snapshot tool and the live usb tool and merged them together. Essentially the new script I made would make a snapshot of the root system but not build the iso… it modified the fstab in the snapshot to make the home directory mount a second partition on the usb. Then loaded the “snapshot” (unpacked iso) to partition 1 of the usb and made it live. Then rsynced the contents of the home directory to partition 2 of the live usb. Then I had a live usb clone of the installed system. This was done every week end. Throughout the week it would only rsync the home directory as the root directory rarely changed (aside from logs/configs). Will see if I can dig it up. It probably will not work straight away and will need to be modified for the newer versions but…

    Computers are like air conditioners. They work fine until you start opening Windows. ~Author Unknown

    #15889
    Member
    Avatar
    skidoo

    need to be able to save and reimplement my changes easily, somehow. Most of the time all my machines can be configured the same
    [..]
    I think I need a 3 step process, where I install default, then add packages, then fix or overlay the default configuration files.

    For the “add packages” step, instead of “injecting” it’s probably best (safest, easiset) to just keep a list of “I want to add ABC packages, and remove XYZ packages” and run an automated “sudo apt install aa bb cc dd && sudo apt purge xx yy zz” script from within live session. Otherwise, we’re unnecessarily burdended by the chore of tracking which dependent packages got pulled in by XX so will need to be injected before XX is injected.

    Toward generating a scripted picklist of “packages I probably want to add” candidates:

    grep ‘.’ /usr/share/antiX/installed-packages.txt | cut -d’ ‘ -f1 |sort>/tmp/alloriginal.txt
    apt-mark showmanual>> /tmp/currentmanual.txt
    comm -1 -3 /tmp/alloriginal.txt /tmp/currentmanual.txt >/tmp/all_newmanual.txt

    “installed-packages.txt” reflects the stock distribution. If the target LiveUSB is a remastered system, prior to diffing against the now-current(installed) system you would need to (apt-mark showmanual>>/live/boot-dev/somefile.txt) generate a fresh manifest ~~ so that you can easily retrieve it when mounting the pendrive from the now-current(installed) system.

    #15891
    Member
    Avatar
    BobC

    Thanks, skidoo, I will give that a try… It sounds like it might cover the question of getting the desired top level packages installed easily.

    Afterwards I will copy in my /home files (possibly from aptik-gtk) and then overlay in the changes to other areas from /home where I keep a copy of those.

    I have two “new to me” i7 laptops needing better operating systems 🙂 They were free and I didn’t know that they actually were in working condition, so they will make excellent “Upgrade” candidates. Upgrading is Mandatory here (a Dr. Who thing)…

    #15927
    Member
    Avatar
    BobC

    I didn’t know about apt-mark, thanks.

    PS: I found that it thinks a number of “standard” antix installed packages were installed manually, so I ran it against a fresh install and will compare that to my modified install, and the differences will be the manually installed ones that I installed myself.

    Once I get that out of it, I’ll just need a little script to read that list and install all those packages.

    • This reply was modified 6 months ago by BobC.
    #15937
    Member
    Avatar
    BobC

    Dave said: Hmm I used to do the same and remember noticing this “problem”. At that time (which was several releases ago) I took the snapshot tool and the live usb tool and merged them together. Essentially the new script I made would make a snapshot of the root system but not build the iso… it modified the fstab in the snapshot to make the home directory mount a second partition on the usb. Then loaded the “snapshot” (unpacked iso) to partition 1 of the usb and made it live. Then rsynced the contents of the home directory to partition 2 of the live usb. Then I had a live usb clone of the installed system. This was done every week end. Throughout the week it would only rsync the home directory as the root directory rarely changed (aside from logs/configs). Will see if I can dig it up. It probably will not work straight away and will need to be modified for the newer versions but…

    Dave, I would be interested to see how you did the rsync stuff there, because maybe I could copy that to refresh configs not handled by aptik-gtk or in place of it if that gets messy.

    I’m making good progress at this point. If I can get it to where I can take a stock antiX install and duplicate my packages and tweaks reliably in a few hours, that would be great… The last couple of times it was all iffy and I would forget this or that and then have to remember how I solved the issues, which sort of discourages me from keeping up to date…

    Thanks 🙂

    #16604
    Member
    Avatar
    skidoo

    Bobc, I’m tinkering with repurposing the code from one of my other projects to serve as a “filepicker for packup/injection” GUI. I’m a bit stymied in choosing a name, and a scope, b/c the app’s framework was intended to serve as a multipurpose tool. As a working title, I was tempted to call it “Bob’s Items-of-Interest Thing”… or “thingamabob” as a shorthand moniker. (Heh heh)

    Day to day, as you recognize a file or group of files or entire directory that you would like to “inject”… you would add an entry for it within thingamabob. Later, when ready to “packup for injection”, you would SelectAll or select some but not all entries & click a button to autogenerate a *.tgz file containing those items. The app would warn you (and would reject) any files|paths which match a blacklisted pattern ~~ in other words, it won’t allow you to accidentally attempt overwriting any (/proc/*, /run/*, et cetera) critical files on the target system.

    Your usage described atop this topic mentioned “injecting/customizing a stock distribution”. For me, that’s such an infrequent task that I’m compelled to target a broader scope of usage. Also, I’m somewhat reluctant to FULLY automate the injection operation; as often as not, I would want to diff a given file pair rather than blindly overwriting the targeted file. With that in mind, I intend to have thingamabob pack (presumably on a known-good-working system) the *.tgz but leave it up to me to: 1) ensure the desired target device is attached, and 2) copy the *.tgz into place in my choice of Live-USB-Storage/username/ directory or base directory of the liveboot, or wherever. Later, when a copy of thingamabob is launched during livesession, on-demand we would have the opportunity to unpack the tgz in order to achieve an update ~~ or, a given run might be intended to extract a tgz which was created right on the same live system while in a known-good state, in order to achieve a “rollback”. Although remastering already proves quick/painless ability to “rollback”, an archived (manually renamed and timesstamped) series of saved linuxfs and/or rootfs files consumes considerable storage space. The thingamabob (re)injection could provide a complementary, lightweight, type of rollback. strikeout = Thinking aloud. TMI, too much info.

    .
    .
    This overly wide screenshot looks blurred to me when squished to fit forum layout.
    So I’m including a link to view the image directly: h t t p s://vgy.me/9FHViV.png
    h t t p s://vgy.me/9FHViV.png
    .

    The repurposed code for this app is pygtk (python2.7 + gtk2). I probably lack the chops and/or time-n-patience to rewrite it as “python3 + shinyNewerGUItoolkit”. I’ve pared teats-on-a-bull features from a framework I’ve used for various bespoke apps and have stuffed code from various modules into a single python script. Its datastore is a plaintext flatfile ~~ for better and/or worse, no database engine required.

    I welcome your (BobC) feedback regarding “packup” functionality; your feedback would help steer the “design decisions”.
    The listview widget provides “click on column headers to sort-by-column” functionality.
    Ability to sort has been sorely absent when tweaking my include/exclude lists which are peppered with inline notes.

    The datastore can include columns which aren’t necessarily (or not always) displayed, and I’m planning to add hidden-by-default columns for DateAdded and DateModified. That metadata, if sortable, would certainly help jog my recollection of what I was doing/thinking at or near a given time (in case I’m rushed and skip the step of typing a description/rationale note).

    (not evident in the screencap) right-clicking the list raises an Add/Edit popup containing data entry fields. Another “vector” I would hope to use, day to day, in building the list is a spaceFM custom command (send this pathstring to thingamabob). For an already listed exact pathname, this would trigger an Edit dialog; otherwise an Add dialog bearing a pre-populated Name(pathstring) field. At that point, we have opportunity to edit the pathstring (to create, for instance, an rsync globbed “include” pattern which targets /this/path/*.txt for migration vs only one exact/specific file).

    One of the fields is a multi-line freeform notes field. This is (has been, can be) a handy place to keep notes regarding “what does this file do? which package placed this file, or references this file… or did I place it manually? What manpage or webpage(s) were helpful toward understanding the syntax used in this config file?” Also, when editing a config file, in addition to creating a .bak copy, I’ve been inclined to jot a note-to-self mentioning (or pasting in) any config lines I’ve altered. In a FileNotes capacity, thingamabob would enable me to pack-n-go take my notes with me. NO, I don’t care “whatif the same annotated file doesn’t exist on another system”. If I spent an hour researching resolve.conf and DNS in/outs and created a note, the note’s content will probably still be useful (and searchable).

    spaceFM both provides ability to assign (DesignMode) custom context menu actions, AND it broadcasts to a socket ~~ announces the pwd pathstring each time the path within the currently focused tab changes. Now we’re cookin’ with gas! Thingamabob could run, minimized to tray, listening to that spaceFM socket… (optionally) popping up and alerting “Hey, we gots notes for one or more files in this directory! Wanna see ’em?”

    The datastore will include an enabled/disabled column. By default, each item (ultimately, each is a rsync inclusion pattern) would be set enabled; toggleable via the Edit popup for an individual item or (TBD) by using a hotkey to toggle the status of the currently focused row. The (TBD, not yet prototyped) bottom pane of the GUI will expose a SelectAll/DeselectAll button, along with various other command buttons. Which and whatall commands? Hopefully you can help me decide whatall.

    Kicking the tires, I’ve considered the prospect of adding columns like “est. filesize” (calculated WHEN? when added? each time the program runs? just prior to an xtra.tgz being populated?) and/or “category” and/or “keyword”… but, really, I wouldn’t have the patience to consistently fill-in all columns for each item record. These optional details can just be typed (or not) into the freeform notes field (will still be searchable/filterable).

    Multi-purpose?
    (don’t put the cart in front of the horse, eh)
    Contrary to the mantra “do one small thing and do it well”, thingamabob could accept drag-drop from browser urlbar and (also) serve as a browser-agnostic bookmark manager. Its “packup routine” can easily filter/ignore any web urls. By leaving the file:// scheme prefix after dragging to add an InterestingItem (scenario, its a local *.html helpdoc) and jotting a note, its row represents “just a uri+searchable note”. By removing the prefix and leaving only /filepath/to/helpdoc.html, it becomes a slated-for-packup InterestingItem. A bottom pane button could expose a “ShowAll||ShowOnly /*” toggle.

    #16651
    Member
    Avatar
    BobC

    You get 3 points for your sense of humor in naming it…

    Now to try to figure out what it does…

    So, its like a list of select/exclude criteria? with a matching optional description column for each?

    That you could feed to a backup or restore program, or basically any other program needing to be fed a list of files?

    PS: read more fully… ahhh, drag & drop, with popup edit on add, very cool. Ideas: ability to group, bookmark idea is great, but doesn’t make sense to mix bookmarks of web pages with file directories to backup, could you add 8 character group and subgroup columns, and how about an integer rating column for sorting, and when you create records, default group to same value as the record you dropped on top of. Add ability to resort by group, and within group, and filter by group to make entries easy to find. Last clicked timestamp would be nice for possible pruning eventually.

    Guess what, now I can make a music playlist where I feed it a list of all the tracks and fill in the genre (ie Music, Blues, rating #1) and rating as I listen to them all, and then use that to select to play them:),
    or as a webpage list of Linux stuff and another subgroup of news web pages,
    or a backup list,
    or list of USB flash drives, etc

    • This reply was modified 5 months, 4 weeks ago by BobC.
    #16659
    Member
    Avatar
    skidoo

    The intended core functionality: a GUI for managing/generating a large “rsync includes file”.
    Notekeeping is secondary, but essential, because by next week I won’t remember WHY i added /some/curly/file.conf to the list.

    “could you add 8 character group and subgroup columns, and how about an integer rating column for sorting, and when you create records, default group to same value as the record you dropped on top of”

    I left php + MySQL back in, geez I can’t even the year… back around the “php v5.3+” days. The app is parsing a one-row-per-line plaintext datafile to populate the liststore. So… no, I can’t. Functionality afforded by relational db goodness is off my plate.

    re: Groups, subgroups
    You’re free to arbitrarily choose your own set of “keywords” or “category” monikers.
    (Tack your category list to the wall. The app probably won’t sport a “dropdown category selectbox”.)
    By selectively including keyword(s) into your choice ~~ description field, freenote field ~~ you’re able to immediately drill-down by typing (idunno, “cake”) into the search filter box. Filtering on “vanilla cake”, would product an even narrower resultset list. {—- The filter provides regex matching, not fancy ANDed queries. If the “vanilla cake” results can be construed to represent a “subgroup”, well that’s peachy. If not, umm, refer back to Rule #1 ?

    re: drag & drop
    Due to a goofy bug which has crept in subsequent to my “crammit all into a single .py file” Bright Idea, the gtk.treeview widget isn’t set to accommodate “natural sort” reordering via drag-n-drop. Because drag reordering is, arguably, only useful when managing a single-screen-tall list, I haven’t felt motivated to debug that (and document the multiple inherent gtk2 treeview widget quirks)

    Ready, set, gaw are you underwhelmed yet?
    “could [..] feed to a backup or restore program, or basically any other program needing to be fed a list of files”
    Check out the “backintime-qt” program. Thingamabob could just output the list, or a filtered set of rows, to textfile. Paste from the textfile into the backintime “Include” field, click Go. That’s leaves the rest of the weekend as actual free time. Well, not quite, but… (you had mentioned spending a weekend to accomplish the chore of injecting customizatons into a fresh installation.) I didn’t intend to depend backintime (or ukopp) into the workflow, but one could choose to do so.

    From thingamabob’s perspective, there’s no agruably “exclude” list. There’s just a “blacklist” of patterns we need to guard against when passing the list to rsync. The top pane of the GUI already sports a tabbed notebook widget (in the screencap, its tabs are set to hidden, and currently only one tab is populated). Called into view via menu entry or lower-pane button, thingamabob could parse the actual “excludes” file (comprised of entries which were initially harvested, manually, from the combined antiX-supplied list + my/your personal lists) and display into the GUI top-pane ~~ for quick access and/or infrequent editing. Six vs half-dozen, a thingamabob button click or menu action could just open the excludes list file in a text editor.

    “bookmark idea is great, but doesn’t make sense to mix bookmarks of web pages with file directories to backup”.
    Thanks, you’re right. The bulk of my archived “local helpdocs” reside on a separate mount anyhow, rather than the root filesystem of the flavor-of-the-moment OS instance.

    #16886
    Member
    Avatar
    BobC

    Yes, as usual, its much easier to come up with expansions to ideas than it is to actually create them and get them working right, but its also true that if you don’t look at possibilities from a wide angle view initially, the design doesn’t start with something where it could easily be expanded later.

    For me, at the moment, I really need to get a packages and dotfile save and deploy system working, because I have the 2 new machines, plus the fast Dell, and a tower that I would like to try it on.

    It needs to handle differences in dotfiles for different hardware, too, like the fast Dell having the difficult to deal with boot and graphics, and some laptops needing firmware for Wi-Fi.

    There is also the question of Do I load all as frugal from here out?, and the idea of using it to load and deploy buster first on one, then on others, and redeploy after a final buster version is out.

    I wish I had 1/2 your ability on this stuff… I keep learning, but its a painfully slow process because my brain is buried at work on old “stuff”…

Viewing 15 posts - 1 through 15 (of 44 total)

You must be logged in to reply to this topic.