.dotfiles management, backup, deployment strategies…

Forum Forums General Software .dotfiles management, backup, deployment strategies…

This topic contains 42 replies, has 6 voices, and was last updated by skidoo Feb 17-2:25 am.

Viewing 15 posts - 16 through 30 (of 43 total)
  • Author
    Posts
  • #16950
    Member

    skidoo

    Several existing backup programs (e.g. backintime-qt) enable creation of multiple “tasks” (batchsets) which can be ticked/unticked prior to performing an overall “backup job”. You could create the backup store from master… and later (having copied the backintime jobs config onto the fresh system), mount the volume contiaing the backup store, launch backintime, and “restore”. Alternatively, and perhaps preferably, just connect to the backup store from the fresh system and use meld “directory comparison” to accomplish a merge (with opportunity to “diff” individual pairs of files ~~ useful when “restoring” networking config files).

    #16982
    Member

    skidoo

    The following produces a file containing a list of current vs stock//as-issued installed packages.

    grep ‘.’ /usr/share/antiX/installed-packages.txt | cut -d’ ‘ -f1 |sort>/tmp/alloriginal.txt
    apt-mark showmanual |sort> /tmp/currentmanual.txt
    comm -1 -3 /tmp/alloriginal.txt /tmp/currentmanual.txt >/tmp/all_newmanual.txt

    If the list is fed to a script launched from the new, stock, system to perfom automated
    apt update && dist-upgrade && apt install pkg1 [..] pkgNN
    commandline arguments can be supplied so that dependent packages are installed (as needed, without requiring exact “order” of listed items…

    …but if months have elapsed, some discrepancies may have crept in (diverted packages, dummy/transitional pkgs replaced during the interim), necessitating special handling.

    Some of the packages beg individual handling (debconf post-install questions, as seen during an interactive install operation), unless you’re content using an apt commandline option (can’t recall which one, offhand) to specify “just automatically answer Yes to everything, accept all config defaults”.

    – – – – –

    It’s a bit of a reach for me to anticipate all the “stuff” you’re dealing with in deploying to disparate laptops. Locally, I haven’t been providing support for any laptop-installed systems and, across our various tower/desktop machines, I can’t even recall the last time I had to track down & install a wierd graphics driver or kernel module. One box (Audigy VBQwhater sound card) is missing a sound driver, but oh well (too bad, so sad).

    For now, I’m burying the the prospect of using a FileNotes GUI and creating day-to-day entries toward maintaining a “packup” list.

    When sitting at a brand new system & wondering “hmm, what else is missing, vs old system”,
    I can mount the drive containing my non-incremental oldsystem backupsets and launch kdiff3-qt
    and perform “directory merge -} simulate” to generate a detailed “differences list” or manually perform diff/merge of any specific config file(s).

    #16989
    Member

    BobC

    Thanks skidoo. I’ve got the process of identifying the packages that need to be added taken care of. I delete so few they are unimportant. As far as the packages, I still need to take that list and get them all installed. I think the right answer is to tell it to install all of them in one string so that it can decide better what to do in what sequence as far as dependencies, etc

    sudo apt-get -y install $(awk ‘{print $1’} pkglist)
    or
    xargs sudo apt-get -y install < pkglist
    or
    sudo apt-get install $(cat pkglist)

    I will test that today.

    For the dotfiles, I will start with what Aptik grabbed by default, and add to it. I am hoping that I’ve been consistently making copies of all non-home directory changes, but sometimes the programs make changes in other places without me knowing about it.

    #17181
    Member

    BobC

    Allrighty then… I’m no linux coder, so I took the simple route of:

    1. creating a script to run on the modified system to see what was added and removed – pkgs.1.mod.sh
    2. move the files created and the script to the new standard system.
    3. and a script to run on the new standard system to be “upgraded” – pkgs.2.std.sh
    4. run sudo apt-get -y install \$(awk ‘{print \$1}’ pkgs.add.txt) to add the packages that need to be added
    5. run sudo apt-get -y purge \$(awk ‘{print \$1}’ pkgs.rem.list)

    I can see that I still have bugs, but its getting close. I need to figure out why it thinks I manually installed all these lib’s, LOL. I will wait till after the scripts work till I post them.

    Thanks for all the help…

    #17186
    Member

    skidoo

    it thinks I manually installed all these

    “manually (explicitly) installed”

    When you (or your scripted operation) perform an apt-get install _____ packagename operation, explicitly naming a given package, its status will be marked “manually installed”. Similarly, on Day1 when booting the stock distribution, most of the pre-installed packages bear that same status (manually installed) b/c they were explicitly listed during the scripted build-iso operation.

    #17228
    Member

    BobC

    Yes, I noticed that and will be extracting a list of automatically installed packages from the modified system and pruning those out of my install list later on.

    I do still have some that I don’t recall installing manually, and it turns out they were installed because they were needed for ksnakeduel which drags in a ton of kde stuff.

    Oh my! it made a mess. I ran my add list and it tried to remove xserver.xorg!!!

    • This reply was modified 3 weeks, 5 days ago by BobC.
    #17438
    Forum Admin
    Dave
    Dave

    Attached is the OLD script as mentioned previously that probably will not work anymore but….

    Attachments:
    1. backup.tar.gz

    Computers are like air conditioners. They work fine until you start opening Windows. ~Author Unknown

    #17578
    Member

    BobC

    Thanks for posting that, Dave. I got the Buster testing system running, and got my additional apps identified and loaded, so the dotfiles are next. By then I am hoping to be able to do it on a Frugal system and create a live USB from it, which I will then test by loading it on a machine my son gave me and an SSD drive I had left over.

    #18488
    Member

    BobC

    I got a default system loaded. I got my packages to load (I ended up creating a list, then removing things from it, down to what I actually wanted to install).

    I found that Aptik-Gtk is trying to help, but not perfect.

    I would need to individually list all files added or changed outside of /home for it to save and restore those, so it seems my ~/changes folder is easier anyway. I will continue my ~/changes method of tracking what I changed, and it will get saved and restored in /home, and then I’ll need to copy the changes into place, which isn’t perfect, but not too bad.

    I will go back thru the thread and see if I see any easier way…

    #18489
    Member
    noClue
    noClue

    Didn’t read all the posts, sorry if it got already mentioned or if it’s unsuitable from some other reason.

    How about FreeFileSync?

    You would have to include/exclude files or folders manually on the first use.
    Backing up or restoring them later would be 1-click job.
    Different jobs can be saved.
    If you make every backup in another folder, you get some sort of versioning.

    Only problem with it would be, the program doesn’t know about newly created dot-files.
    That’s obviously not a problem if you are backing up complete folders.

    How strange is to be anything at all. (Alice in Wonderland)

    #18498
    Member

    BobC

    AMAZING!!!

    It all worked on pretty much the first try.

    I used my pkgcomp scripts on the old HP DV9917 system and new Dell M2400. Ran aptik-gtk backup for users, groups, and home, put all that into the live-usb/home/demo folder on the flashdrive. Installed on the new pc, ran apt-get update, ran the 2nd half of the pkgcomp, edited the list for what i wanted of the missing ones, installed that with apt-get install line generated, installled aptik and aptik-gtk with apt, ran aptik-gtk as root, did the restore of the same things, did a reboot from terminal, and all is well. just need to reimplement the few program and config changes outside of /home, and i’ll be done.

    That was much, much better than last time.

    I even got s2both working for hybrid suspend and hibernate.

    I got all the program changes an such migrated. I think the key is to change as little as possible outside of /home, and not to install anything outside the normal packages from that repo unless its really worth the ongoing effort. I removed about half of my changes as a result, and hopefully that will make the process easier in future. I really liked the Tilde command line editor, but the effort to tweak and install it was just too much.

    PS: I went back to LXTerminal as my default terminal, and now have Midnight Commander working with Geany as its external editor, and have it making automatic backup copies of everything I edit. I don’t think I could ask for a better setup than that, and that cut my number of modifications to the default system dramatically.

    My next task is to reverse the process, and reload the other 1/2 of the HP Laptop’s SSD the same way from the new machine and see if it isn’t both reliable and easy by comparison, and once I’m happy with the way it runs, I’ll clear the original side of the SSD to load Buster onto.

    • This reply was modified 1 week, 3 days ago by BobC.
    • This reply was modified 1 week, 2 days ago by BobC.
    #18538
    Member

    BobC

    So, a procedure to save and restore to a different machine
    1. save all non /home/yourself changes in a directory structure ~/changes-hostname-yymmdd with each file in the relative folder underneath where it was added or change with all the attributes it had as well. if the file is literally machine dependent, make a 2nd copy of the file suffixed by the hostname so you don’t forget
    2. don’t load things that are difficult to reload, at minimum, if no .deb, forget it if possible
    3. limit what you put in /home. put downloaded data file stuff somewhere else and back it up/restore it separately
    4. run the packagecomp #1 modified pc script (pkgs1.mod.sh) on the machine being saved to make a list of what was installed
    5. get the aptik .debs and install them to do the saves. save to a place outside of your own /home because you want it to backup your /home. I just save users, groups, and /home with it. I take the folder and copy it to the live-usb folder on a freshly made and tested antiX flashdrive, as well as the aptik .deb files and packagecomp folder
    6. load the new machine, connect to the net, install antiX saving live changes. make sure you have all the live usb stuff copied off the flashdrive to somewhere outside of your /home, and then you can reboot into the stock antiX system and make sure its ok before loading all your changes.
    7. run the packagecomp #2 standard pc script (pkgs2.std.sh) on the freshly installed machine to find out what packages need to be added. When the editor comes up, remove things you don’t need. The script will make a list for you of what you decided to skip, just in case.
    8. run the apt-get update and apt-get install with command line provided
    9. use apt to install the aptik .debs
    10. run aptik-gtk from the run box as root
    11. restore only the things saved above
    12. overlay the changes from your ~/changes-hostname-yymmdd which got restored by aptik
    13. update-grub
    14. reboot, and hopefully welcome to a totally familiar system with everything just as you left it, but running on a shiny new system (LOL, or old clunker for the kids)

    It took about 3 hours to do, not bad for a modified system.

    I added a zip file of the scripts

    • This reply was modified 1 week, 1 day ago by BobC.
    Attachments:
    1. packagecomp.zip
    #18546
    Member

    skidoo

    “don’t load things that are difficult to reload, at minimum, if no .deb, forget it if possible”
    Aw, that would sink my ship.
    I’ll just say I’m glad to hear you’ve found a timesaving workflow that suits you.

    How about FreeFileSync?

    Have you used it on an antiX system? If so, which version of FreeFileSync?
    I had tested v9.3 on antiX17(stretch) a while back, and it was ill-behaved.

    #18549
    Member

    BobC

    It was more of a reminder to myself that for every time that I violate that rule it will make trouble on every migration.

    If I had many worthy programs that were cobbled or tweaked by myself, I think the best solution would be keep all the sources off somewhere else, and have an automated sync routine to bring them in.

    The freeSync idea didn’t come up till the 11th hour and I opted to minimize the issue to where it was a 15 minute task doing it by hand, and leaving the idea of automating the post install, post configuration, resync part of the task list as a separate, smaller problem that I could look at, solve and test separately.

    To be honest, my main laptop was desperately crippled, with apt wanting me to download and install 3/4 of a gb to fix it, and IF that had worked, it still would have the nVidia mess, so that plus the loading of the new Dell m2400 and Alienware machines forced me to get an efficient migration method working, if I wanted to be running machines setup to my liking.

    LOL, I bet you have 50 or 100 programs like that, not one or two like me…

    I really liked using Tilde, as it was as perfect of a command line editor as I could ask for, but getting Geany working from midnight commander as it’s editor and making good backups of changes made Tilde expendable.

    The same went for mrxvt-mini. I really liked it, and it was super efficient, but not worth spending hours on an old unsupported program, when LXTerminal works well, even if takes a lot more memory.

    Thanks much for everyone’s help 😁

    #18550
    Member

    BobC

    Aptik has the ability to restore a list of individual files (I forget if it could do directories too) at the end, but I didn’t try it. Maybe that list could be generated for it from my changes folder, if not, the folder maybe could be used directly via a sync routine. Dave and bitjam had also provided some ideas, as well as the one you had, and it would be better to really think it through and find a good long term solution rather than do something hasty that would cause trouble again later.

    Just thinking aloud. I always like breaking big problems into smaller ones and solving them one at a time.

Viewing 15 posts - 16 through 30 (of 43 total)

You must be logged in to reply to this topic.