lazaruk.com - Development

Moving to Linux, Issues and Lessons Learned

2024-07-05T00:00:00-06:00

Of course when moving from over 30 years of using Windows to a Linux system, I made a few mistakes and had to look up a few things.

Updates to the dock

In playing around with various dock extensions I found that my default dock had a delay of 3-5 seconds between an app being launched or closed and the icon being added or removed from the dock. I figured that since the Pop!_OS dock is a customised version of Dash to Dock, and I had installed that extension, I had probably corrupted something. Research led me here, which may have fixed the problem. I also went back to the gnome extensions site and disabled the “Ubuntu AppIndicators” extension, even though it says that extension does nothing and it seems to automatically re-enable itself. This action appeared to fix the problem but in hindsight that doesn’t really make sense to me and further testing would be required.

Partial loss of right-mouse click context menus

After a reboot one day I found that my right-mouse button was not working on the desktop or in the default file manager. It worked in my browser, other file managers, and most other applications. Initially I thought it was something in the gnome-tweaks application, and that resetting to default values had fixed it. But this was not the case. After more investigation I determined that the problem only existed on the non-primary monitor. I could right click fine on the primary monitor, no matter which of the panels was set to be the primary. But the others - no right click. The actual solution to this was to remove the monitor config files from ~/.config, then log out and in again. Sigh.

Suspend

Doesn’t work very well when the graphics are set to nVidia mode. Seems to work fine in integrated and hybrid mode, although then I wouldn’t be able to use the external monitors.

Internxt

I purchased a large block of space from Internxt to use for syncing, since my previous file sync solution was not supported on Linux. What a scam. Internxt is a scam. Internxt has been refusing to honour their money-back guarantee for half a year now, despite my first requests to them being well within the 30 day “warranty” window. The service sort of works for use cases that don’t help me, but doesn’t work at all for my use cases. I understand that I should have tested this before I made a financial investment with them, but I did realise the error and request a refund quickly. That they have refused to honour their refund policy is despicable. Buyer beware with Internxt - Internxt is a scam.

Watching the logs

journalctl -f

tracker-miner-fs-3

Had some troubles with this, which became visible with watching the journalctl log. Seemed that the database was corrupt. Some research showed that it was super risky and maybe impossible to remove the service. So I reset it, and the problems went away.

Get the status tracker3 status

Reset it tracker3 reset -s -r and then restart it tracker3 daemon --start

Note the “3” on the end of the executable.

Moving to Linux, Building my System

2024-07-05T00:00:00-06:00

Herein lies all the things I did to get my system to the state it is in today. Mostly this is a record so that I can re-create it if need be. But hey, it might also be helpful to someone.

Install the system. Just accept all the defaults from the installation media. Make sure to enable the disk encryption and record the passwords.
There are some great points here. I also probably am writing up this document because of being inspired by Willi. Thanks!
Install Timeshift to capture backups of the state of the system. This still needs more testing to determine if it’s useful.
NoteBook FanControl
1. I found that the fans were not coming on effectively and the system was a bit hot. Installing this app and setting it to the “HP Spectre x360 Convertible 15-ch0xx” config has fixed this up.
As part of my initial requirements to make the system look as much like Windows 11 as possible, to aid in my unconscious acceptance of the system, I’ve installed Dash to Panel. The configuration of it and the default dock are generally:
1. In the system settings:
  1. Disable the default dock in the Settings > Desktop > Dock.
  2. Toggle some settings for the Top Bar in Settings > Desktop > Desktop Options. Remove the Workspaces and Applications Buttons. The Super Key Action should continue to be set to Launcher. Window Controls should have both Minimize and Maximize Buttons enabled.
2. In Dash to Panel settings:
  1. The settings are backed up in ~/Documents and the differences between my settings and the defaults are not necessarily all captured below. Need to expand on this as further tweaks are identified.
  2. Position: remove the Applications button and set the Taskbar to Monitor Center.
  3. Style: Set App Icon Margin to 4. Tweak Animate hovering app icons settings.
  4. Running indicator: set focused app to Metro and unfocused apps to Dashes.
  5. Panel style: Override panel theme background opacity and set it to 50%, and enable the Dynamic background opacity.
File Syncing / Backup
1. This has been quite an adventure.
2. My preferred sync option, Proton, doesn’t have a Linux app for their cloud drive (as of early July 2024). I tried Internxt and have had more problems than success with it. At this point I’m taking advantage of my free Dropbox account and Mega account to do some syncing. I also purchased a lifetime 100GB account from Filen which is working great. So right now I’m basically leaving everything in the various locations they were in before I moved to Linux and if I ever need something that’s when I get it to the local system and find a new permanent home for it.
  1. The big advantage right now is that Mega and Filen allow me to sync the local directories without moving them. Internxt allows me to backup the directories in place (except for the errors with the backup service), but if I wanted to sync them I need to move the directories to the Internxt Drive directory, and then symlink from my home folder. I don’t really want to do that and have to manage it (plus, it doesn’t reliably work right now).
3. I also needed to capture game save information for the 2 or 3 games that I ever play. These were purchased through Steam for my Windows system. Thankfully they are functional on the Linux system with the Steam ProtonDB Windows emulation. To backup the game save information, I found the Windows filesystems in home/
  /.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/compatdata/ and added those to my sync client.
  1. I chose to not use Steam Cloud here because I just have too many save files right now. When I get those reduced to a volume that can reliably live in Steam Cloud then that’s the path I’ll change to.
4. I tried using Déjà Dup to create the backups and while it works well there isn’t a lot of reason to add another tool to the backup pipeline when Mega and Filen are syncing things just fine. I did try using Déjà Dup to run backups and store those in my Internxt Drive and let that sync up but that introduces a potential data loss between the time I change a file and the time of the next backup and sync. Again, since Mega and Filen sync in near real time there was no reason to use Déjà Dup here.
Adding launcher icons for things that don’t otherwise have them
1. For example, Anaconda.
2. Thankfully, Joe Lotz has a page for this. Thanks again Joe.
3. Just crete a desktop file in the applications directory: gedit ~/.local/share/applications/<application name>.desktop.
4. And then add data to the file:
```
[Desktop Entry]
Version=1.0
Type=Application
Name=<app name>
Exec=<path>
Icon=<path>
Terminal=false
```
1. Since I’ve only had to do this for Anaconda so far, I can make a list of the app icon locations fairly quickly:
  1. /home/<username>/anaconda3/lib/python3.7/site-packages/anaconda_navigator/app/icons/Icon1024.png
Getting the graphics to work

Pop!_OS has a few graphics options built in to the power menu. Initially I set mine to hybrid, thinking that it would be best to use the lower power integrated graphics as the default and then use the discrete GPU when necessary. Now, it became necessary when I was running some machine learning python code, which I was able to get running by adding the command __NV_PRIME_RENDER_OFFLOAD=1 <application>, per the support page.

But when I tried running switching between GPU enabled apps and integrated apps, especially when using the docking station, I found the system started having power problems and would inevitably freeze. Sometimes these freezes could be recovered by removing the system from the dock, and sometimes I needed to hard reboot and lose whatever I was working it.

I also found that I was unable to use my two external monitor plus laptop panel setup as the resolutions and zoom settings could not be set such that all three panels were useful at the same time.

I suspect that my laptop simply does not use the discrete GPU for the external display(s) when in hybrid mode. It looks like maybe this is a bug with xorg, or that the problem may be solved with the new Cosmic DE.

The solution was to just switch to using the nVidia graphics mode. Since then the performance with the dock has been fine, the system is not freezing when switching between applications, and I can once again use all three panels at the same time.
Integrate AppImages
1. Use AppImageLauncher
2. Designate a directory to store the appimages, and set something to back it up if you like.
3. Install the launcher:
```
sudo add-apt-repository ppa:appimagelauncher-team/stable
sudo apt update
sudo apt install appimagelauncher
```
1. Run the app and customise the integration directory to point to your designated directory.
2. That’s it. Running appimages will pop up the launcher to ask if you want to move them and add launcher icons.
3. Thanks to ostechnix.
AntiVirus
1. I get that a lot of advice says to not bother installing AV for Linux, but I’m hardcoded to need to see AV on my workstation. To this end, I installed ClamAV and ClamTK from the Pop!_Shop to be able to use it quickly.
Keeping lists of installed applications
1. To restore from a crash I’ll need to know what is installed.
  1. apt list --installed
  2. ls /usr/share/applications
  3. flatpak list
Screenshots
1. The default screenshot app is fine, but I like to be able to do quick pixelisations or annotations. On Windows I’ve been using Greenshot for years, but they have no Linux option.
2. Thankfully, Flameshot exists. Does everything I need it to do on Linux.
3. Install the DEB from their GitHub so that you can configure the system to use the Print Screen button to take a screenshot with the flameshot gui command. There may be a way to do this with the flatpak version that is available through the Pop!_Shop.
Changing the behaviour when closing the laptop lid
1. Thankfully I didn’t need to do any of this. But just in case this might be useful
The fingerprint reader
1. Apparently is not supported by either the gnome option or any of the other options I was able to find and try, such as fprint.
LibreOffice
1. Again, I’m interested in kinda forcing myself to stick with Linux by making it look as much as possible like my more familiar Windows surroundings.
2. So, tweaking LibreOffice a bit doesn’t hurt.
3. I also found that the versions of LibreOffice that come with Pop!_OS don’t seem to upgrade to the newest versions. To resolve this I did a fresh install of the Linux versions of the apps, and then removed those older ones that came with the OS.
Photo Manager
1. One of my big problems is that over the years my photos have become very duplicated and spaghettified. They are spread over hundreds of folder and renamed here and there in attempts to identify or save meta data. At times I’ve done mass restores to separate folders that then were merged - it’s a nightmare.
2. After looking through some options, I settled on digiKam to try and help me sort this out. It looks like it does a great job of identifying duplicates which is the first step for me. Now I can go through all the files I have and sort them as I import them and remove the straight up duplicates.

Customise Apps

vscodium
1. Change the terminal with the command palette terminal: select default profile.
2. Set up sync with the Settings Sync extension. Make sure it’s set to Auto Download and Auto Upload, add and remove extensions, and quiet sync (otherwise it keeps popping up an output window and taking the focus from any terminals that are running).
PyCharm
1. Change the terminal here too - use the New Terminal
pandoc
1. Required for some of my static sites. pandoc is a pain but it’s easy to fix thanks to this post. Basically, in conda you can get it working with conda install pandoc, and without conda there are other things that need doing.

Merging Git Repositories

2024-06-27T00:00:00-06:00

I had a bunch of separate git repositories that I wanted to merge into one, while also maintaining the existing git history as much as possible.

There were three cases: one where the original repository is in a directory which can be simply renamed to fit into the new structure, one where the repository was in a subfolder which needed to be moved up a level or two, and one where there were multiple repositories in a directory which needed to be all be folded together into one directory in the new structure. The new repository should be reformatted so that it looks like all the commits were originally made in the new directory structure. In all cases there are loose files which had been added here and there over time and then not added to the repository at all. Overall, a big mess that I just wanted to get cleaned up, all added to a single repository, and then archived.

Turns out this is time consuming but not technically challenging. Thanks to this post from Andresch Serj, and reinforced with this article on SlingAcademy.

Both of the cases require the use of git-filter-repo. To get this to work I simply downloaded the script, placed it into a new ~/opt directory, and set the file to executable. I then added this ~/opt directory to my path with export PATH=/home//opt:$PATH. I know this is ephemeral but it suits my purposes right now as I’m not sure I’m going to need to add more things to ~/opt in the future.

Case 1

In this case, the source directory contains the development material. But it should be bundled together into a single subfolder so that it fits in the target directory as a sub-project, rather than hanging directly off the root, to make room for further sub-projects to be added.

root
|- target new directory
|-- target new repository and tracked files
|- existing source directory
|-- existing source repository and tracked files
|-- loose files

root
|- new target directory
|-- subdirectory - exact copy of the source directory
|-- new repository blended with original repository

On the terminal, cd into the source directory. Use the command to bundle that material into a new sub-sub-project directory:

git filter-repo --to-subdirectory-filter <sub-sub-project> --force

So, if have a folder structure like this:

root
|- 2024_update
|-- source_code

and I want to have it look like this:

root
|- projectAlpha
|-- 2024_update
|--- source_code

then I first have to create the desired directory structure in place, with this command issued from the /2024_update folder:

git filter-repo --to-subdirectory-filter 2024_update --force

This will leave us with the intermediate structure of

root
|- 2024_update
|-- 2024_update
|--- source_code

Now you can go to the target repository and merge the modified repository in.

Note: In the git remote command be sure of your spelling. The command will not give any error messages if you get the path wrong.

cd <into the folder with the target repository>
git remote add <name> /path/to/the/modified_repository
git fetch <name> --tags
git merge --allow-unrelated-histories <name>/<branch name>
git remote remove <name>

The result of this will be the desired target structure and the git repository with the original timestamps and appearing to be commits to the current directory structure. Gather up and add the loose files and commit them to the new directory.

Case 2

Here, the source repository is in a subfolder which needs to be moved up.

root
|- target new directory
|-- target new repository and tracked files
|- existing source directory
|-- loose files
|-- source subfolder
|--- existing source repository
|--- loose files

root
|- new target directory
|-- subdirectory - exact copy of the source directory
|-- new repository blended with original repository

The process here is mostly the same as in case 1. You just need to use the git filter-repo command to create the target directory structure from the bottom up.

So, if have a folder structure like this:

root
|- 2024_update
|-- 2023_proposal
|--- july_presentation
|---- source_code

and I want to have it look like this:

root
|- projectAlpha
|-- 2024_update
|--- source_code

then I first have to create the desired directory structure in place, with this command issued from the july_presentation folder:

git filter-repo --to-subdirectory-filter 2024_update --force

This will leave us with the intermediate structure of

root
|- 2024_update
|-- 2023_proposal
|--- july_presentation
|---- 2024_update
|----- source_code

and the git repository will believe the root of this is the 2024_update directory. Now you can merge this repository into the target repository with the above commands to get the desired structure.

Case 3

This case is a touch more complex as there are multiple repositories that need to be merged together to give the appearance that they were always in one repository and under the same parent directory.

root
|- target new directory
|-- target new repository and tracked files
|- existing source directory
|-- loose files
|-- subdirectory project 1
|--- original repository 1
|--- loose files
|-- subdirectory project 2
|--- original repository 2
|--- loose files
|-- subdirectory project 3
|--- original repository 3
|--- loose files

root
|- target directory
|-- subdirectory1 - exact copy of the source subdirectory 1
|-- subdirectory2 - exact copy of the source subdirectory 2
|-- subdirectory3 - exact copy of the source subdirectory 3
|-- new repository blended with all the original repository

For this I found it easiest to go through the subfolders and use the above git filter-repo command to build the desired directory structure for each of the repositories. Then I used the fetch and merge commands above to merge each of those repositories into a new temporary repository, named for the desired new parent folder name. Once all the smaller repositories were folded together the git filter-repo command is used on the temporary repository to bundle it all together into a single subfolder - and then finally that repository can be merged into the target repository.

All Cases

Done. Not the end of the world, really. Now you can review the folder structure of the new repository to make sure things moved as they should have, and check the git history to see if the merging worked as expected, maintaining the timestamps and respecting the new folder structure. If it didn’t, stop now and re-clone the new repository and do it all over again. If the git commands worked properly but there are some objects in the original directories still well, those were added to the directories but never to any repository. Just add them to the new target directory now so they are captured.

When everything looks as it should, sync up to GitHub, or archive it however you like. Then you can delete all the old directories so you don’t in the future have to wonder if you ever merged them! Clean as you go!

Hello, world.

2022-08-14T14:11:00-06:00

Well it only seems fitting that the first post I write for this iteration of the site would be a description of how I made it. As I go back and add content from the old site(s) to this one I’m going to manually update the timestamp to the original posting time. So things may be older on the site - but this is the first post. This site is a static site generated with Pelican and hosted on Netlify. I chose Pelican because I wanted a static site generator, and I wanted it to be associated with Python if possible, and I wanted to use Markdown to write the actual posts and files. I don’t require anything like Django right now … maybe later.

Assumptions: You have a GitHub account, and GitHub desktop is installed and configured. Anaconda is installed and configured. You’ve got a Netlify account.

Setup

I like Anaconda, so I defined a new environment there. I double checked that python was at least version 3.8, and installed pelican and markdown, along with whatever dependencies those brought about.
Since I prefer to use GitHub desktop I fired that up and clicked on File > New Repository. I gave my new repo a name and description and pointed the local path to the repos folder to let the application create the subfolder based on the name of the repository. I created the README so I could populate it later. Since I was creating my own Git ignore file later I did not choose one here and I since this is a Pelican website I used the GNU AGPLv3 license to respect the license of the original project.
Now I opened the project folder and create a file called .gitignore. Into .gitignore I added the default Python exclusions from gitignore.io, and at the end of the file I added
```
# Pelican output directory
    output/
```
After committing that to the repo the standard Python bits, as well as the output directory of the site itself, are not going to be pushed to GitHub. That’s fine since Netlify will be rebuilding the output directory locally every time I commit a new change to the repo.
From inside the project folder, I setup the pelican structure with the command pelican-quickstart. This launched a configuration wizard. I accepted the defaults except for:
- “What will be the title of this web site?”
  - This is written to the pelicanconf.py file as the SITENAME
- “Who will be the author of this web site?”
  - Written to pelicanconf.py as AUTHOR
- “Do you want to specify a URL prefix?”
  - Set this to N. When I specified Y then the URL was used instead of relative paths for images which made local development pretty much impossible.
- Get the timezone for the site from Wikipedia, and enter it when prompted.

Customise Pelican and Prepare It For Netlify

I wanted to have a bit of a theme on the site from the beginning, so I hunted around on pelicanthemes.com and settled on the nest theme. After messing about with it a bit I discovered that it needed to be changed to work with the version of Pelican I was using. So I forked the original theme and applied the changes to the fork.
- To do the changes I edited base.html on the theme and changed four instances of ‘|format(’ to be ‘.format(slug=’, so that CATEGORY_FEED_ATOM|format(category.slug) becomes ‘CATEGORY_FEED_ATOM.format(slug=category.slug)’.
After that, I created a /theme folder in the project folder, and downloaded the theme to it.
In the project folder under /content, I added an images folder and a file there for the logo of the site. I scaled it down to 25x25. I also added an image for the background of the header.
To pelicanconf.py I added the default configuration information for the theme. Among other customisations for various strings I pointed the site logo to ‘/images/logo.png’, including the path, and homepage header image specified as the filename only, ‘header-scaled.jpg’.
That should be the base of the site. I committed those changes to the repo and test built the site locally with
```
pelican content -d -s publishconf.py -t theme --autoreload
pelican --listen
```
… which you can now combine on Windows as
```
pelican content -d -s publishconf.py -t theme --autoreload --listen
```
Alternatively you can skip the second command for –listen and use a local server like Live Server in vscode to actually view the site.

The -d flag causes pelican to delete the content directory before rebuilding the site. The --autoreload flag tells pelican to keep watch on the content folders and rebuild the site automatically when content is added, edited, or removed. As expected, that gave me a functioning site on http://localhost:8000 with the logo and background images I specified.
1. Special usage note: until the site is building reliably, just use
```
pelican content -d -s publishconf.py -t theme
```
…so you can see the console errors in the build process.
Now that the shell is working, I created a requirements.txt file of the working environment, both as a checkpoint myself and also so Netlify will be able to duplicate it, with the command pip freeze > requirements.txt.
When I tested this originally, Netlify was deploying the site with Python 2.7 which was not at all working. So to specify the version of Python I was using locally, 3.8, per the docs I created a runtime.txt file in the root of the project, and added 3.8 to that file.
After committing those changes to the repo, it’s ready to go. So I published it to GitHub with the desktop app. I left the repo set to private at this point.

Deploy the site to Netlify

All that remains is to deploy the site with Netlify.

I logged in to Netlify and selected Sites > Add New Site > Import an existing project.
On the next page, I clicked GitHub.
- The first time I did this I needed to connect my Netlify account to my GitHub account by following the prompts.
Once GitHub and Netlify were connected, I did a search on my repos to find the one for the website, and selected it.
On the next page:
- on branch to deploy I selected main.
- The base directory I left blank.
- The build command is pelican content -s publishconf.py -t theme
- The publish directory is output.
That’s it. I clicked on Deploy Site and awaited …

… the shell of my site was deployed to Netlify. Volia.

After that all that remained was to re-direct my domain to point here, and configure the Let’s Encrypt certificate to enable HTTPS. This is available under the Netlify Site settings for the new site. Following the directions on the Netlify pages this change took me all of 4 minutes of editing on my DNS server, and the deployment of the SSL certificate was a couple of clicks and otherwise automatically done in a few minutes.

And that’s it. After several years of stalling I have completed a reasonable starting point for a new static site version of the website and deployed it. Took me a couple of hours to figure it all out, and maybe 20 minutes start to finish for the final production version. Hope this helps.

Thanks!

Raz, for general direction here and here
Frank Corso for the general directions here
Miraj Patel for the pointers to hack the theme so it would work with Pelican 4 here

Add to Pelican - Sitemap and Robots.txt

2022-09-11T00:00:00-06:00

Finally with some more time on my hands I fixed up the last bit of the site migration from Wordpress to Pelican: adding the sitemap and robots.txt file.

The robots.txt file was easy, as this is a static file in the first place. Just create the file in /content/extra and on the next build it will be available in the output.

The sitemap is a bit more involved. Fortunately, there’s a plugin for that.

Add the sitemap plugin to your environment.

python -m pip install pelican-sitemap

Update your requirements.txt so the deployments to other environments, like Netlify when this is published, also add the plugin.

pip freeze > requirements.txt

Add the configuration of the plugin to pelicanconf.py file.

    SITEMAP = {
    "format": "xml",
    "priorities": {
        "articles": 0.5,
        "indexes": 0.5,
        "pages": 0.5
    },
    "changefreqs": {
        "articles": "monthly",
        "indexes": "daily",
        "pages": "monthly"
    },
    "exclude": ["tag/", "category/", "author/"]
}

And … that’s that. Super easy. Pelican is great.

Fixing my Pelican to display images

2022-09-02T00:00:00-06:00

I found that the Pelican site theme I was using was automatically expanding images to 100% width, which I didn’t want to have. I also wanted to use captions on my images.

Thanks to Joe Lotz I was able to sort this out.

This involved switching from the Pelican provided Python Markdown to Pandoc Markdown. This adds a whole bunch more Markdown functionality to the site.

Add Pandoc and the Pelican Pandoc reader to your environment.

python -m pip install pandoc
python -m pip install pelican-pandoc-reader

Update your requirements.txt so the deployments to other environments, like Netlify when this is published, also do this.

pip freeze > requirements.txt

I wanted to extend the site CSS to use my own CSS file. But I also wanted to not make a lot of changes to the source CSS of the theme if possible. So I created a new CSS file under /theme/static/css and then updated /theme/templates/base.html and added my css file in last. I also had to edit the pelicanconf.py file and change the line NEST_CSS_MINIFY = True to NEST_CSS_MINIFY = False; without this change the minified css file is used instead and my custom css is ignored.
Now I was able to comment out the width: 100% and height: 100% from nest.css, and add the captioning CSS that I wanted to use to lazaruk.css.

This also served as a good reminder that css settings for images override the settings on individual images. Because I wanted to use the individual settings to define image size I needed comment those out of the css completely.