tuxette-chix

Not enough space on boot partition

Tue, 25 Jul 2023 11:26:00 +0200

When your ubuntu distribution is installed using the LVM partition system, you can run into an issue while performing updates and upgrades of the distribution because, at some points, the /boot partition becomes too small. This results in messages of this type:

Not enough free disk space 

The upgrade has aborted. The upgrade needs a total of 617 M free 
space on disk '/boot'. Please free at least an additional 282 M of 
disk space on '/boot'.

When this happens,

check the content of your /boot partition (with ls /boot) and remove all unused kernels. This can be performed with
```
sudo apt autoremove --purge
    
```
or, if you have old kernels and images, this sometimes require that you remove them one by one by removing the linux-headers-... and linux-images-... packages along with the remaining files in /boot;
if this is still not enough, edit (as sudo) the file /etc/initramfs-tools/initramfs.conf and change the line
```
COMPRESS=lz4
    
```
to
```
COMPRESS=xz
    
```
and run
```
sudo update-initramfs -u -k all
sudo apt autoremove --purge
    
```

This should be enough to solve the problem.

nextcloud was extremely slow (and how I solved it)

Fri, 30 Dec 2022 08:36:00 +0100

Nextcloud was extremely slow since a couple of months: all tentative to synchronize tasks or even keepass2 file on android failed, updating the task list in nextcloud took more than 10 minutes and connexion to the web interface also took several minutes. I tested a few things to improve performance and ended up finding the origin of the problem and fixing it.

Configuring caching
Increasing swap size
Setting properly CRON for maintenance tasks

Configuring caching

Reference: Nextcloud documentation

One of the main source of performance improvement for nextcloud is supposed to be a proper configuration of caching with redis. It can be installed using:

sudo apt install redis-server php-redis

and the installation is properly made (and server started) if the following command line

ps ax | grep redis

returns something like:

1920663 ?        Ssl    0:00 /usr/bin/redis-server 127.0.0.1:6376

(the port is usually 6379 instead of 6376).

Then, nextcloud has to be configured to use redis for caching. I used the following setting, entered into the `config/config.php` file of the nextcloud installation directory:

'memcache.local' => '\OC\Memcache\APCu',
'memcache.distributed' => '\OC\Memcache\Redis',
'redis' => [
  'host' => '/run/redis/redis-server.sock',
  'port' => '0',
],
'memcache.locking' => '\OC\Memcache\Redis',

which uses redis through a socket. The socket itself is configured by editing the file `/etc/redis/redis.conf` and by checking the unix socket path:

unixsocket /run/redis/redis-server.sock

Finally, make sure that redis is in the group `www-data` and restart redis and apache2 to ensure the proper use of the updated settings:

sudo usermod -a -G redis www-data
sudo systemctl restart apache2
sudo systemctl restart redis-server

Increasing swap size

Another problem that I encountered was the fact that my memory was almost entirely used and that the swap was full (use `top`, `htop` of `free -h` to check your memory usage). After having killed some of the processes that were running for a while, I freed swap in order to setup a larger swap file:

swapoff -a

I doubled the swap size to 1Mb by creating a swap file with the following command line:

sudo dd if=/dev/zero of=/media/swapfile.img bs=1024 count=1M

on which I put proper permissions and that I registered as a swapfile:

chmod 600 /media/swapfile.img
mkswap /media/swapfile.img

To make the change persisting even after reboot, I also edited the file /etc/fstab to comment out the previous line corresponding to the former swap (/dev/sda3 in my case) and to replace it by the following line:</p>

/media/swapfile.img swap swap sw 0 0

Setting properly CRON for maintenance tasks

Reference: Nextcloud help page on background jobs

However, none of these two previous fixes solved my main issue. I also noticed that nextcloud was slow for my user but not for some other user (for instance, I have a user dedicated to administration of the app that was not having the same type of issue). I then found out [this conversation](https://github.com/nextcloud/server/issues/6203) and saw that the exact same problem was occuring with me: logged on the web interface, in "Personal settings/Security", I had thousands of reported opened sessions).

As the admin user, I thus went to "Administration settings/Administration/Basic settings" and set background jobs to be run through CRON (last option) instead of AJAX. In parallel, on the server, I first run the maintenance task once through the command line, from within the nextcloud installation directory:

sudo -u www-data php -f cron.php

This task took a long time to run (approximately 15 minutes I would say) but ended up with no error and the old sessions from my user account were revoked. The connexion to this account from my smartphone, thunderbird and the web interface became normal again. Most of the opened sessions have been generated by DAVx5 so I'll keep an eye on it in the future.

To finish, I also set up an automatic task to run the CRON script regularly. Advised frequency is 5 minutes but since the previous run took more than this time, I set a frequency time of 60 minutes, which should be enough for my personal instance:

crontab -u www-data -e

and adding the line

*/60  *  *  *  * php -f /var/www/nextcloud/cron.php

properly sets the CRON job.

grub-efi-amd64-signed kept back during upgrade

Fri, 30 Dec 2022 08:26:00 +0100

A recent upgrade of my package versions on Xubuntu 22.04 LTS led to the following error:

The following packages have been kept back: grub-efi-amd64-signed

where, indeed, the package `grub-efi-amd64-signed` could not be updated. The reason is that the list of dependencies related to this package has changed. The following command line fixes the upgrade:

sudo apt install --only-upgrade grub-efi-amd64-bin

Getting rid of firefox based on snap on Ubuntu 22.04

Tue, 25 Oct 2022 09:26:00 +0200

Since Ubuntu 22.04, firefox is based on snap, with various side effects, including inability to download files in fuse.fs mounted partitions. This post explains how to install firefox from mozilla repository instead.

Special thanks to Stéphanie who had a problem with her mounted partitions and pointed the issue to me so that I could fix it for her!

Important warning! Before you start, be sure to have saved your user profile (it is especially important if firefox stores for you your passwords or other login information). This can be done:

either by activating your firefox sync account. In this case, activating the same sync account on any brand new firefox will automatically activate your formally saved passwords and downloaded plugins;
or directly save the snap user profile (included in ~/snap/firefox/common/.mozilla/firefox). After firefox has been reinstalled, it can be copied into ~/.mozilla/firefox/.

Special thanks to >Pierre who had to experience the loss of his data and pointed that issue to me.

Finally, to install the mozilla repository based version of firefox, first remove firefox based on snap:

sudo snap remove firefox

and add mozilla PPA in you source list:

sudo add-apt-repository ppa:mozillateam/ppa

Then, as sudo, create a file /etc/apt/preferences.d/mozilla-firefox where you put the following lines:

Package: *
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

Package: firefox*
Pin: release o=Ubuntu
Pin-Priority: -1

To finish,

sudo apt update
sudo apt upgrade
sudo apt install firefox

should be enough to properly install firefox (and as an extra surprise, install a better version of thunderbird, compatible with more plugins, including the TbSync plugin that is the only one to handle Active Directory synchronization and is not functional in Ubuntu official release of thunderbird).

Encrypting some files in git repositories with git-crypt

Tue, 09 Aug 2022 11:26:00 +0200

This post describes how to encrypt some files (containing secrets) in git repositories in a transparent way. The chosen solution encrypts files before they are pushed to the remote and decrypts it at pull locally (so you have to make sure that your secrets are well protected on your local computer).

Different tools exist to perform this task, including BlackBox, SOPS, transcrypt, git-secret. I chose git-crypt because it is available through a Ubuntu package, is really fully transparent, is well referenced, cited, and used, is regularly maintained, and can work over GnuPG.

First step: as a maintainer, initialize your git-crypt repository

git-crypt is easily installed with apt (install gnupg if you do not have it installed already):

sudo apt install gnupg git-crypt

Locally, initialize your git repository as usual and add the git-crypt initialization afterwards:

git init
git-crypt init

Then, create a .gitattributes file including the list of files and directories that you want encrypted in your remote repository:

secretfile filter=git-crypt diff=git-crypt
*.key filter=git-crypt diff=git-crypt
secretdir/** filter=git-crypt diff=git-crypt

Note that, to include all files of a given directory secretdir/, the syntax is secretdir/** and not secretdir/*!

GnuPG users can be added by ID or email by:

git-crypt add-gpg-user tuxette@my-domain.org

This creates an encrypted GPG key within .git-crypt/keys, which is automatically commited.

More information on GnuPG is available at this page (unfortunately in French).

Then, configure your remote directory (supposed, here, to be empty; this is done with git remote add ...), add and commit all the files that are supposed to be versionned, in addition to the file .gitattributes and push. Further add, commit, pull, push commands can then be used as usual.

Second step: as a new user to an existing repository, set your git-crypt configuration

You need first to install gnupg and git-crypt with:

sudo apt install gnupg git-crypt

Then, start by generating your GPG key (if you don't have one already) with:

gpg --gen-key

Once done, you can check it with:

gpg --list-keys

that should look like:

pub   rsa4096 2016-10-31 [SC]
      551C582A867ABF1865E86006378CDF2A339F144E
uid           [ultimate] Tuxette Chix <tuxette@my-domain.org>
sub   rsa4096 2016-10-31 [E]

which you can export using

gpg --armor --export 551C582A867ABF1865E86006378CDF2A339F144E > my_public_key.pub

(the ID 551C582A867ABF1865E86006378CDF2A339F144E has to be adapted to your own key). Send this file to the repository admin and waits for her/him to allow you to access the git-crypt repository.

When everything is ready, all you have to do is simply:

git clone gitolite@git.my-domain.org:my-nice-git-repo.git
git-crypt unlock

You can then add, commit, pull and push as usual.

Second step bis: as an admin, add a new user to use secrets

When a new user sends her/his GPG key, add it to your GnuPG configuration with:

gpg --import new-public-key.pub

You can check that it has properly been added with

gpg --list-keys

You may need to "trust" the key before you can proceed. You can do it by signing it:

gpg --edit-key 551C582A867ABF1865E86006378CDF2A339F144E

then type sign, quit, and finally yes to save the edition of the key.

Finally, in the git repository, run:

git-crypt add-gpg-user <new_user_email@other-domain.org>

where the email is the one referenced in the key (you can also add a user with the key ID). Do not forget to push the changes (the new user's encrypted key)!

Installing multiple versions of R alongside and corresponding renv settings

Mon, 08 Feb 2021 10:26:00 +0100

This post describes how to properly install multiple versions of R alongside on Ubuntu OS and to set up a properly controlled renv environment.

Preparation

It is better to start with a fresh installation of R. However, if R is already installed, especially if it is installed from the Ubuntu repository, it is better to get rid of the old version first:

sudo apt autoremove --purge r-base r-base-dev r-recommended

In addition, removing the packages is not enough to remove R most of the times. A safe approach is to simply change the binary name in /usr/bin with:

sudo mv /usr/bin/R /usr/bin/R2

(which is an ugly way to solve this issue, indeed).

Installing R from scratch

Inspired by this tutorial, we can now install any R version obtained from the RStudio repository (under the form of a .deb package). The following lines of codes are given for R version 4.0.3 and Ubuntu 20.04 LTS (they have to be adapted for other versions of R and other OS and OS versions):

export R_VERSION=4.0.3
export UBUNTU_YEAR=2004
curl -O https://cdn.rstudio.com/r/ubuntu-${UBUNTU_YEAR}/pkgs/r-${R_VERSION}_1_amd64.deb
sudo dpkg -i r-${R_VERSION}_1_amd64.deb
sudo apt install -f

At this stage, R should be properly installed and work with:

/opt/R/${R_VERSION}/bin/R

The last stage of the installation would ensure that the R binary is properly linked into a bin folder:

sudo ln -s /opt/R/${R_VERSION}/bin/R /usr/local/bin/R
sudo ln -s /opt/R/${R_VERSION}/bin/Rscript /usr/local/bin/Rscript

If you are using RStudio with multiple R versions installed as described here, you can choose between them as described here.

`renv` environment

Within our project directory, if the renv.lock file is already present and has never been used, the virtual R environment is installed with the following R command lines:

install.packages("renv")
renv::init()

In our case, some packages have shown additional issues:

nloptr required the installation of an additional Ubuntu library:
```
sudo apt install libnlopt-dev
```
some packages depended on curl and required the installation of:
```
sudo apt install libcurl4-openssl-dev
```
ps had a problem with a wrong installation directory for sed that can be resolved creating a symbolic link:
```
ln -s /bin/sed /usr/bin/sed
```

When the renv initialization process is interrupted, it can be resumed with:

renv::restore()

Special case of Bioconductor packages

Bioconductor packages can produce errors at installation within renv. The reason is that the Bioconductor packages are linked to a Bioconductor version that is not itself well handled by renv. Hence, you first have to check that your local Bioconductor version is identical with the Bioconductor version of the initial renv installation (for ASTERICS, it is 3.12). To check your local Bioconductor version, do:

BiocManager::version()

(BiocManager has previously been installed with renv::install("BiocManager") for instance). You have to specify:

BiocManager::install("3.12")

and to install your Bioconductor packages with:

BiocManager::install("pkg_name")

before proceeding to renv::restore(). If it still does not work, check your renv.lock file and replace

"Source": "Repository",
"Repository": "Bioconductor"

"Source": "Bioconductor",

for Bioconductor packages.

Branch management in asterics

Mon, 08 Feb 2021 10:26:00 +0100

This post describes the best practices for asterics in branch management.

Creating a new branch

Branches are created at the head of dev as follows (within asterics repository):

git checkout dev
git pull
git branch dev-new
git checkout dev-new
git branch --set-upstream-to origin/dev-new

Merging a branch

Feature branches are merged into dev when they reach a stable state. Before merging, ensures that R tests are all successfull (start a new R session at the project root and proceed as described here.

When you decide to merge, go to the ASTERICS project on ForgeMIA, into the menu (on the left) Repository/Branches and click on the "Merge request" button of the branch that you want to merge (be careful to ask for a merge into dev and not another branch). Once the merge resquest is performed, ask for someone else to review your code and to perform the merge (this is a two-step process during which you first have to review and correct possible conflicts and then to approve the merge).

Deleting a branch

The person merging its branch into dev is responsible for deleting his/her branch locally and remotely (after it has been merged and pushed):

git branch -d dev-new
git push origin --delete dev-new

If this change is not properly spread with pull on a local repository, you can check which branches are present (with git branch -a) and if they fit the branches as described on the forge. If not, delete the branch locally (as described above) and prune its remote counterpart:

git remote update --prune

R tests in asterics

Mon, 08 Feb 2021 10:26:00 +0100

This post describes how R unit tests are organized and managed in asterics using testthat.

Tests organization

Tests are present in the directory backend/R/tests in separate files starting with test- (this is mandatory for the test to be run). In addition, as a naming convention, each test refers to a single function and is named from this function (for instance, the tests for r_multivariate_dotplot are in the file test-rmultivariatedotplot). They are all based on the testthat package and can be executed with this **R** command line:

test_dir("backend/R/tests")

(if launched from the project root directory)

Automatic testing at the project startup

Tests are launched at the project startup within **R** with:

tests <- TRUE
source("backend/R/conf_wd.R")

Tests and branch merging

When a branch is merged,

either this branch pre-existed before the merging of tests into dev and the merging of the branch has to be done with: 1/ git merging of the branch into dev; 2/ testing everything with a session restart; 3/ systematically correcting failed tests and documentation accordingly directly in dev,
either this branch was creating after test support has been merged into dev. In this case, tests have to be performed and fixed along with documentation before merging into dev.

mysql installation

Tue, 11 Aug 2020 11:26:00 +0200

This post describes how to install and secure mysql and phpmyadmin on Ubuntu server 20.04 LTS.

mysql installation

Refs: https://www.digitalocean.com/community/tutorials/how-to-install-linux-apache-mysql-php-lamp-stack-on-ubuntu-20-04-fr and https://ostechnix.com/install-phpmyadmin-with-lamp-stack-on-ubuntu-20-04-lts/

Installation of mysql is performed with the following command lines:

sudo apt install mysql-server
sudo mysql_secure_installation

with the options:

Would you like to setup VALIDATE PASSWORD component?

yes and

There are three levels of password validation policy:

that is chosen between

LOW    Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary

In addition, anonymous users are removed, root is only allowed to connect from localhost and the test database is removed.

phpmyadmin installation

phpmyadmin is further installed with:

sudo apt install php libapache2-mod-php php-mysql phpmyadmin php-mbstring php-zip php-gd php-json php-curl

where the configuration is made with

web server: apache2
connexion default: unix socket
authentication plugin: default
mysql database name: MYDATABASE
mysql username : MYUSER@localhost

If an error appears stating:

Error with password: type abort

you need to temporarily disable the validate password component:

sudo mysql
UNINSTALL COMPONENT "file://component_validate_password";
exit

and relaunch phpmyadmin installation

sudo apt install phpmyadmin php-mbstring php-zip php-gd php-json php-curl

sudo apt install -f

before re-installing the component:

sudo mysql
INSTALL COMPONENT "file://component_validate_password";
exit

The installation is completed with:

sudo phpenmod mbstring
sudo systemctl reload apache2

to enable php in apache2 configuration and with:

sudo mysql
CREATE USER 'MYUSER'@'localhost' IDENTIFIED BY 'MYPASSWORD';
GRANT ALL PRIVILEGES ON *.* TO 'pma'@'localhost' WITH GRANT OPTION;

to create a user for phpmyadmin (if not already performed during the installation). The connexion to phpmyadmin can be done at http://mydomain.org/phpmyadmin or a virtual host can be created after editing

/etc/apache2/conf-available/phpmyadmin.conf</a> to comment out the automatic redirection (it is advised to configure SSL only connexion as well).
</p>

phpmyadmin configuration
phpmyadmin configuration file is /etc/phpmyadmin/config.inc.php where cookie authentication are enabled with the option:
  
$cfg['Servers'][$i]['auth_type'] = 'cookie';
  
  and root login is disabled with:
  $cfg['Servers'][$i]['AllowRoot'] = FALSE;
  


To allow fail2ban to secure phpmyadmin connexion, the logs need to be activated with the following option:
  
$cfg['AuthLog'] = 'syslog';
$cfg['AuthLogSuccess'] = false;
  
  which leads to the receive this type of message into /var/log/auth.log:
  Aug 14 13:55:51 chix phpMyAdmin[176689]: user denied: FAKEUSER (mysql-denied) from XX.YYY.YYY.ZZ
  
  when users try to login without success. The edition of /etc/fail2ban/jail.local with the addition of:
  [phpmyadmin-syslog]
enabled = true
port = http,https
filter = phpmyadmin-syslog
logpath = /var/log/auth.log
backend = %(syslog_backend)s
  
  leads to ban IPs with too many unsucessful login attempts. This configuration is enabled with:
  sudo systemctl reload fail2ban

jekyll installation with git

Tue, 11 Aug 2020 11:26:00 +0200

This post describes how to install jekyll (and necessary plugins) on your server (Ubuntu 20.04 LTS) so as to automatically generate and publish your websites through git versionning.

The best seems to start by not installing the jekyll package from ubuntu public repositories. The reason is that it is best to have local gem installation to avoid versionning problems with pluggins installed using ruby and gems directly. To do so, you start by installing the ruby packages:

sudo apt install ruby-full build-essential zlib1g-dev

and you switch to your git user (on my server, called "git") to update its local profile:

sudo su git
echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME="$HOME/gems"' >> ~/.bashrc
echo 'export PATH="$HOME/gems/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

jekyll (and plugins) can then be installed with the gem command:

gem install jekyll bundler
gem install jekyll-scholar

At this stage, the user gitolite should be able to use the command jekyll build to generate a website created for jekyll.

The last step is to automate the generation and publication of the website when a modification is pushed on its git repository. Suppose that the apache VH points to the directory /var/www/blog in which the website is published. First, you have to ensure that this directory is owned by the user "git":

chown -R git:git /var/www/blog

The second steps consists in creating a script that will build and copy the website content. To do so, if the website is versionned in the directory </code>/var/lib/gitolite3/repository/blog.git</code>, you create and edit a file </code>/var/lib/gitolite3/repository/blog.git/hooks/post-receive</code> that contains:

GIT_REPO=/var/lib/gitolite3/repositories/blog.git
TMP_GIT_CLONE=/var/lib/gitolite3/tmp/myrepo
PUBLIC_WWW=/var/www/blog

git clone $GIT_REPO $TMP_GIT_CLONE
cd $TMP_GIT_CLONE
jekyll build  -d $PUBLIC_WWW
cd ~/
rm -Rf $TMP_GIT_CLONE
exit

This file should be made executable with

chmod ug+x /var/lib/gitolite3/repository/blog.git/hooks/post-receive

and... that's it!

tuxette-chix

Not enough space on boot partition

nextcloud was extremely slow (and how I solved it)

Configuring caching

Increasing swap size

Setting properly CRON for maintenance tasks

grub-efi-amd64-signed kept back during upgrade

Getting rid of firefox based on snap on Ubuntu 22.04

Encrypting some files in git repositories with git-crypt

First step: as a maintainer, initialize your git-crypt repository

Second step: as a new user to an existing repository, set your git-crypt configuration

Second step bis: as an admin, add a new user to use secrets

Installing multiple versions of R alongside and corresponding renv settings

Preparation

Installing R from scratch

renv environment

Special case of Bioconductor packages

Branch management in asterics

Creating a new branch

Merging a branch

Deleting a branch

R tests in asterics

Tests organization

Automatic testing at the project startup

Tests and branch merging

mysql installation

mysql installation

phpmyadmin installation

phpmyadmin configuration

jekyll installation with git

`renv` environment