Vimalkumar Velayudhan

Installing Conda Using the Miniforge Installer

2026-03-07T00:00:00+00:00

Table of Contents

About Conda
Step 1: Downloading Miniforge Installer
Step 2: Running the Installer Script
Step 3: (Optional) Disabling Auto-Activation of base Environment

About Conda#

Conda is a software and environment manager. You can use it to create an isolated environment where you can install a software along with its dependencies i.e., all the software it depends upon to function.

There are three different sources from which you can get Conda — Miniconda, Anaconda, and Miniforge. In this tutorial, I will install Conda using the Miniforge installer.

Note
I run these commands on Linux Mint 22.1. These steps should also work on Ubuntu 24.04 LTS.

Step 1: Downloading Miniforge Installer#

Visit https://conda-forge.org/ and click on the Download Installer button.

This should take you to the Download the conda-forge Installer page.

Under Miniforge Latest Release, click on the Linux x86_64 (amd64) option.

This will download the installer file to your computer as Miniforge3-Linux-x86_64.sh.

Step 2: Running the Installer Script#

Open a terminal.

Change in to the directory where you downloaded the installer script. In my case this was /home/vimal/Downloads:

$ cd ~/Downloads

Then, run the installer script using bash:

$ bash Miniforge3-Linux-x86_64.sh

Press the ENTER key. You will need to answer the following prompts to continue.

Q1: Miniforge License Agreement#

First, you will need to read the Miniforge license¹ agreement and accept the terms to proceed with the installation. To scroll down, press the ENTER key.

To proceed, type yes at the >>> prompt and press the ENTER key.

Do you accept the license terms? [yes|no]
>>> yes

Q2: Where to Install Miniforge#

Next, you will need to select a location for installing Miniforge.

Press the ENTER key to use the default — /home/user/minforge3. You can also type in a different path after the >>> prompt, if you wish.

Miniforge3 will now be installed into this location:
/home/vimal/miniforge3

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/home/vimal/miniforge3] >>>

Press the ENTER key to proceed.

Q3: Do You Wish to Initialize Conda?#

In the final step, you will need to decide if you wish to initialize conda.

Typing yes here will have the following effects:

The conda command will be available in your PATH. ²
The Conda base environment will be activated automatically when you open a terminal session.

Type either yes or no and press the ENTER key to proceed:

Do you wish to update your shell profile to automatically initialize conda?
...
Proceed with initialization? [yes|no]
[no] >>> yes

Note
I have answered yes here, but I will disable auto-activation of the Conda base environment in Step 3. This is because I prefer to activate conda environments only when I need it.

Check if Conda is Installed Correctly#

Open a new terminal session:

(base) vimal@mint-zara:~$

Notice the (base) label at the beginning of the prompt. This means the Conda base environment is now active.

Now, type:

$ conda --version

Example output:

conda 25.3.1

If you receive an output like the above, Conda is installed and ready to use.

Step 3: (Optional) Disabling Auto-Activation of base Environment#

If you answered yes in Step 2 Q3, the Conda base environment will be activated automatically whenever you open a new terminal session.

Note
This step is only necessary if you prefer to activate conda environments manually.

Type the following command:

$ conda config --set auto_activate false

There will be no output, but this command will update your Conda configuration. If you open a new terminal, you will notice the (base) label will no longer be visible.

Now, if you wish to activate the Conda base environment, type:

$ conda activate

To exit the environment, type:

$ conda deactivate

—

Footnotes:

Miniforge installer code uses the open source BSD-3-Clause license, but the packages that are bundled with the installer might have their own open source licenses. More information at https://github.com/conda-forge/miniforge/blob/main/LICENSE. ↩
Otherwise, you will need to update your PATH variable in ~/.bashrc to include the path to your miniforge3 installation. ↩

Getting the Lineage of an Organism Using Python, BioSQL, and Taxonomy Data

2018-07-14T00:00:00+01:00

Table of Contents

Introduction
Step 1: Install Required Software
Step 2: Create a Database for Storing Taxonomy Data
Step 3: Import NCBI Taxonomy Data Into Database
Step 4: Run Python Script to Get Lineage

Introduction#

You can retrieve the complete taxonomic lineage of an organism by querying the NCBI Taxonomy database using the Entrez API. The method described here works offline and will be useful if you need to retrieve the taxonomic lineage for a large number of organisms at once.

Much of the information in this blog post comes from the documentation of the BioSQL and BioPython projects. In addition, I wrote a Python script to query the taxonomy database and return the complete lineage of an organism.

Here is how the script works:

$ python3 lineager.py -n Bos taurus

And the resulting output:

2018-07-14 04:47 INFO     Processing organism name provided
                          at the command line: Bos taurus
Organism,Lineage
Bos taurus,cellular organisms;Eukaryota;Opisthokonta;Metazoa;
Eumetazoa;Bilateria;Deuterostomia;Chordata;Craniata;Vertebrata;
Gnathostomata;Teleostomi;Euteleostomi;Sarcopterygii;
Dipnotetrapodomorpha;Tetrapoda;Amniota;Mammalia;Theria;
Eutheria;Boreoeutheria;Laurasiatheria;Cetartiodactyla;
Ruminantia;Pecora;Bovidae;Bovinae;Bos;Bos taurus

You can also provide an input file (-f), containing names of organisms. See the example towards the end of this blog post.

Step 1: Install Required Software#

These steps were tested in Ubuntu 24.04 LTS with the following programs installed.

MySQL 8.0.45
For storing taxonomy data.

Perl 5.38.2 with the DBI, DBD, and libwww-perl modules
For initialising the database with the BioSQL schema and for importing taxonomy data. Optionally, the perl-doc package for viewing documentation.

Python 3.12.3 with the MySQLdb package
For running the lineager.py script.

Note:
sudo privileges are required only for installing packages, creating the database, and creating the database user.
All other commands are run from a regular user account.

Install MySQL using the command:

$ sudo apt install -y mysql-server

Once the installation is complete, check if the MySQL server is up and running using the command:

$ ss -tap | grep mysql
LISTEN 0      151        127.0.0.1:mysql       0.0.0.0:*    users:(("mysqld",pid=1323,fd=23))
LISTEN 0      70         127.0.0.1:33060       0.0.0.0:*    users:(("mysqld",pid=1323,fd=21))

Note:
For additional information on installing and configuring MySQL, please refer to the Installing MySQL section of the Ubuntu Server documentation.

Install remaining packages using apt:

$ sudo apt install -y libdbi-perl libdbd-mysql-perl \
  libmysqlclient-dev python3-mysqldb perl-doc \
  libwww-perl

Step 2: Create a Database for Storing Taxonomy Data#

$ sudo mysql -u root

Create Database#

Run the following SQL command. Note the use of backticks (`) around the database name (biosql), instead of single quotes:

CREATE DATABASE `biosql` COLLATE 'utf8_general_ci';

Create Database User#

Run the following command to create a database user. Also, grant this user permissions to access and modify the biosql database.

Note:
Replace ‘your-password-here’ with a strong password in the command below.

CREATE USER `lineager`@`localhost` IDENTIFIED BY 'your-password-here';
GRANT ALL PRIVILEGES ON `biosql`.* TO `lineager`@`localhost`;

Type exit to quit the MySQL shell.

Initialise Database With BioSQL Schema#

First, create a file named .my.cnf in the user HOME directory with the following content:

[client]
user = lineager
password = your-password-here
database = biosql

This file is used for connecting to the database from the lineager.py script. Replace your-password-here with the password you set above in Step 2.

Since the .my.cnf file contains the database password in plain text, it is a good idea to make it readable only for the current user. To do this, use the following command:

$ chmod 600 ~/.my.cnf

Next, download the BioSQL schema:

$ wget https://github.com/biosql/biosql/archive/refs/heads/master.zip

Extract the archive and change directory:

$ unzip master.zip
$ cd biosql-master

Initialise database by running the biosqldb-mysql.sql script:

$ mysql -u lineager -D biosql < sql/biosqldb-mysql.sql

Step 3: Import NCBI Taxonomy Data Into Database#

While still in the biosql-master directory, create a directory to store taxonomy data and change into it:

$ mkdir taxdata
$ cd taxdata

Download the taxonomy database (~70MB) and its MD5 checksum:

$ wget -c https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz
$ wget https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz.md5

Ensure the file has downloaded correctly by verifying its MD5 checksum. A successful download should return “OK”:

$ md5sum -c taxdump.tar.gz.md5
taxdump.tar.gz: OK

Uncompress the archive:

$ tar zxvf taxdump.tar.gz

Before running the script to import taxonomy data, set the DBI_PASSWORD environment variable. This is the same password you set for the database user in Step 2:

$ export DBI_PASSWORD='your-password-here'

Now run the script to import taxonomy data into the database:

$ cd ..
$ perl scripts/load_ncbi_taxonomy.pl --dbname biosql --dbuser lineager

Note:
This will take a long time to complete. Leave the terminal session open.

Output:

Loading NCBI taxon database in taxdata:
    ... retrieving all taxon nodes in the database
    ... reading in taxon nodes from nodes.dmp
    ... insert / update / delete taxon nodes
    ... updating new parent IDs
    ... (committing nodes)
    ... rebuilding nested set left/right values
    ... reading in taxon names from names.dmp
    ... deleting old taxon names
    ... inserting new taxon names
    ... cleaning up
Done.

To view all the options supported by the script, use the command:

$ perl scripts/load_ncbi_taxonomy.pl --help

Step 4: Run Python Script to Get Lineage#

Download the source code of lineager using wget:

$ cd
$ wget https://codeberg.org/vimalkvn/lineager/archive/master.zip

Alternatively, clone the Git repository:

$ git clone https://codeberg.org/vimalkvn/lineager

Run the lineager.py script with the name of the organism, for example:

$ cd lineager
$ python3 lineager.py -n Escherichia coli

Output:

Organism,Lineage
Escherichia coli,cellular organisms;Bacteria;Pseudomonadati;Pseudomonadota;Gammaproteobacteria;Enterobacterales;Enterobacteriaceae;Escherichia;Escherichia coli

You can also get the lineage for multiple organisms at once. To do this, create a text file containing names of organisms like this:

Canis lupus familiaris
Bos taurus
Escherichia
AMBIGUOUS
Arabidopsis thaliana

Save the file as input.txt and then, run the script like this:

$ python3 lineager.py -f input.txt

When complete, an output file lineage.csv will be generated in the same directory containing lineage information for all organisms in the input file.

If you have any questions or comments, please send me an email.

Running Galaxy Project on Ubuntu 16.04 LTS

2017-11-30T00:00:00+00:00

Updates

Aug 18, 2021

The title of this blog post is changed from “Setup your own instance of the Galaxy Bioinformatics platform on DigitalOcean” to “Running Galaxy project on Ubuntu”, as these steps can be performed on a standalone Ubuntu system and on VPS providers other than DigitalOcean.

The URL of this blog post is changed from https://vimal.io/setup-your-own-galaxy-bioinformatics-instance-on-digitalocean/ to https://vimakvn.com/running-galaxy-project-on-ubuntu.

Minor updates to text, links, and formatting.

I will use DigitalOcean as the VPS provider in this tutorial, but you can perform these steps on any Ubuntu system.

When complete, you will have an instance of Galaxy which can be used to host tools developed by the research group.

Let’s begin by creating the virtual server.

Step 1: Create the virtual server (droplet)#

First, you will need an account on DigitalOcean, which is a paid service.

If you would like to support my work, you can use this referral link to sign up on DigitalOcean. You will get a $100, 60-day credit.

Once you have confirmed your email account and completed the verification step by adding a payment method, your can start creating virtual servers — also known as droplets.

Video: Creating a droplet

The server we are going to create will be based on Ubuntu 16.04 LTS and will cost about $5 per month for the basic configuration — 512MB Memory, 1 CPU and a 20GB Hard disk.

To create a new droplet:

Click on the Create Droplet button.
In the next page, click on the $5/mo server panel in the Choose a size section.
Leave all other options at their defaults and then click on the Create button at the bottom of the page.

Once the droplet is created, you will receive an email with details for accessing the droplet using SSH — IP Address, Username and Password. You can connect to the droplet using the ssh command like this ¹

ssh root@server-ip

You will be prompted to change the root password when you first login.

IMPORTANT
Please make sure to destroy the droplet if you are only using it for the purpose of this tutorial. Droplets that are powered down will still be charged after the credit is used up!

We can now proceed towards configuring the droplet.

Step 2: Configure droplet#

2.1 Create user with administrative access#

For all administrative functions like installing software, creating user accounts, database and configuring the web server, we will create a separate account with administrator privileges. This can be done by adding the user to the sudo group. In the following example, I am creating a new admin user called vimal and setting a password:

useradd -m -s /bin/bash -G sudo vimal
passwd vimal

ssh vimal@server-ip

2.2 Add swap space#

The database migration scripts that run when Galaxy is first launched require more memory than we have available (512MB). We can make use of a swap file so this process completes.

To create and activate a 2GB swap file in the Root (/) partition, the following commands can be used:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Enable swap on boot by adding the following line to /etc/fstab:

/swapfile   none    swap    sw    0   0

NOTE
This is only a temporary measure.

Frequent use of swap will degrade performance of both the application and the hardware. Also, programs requiring more memory (example: alignment) will fail to work. It is better to upgrade to a plan with more memory (RAM) for better performance.

We can now proceed towards installing required software.

2.3 Install required software#

First, refresh the package repositories, then upgrade system packages and then remove packages that are no longer required. All these steps can be done using the following command:

sudo apt update && sudo apt -y upgrade && \
  sudo apt -y autoremove

Install build tools, Git, Apache web server and required modules, Virtualenv for creating virtual Python environments, libraries and functions required for building Python packages, the PostgreSQL_ database server, client and libraries:

sudo apt -y install build-essential git apache2 \
libapache2-mod-xsendfile virtualenv python-dev \
postgresql postgresql-contrib postgresql-client \
libpq-dev

2.4 Set PostgreSQL administrator password#

Set a password for the PostgreSQL administrator account — postgres using psql:

sudo -u postgres psql template1
postgres=# \password postgres
postgres=# \q

2.5 Enable required Apache modules#

Enable modules necessary for running Galaxy under Apache. Restart apache2 service for configuration to take effect:

sudo a2enmod rewrite xsendfile expires proxy proxy_http deflate headers
sudo systemctl restart apache2

2.6 Create Galaxy user accounts and database#

Create a Linux user account called galaxy and set a password for this user:

sudo useradd -m -s /bin/bash galaxy
sudo passwd galaxy

As PostgreSQL administrator, create a database user called galaxy and a database owned by that user, also called galaxy:

sudo -u postgres createuser -P galaxy
sudo -u postgres createdb -O galaxy galaxy

Step 2: Configure the Apache web server#

We will serve the Galaxy web interface from a sub directory like
http://server-ip/galaxy

To do this, create a new VirtualHost configuration by creating a file
/etc/apache2/sites-available/galaxy.conf
as the admin user and add the following content:

<VirtualHost *:80>
    # ServerName is currently set to the IP address. When a domain name is
    # available, this directive should be updated
    ServerName server-ip
    ErrorLog ${APACHE_LOG_DIR}/galaxy-error.log
    CustomLog ${APACHE_LOG_DIR}/galaxy-access.log combined

    RewriteEngine On
    RewriteRule ^/galaxy$ /galaxy/ [R]
    RewriteRule ^/galaxy/static/style/(.*) /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L]
    RewriteRule ^/galaxy/static/scripts/(.*) /home/galaxy/galaxy/static/scripts/packed/$1 [L]
    RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L]
    RewriteRule ^/galaxy/favicon.ico /home/galaxy/galaxy/static/favicon.ico [L]
    RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt [L]
    RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P]

    <Location "/galaxy">
        # Compress all uncompressed content.
        SetOutputFilter DEFLATE
        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary
        XSendFile on
        XSendFilePath /
    </Location>

    <Location "/galaxy/static">
        # Allow browsers to cache everything from /galaxy/static for 6 hours
        Require all granted
        ExpiresActive On
        ExpiresDefault "access plus 6 hours"
    </Location>
</VirtualHost>

Save the file and then enable the site i.e., this VirtualHost configuration and reload Apache:

sudo a2ensite galaxy
sudo systemctl reload apache2

Step 3: Download, configure and start Galaxy#

Clone the current stable galaxy release ² using Git:

git clone -b release_17.09 \
   https://github.com/galaxyproject/galaxy.git

Create a Python 2 virtual environment for galaxy:

cd galaxy
virtualenv -p python2 galaxy_env

Create a configuration for Galaxy by making a copy of the sample configuration:

cp config/galaxy.ini.sample config/galaxy.ini

Make changes in galaxy.ini as given below. There are detailed comments in this file as to what these options do.

As we are using Apache as the proxy server uncomment the filter-with option and specify the cookie_path option in the [app:main] section:

[app:main]
filter-with = proxy-prefix
cookie_path = /galaxy

As we are using PostgreSQL, the database_connection setting in the Database section should be changed to the following:

database_connection = postgresql:///galaxy?host=/var/run/postgresql

For better performance with large database queries in PostgreSQL, we can also set the following:

database_engine_option_server_side_cursors = True

Also to use Apache to handle file downloads instead of Galaxy’s own HTTP server, we will need to set the following apache_xsendfile option in the [app:main] section:

apache_xsendfile = True

As this is a production server, disable live debugging:

use_interactive = False

Finally, in the Users and Security section, add your email address to the admin_users variable:

admin_users = you@email.com

If public access to this instance is not desired, disable anonymous access and disable user registration:

require_login = True
allow_user_creation = False

Save galaxy.ini, activate virtualenv and then start Galaxy in daemon mode:

# Activate the virtualenv
source galaxy_env/bin/activate

# Start Galaxy in daemon mode
./run.sh --daemon

A log file, paster.log will be generated in the same directory. Once the message Entering daemon mode is displayed, use the command:

tail -f paster.log

to view the log. When the script has finished initialisation, the following message will appear:

serving on http://127.0.0.1:8080

At this stage, visit http://server.ip/galaxy, register an account with the same email address specified under admin_users and login.

Step 4: Improving security and performance, updating Galaxy#

4.1 Secure SSH access#

In the default setup, SSH runs on port 22 and allows all users to login with a password including root. These are some additional steps for securing SSH access. All these configuration changes should be made in the file /etc/ssh/sshd_config.

Change default port#

The port that SSH listens on is specified in the Port option:

Port 22

To prevent unauthorised login attempts on the default SSH port (22), the recommendation is to change the port to a number less than 1024. The port that is selected should not currently be used by any other service. You can check the list of ports currently assiged to services here or lookup IANA port assignments.

Once the port is changed, restart the sshd service, open the specified port in firewall (Refer: Configure firewall section below). Test if SSH access is working and then close port 22 in firewall.

If you do need to login as root, you can do so using su - once logged in with a regular user account:

PermitRootLogin no

Disable password authentication#

As an increased security measure, setup SSH key based authentication. Here is a tutorial explaining the procedure. Once all users have SSH keys setup and working (important!), disable login with password completely:

PasswordAuthentication no

Restrict users with SSH access#

Only the users specified here will be allowed to connect. Accepts a space separated list of user names::

AllowUsers galaxy vimal

4.2 Configure firewall#

A firewall can be enabled and rules can be configured to allow access only to specific ports and services.

NOTE

This following commands assume that SSH is running on port 822 instead of the default (22). If your configuration is different, please change the port accordingly.

One possibility would be to use the Cloud Firewall function available on DigitalOcean and create a configuration which can be applied to the droplet. A tutorial is available here.

An example configuration allowing access to SSH(822), HTTP(80) and HTTPS(443) is below:

An alternative firewall configuration using UFW:

ufw allow 822
ufw allow http
ufw allow https

Check status

ufw status

Output:

Status: active

To                         Action      From
--                         ------      ----
822                        ALLOW       Anywhere                  
80                         ALLOW       Anywhere                  
443                        ALLOW       Anywhere                  
822 (v6)                   ALLOW       Anywhere (v6)             
80 (v6)                    ALLOW       Anywhere (v6)             
443 (v6)                   ALLOW       Anywhere (v6)

4.3 Enable HTTPS#

For encrypted web communications, it is essential to use an SSL certificate and re-configure Apache. Free SSL certificates can be obtained from Letsencrypt or Cloudflare.

The following procedure uses a self-signed certificate (for demonstration only, not recommended).

Enable Apache SSL HTTPS site configuration and the SSL module:

a2ensite default-ssl.conf
a2enmod ssl

Now modify the port in galaxy virtualhost configuration from 80 to 443 and reload apache. It should now be possible to access the web interface at https://server.ip/galaxy.

4.4 Keep your Galaxy instance up to date#

The Galaxy mailing list provides information on new releases and any security vulnerabilities that have been discovered.

You can use the following command to get any updates that have since been issued after the release (17.09 in this example):

git checkout release_17.09 && git pull --ff-only \
  origin release_17.09

References

Get Galaxy
Running Galaxy in a production environment
Proxying Galaxy with Apache
UFW, Ubuntu Community Wiki
An Introduction To DigitalOcean Cloud Firewalls
PostgreSQL, Ubuntu Server Guide
Apache 2 Web Server, Ubuntu Server Guide
How To Set Up SSH Keys, DigitalOcean

Footnotes

Cover photo by Rafael Cerqueira on Unsplash.

If you are on Windows, you can use an SSH client like Putty_. ↩
When this post was written, the latest stable release of Galaxy was 17.09. Visit https://galaxyproject.org/admin/get-galaxy/ to find the latest stable release. ↩

Docker Images for Bioinformatics Software

2017-08-03T00:00:00+01:00

These are all published in the Docker Hub. If you have Docker installed and working, you can get them using

docker pull vimalkvn/image_name

Most of these images are built using a Dockerfile and so can be customized easily if required. The Dockerfile’s, scripts and associated help (README) can all be found in the sysadminbio under docker-images on GitLab.

Wrapper script to run containers#

Most images have a wrapper script included in their respective directories which can be used to run a container based on the image. For example, in the case of the VIGA program, this script is called run-viga. This is a simple BASH script that calls the docker command like this

docker run --rm \
     -e LOGNAME=${CURRENT_USER} \
     -e USER=${CURRENT_USER} \
     -u ${CURRENT_UID}:${CURRENT_UID} \
     -v /data/databases:/data/databases:ro \
     -v $(pwd):/wdir \
     -w /wdir \
     vimalkvn/viga $*

--rm ensures container is removed after it has been used.

The -e and -u options set the user and group id’s of the files generated by the container to that of the current user i.e., the user running the docker command. Without this, files generated will all be owned by root or another user set in the Docker image.

The -v option is specific for different images and can be modified if necessary. The current directory is also exported set as the working directory (-w) for the container so input files will be accessible and any generated output files will also be saved here.

Note: the wrapper scripts are not necessary to use any of the images.

It is only provided for convenience. Containers can be created from the images directly using the docker run command but this would mean that users will need access to the docker command i.e., they should belong to the docker user group which is equivalent of having root access. That is why this approach would be better. Users will just need to be given sudo access to the wrapper script.

The following is a summary of all available images.

VIGA#

VIGA is a de novo viral genome annotator. This image has all dependencies needed to run VirAnnot including Aragorn, HMMER, BLAST+, Diamond, Prodigal, BioPython, LASTZ, Tandem and Inverted Repeats Finder.

You can pull this image using the command

docker pull vimalkvn/viga

Note: This program was previously called as VirAnnot.

QIIME#

QIIME is an open source bioinformatics pipeline for performing microbiome analysis of raw DNA sequencing data.

The QIIME image has version 1.9.1 installed along with all dependencies and the latest version of R (for Ubuntu 14.04) and packages (ape, biom, optparse, RColorBrewer, randomForest, vegan, DESeq2 and metagenomeSeq)

You can pull this image using the command:

docker pull vimalkvn/qiime

DADA2#

DADA2 is a program for fast, accurate, single-nucleotide resolution for amplicon data.

The DADA2 image has version 1.4.0 of the package installed along with R 3.4.0.

You can pull this image using the command

docker pull vimalkvn/dada2

This container simply runs the Rscript command and expects an R script as the first argument. If a command other than Rscript is required, it can be specified using the --entrypoint docker argument.

PyFBA#

PyFBA is a python implementation of flux balance analysis to model microbial metabolism.

The PyFBA image will have the latest version of PyFBA from Github along with all the dependencies pre-installed - GLPK, PyGLPK, libSBML, lxml, Beautiful Soup 4 and the ModelSEED database.

You can pull this image using the command:

docker pull vimalkvn/pyfba

OpenLava-LSA#

An OpenLava docker image configured for the LSA package.

You can pull this image using the command:

docker pull vimalkvn/docker-openlava-lsa

Please consult the README in the repository for information on usage.

Cutadapt#

Cutadapt is a program for removing adapter sequences from sequencing reads. The cutadapt image has version 1.13 of the package installed.

You can pull this image using the command:

docker pull vimalkvn/cutadapt

If there is a problem with any of these images, you can file an issue on GitLab.

An Unofficial VirtualBox Image for QIIME 1

2017-08-02T10:30:00+01:00

The developers of QIIME provide a VirtualBox disk image for QIIME 1.9.1. Instructions for using this image is available here. This is an alternative (unofficial) image with the same version of QIIME but with some features and enhancements as described below.

Note

The QIIME developers are recommending that new users start using QIIME 2 rather than QIIME 1 (the version this image includes). I had created the original version of this image when I was using QIIME 1 for my work. This image is only provided with the hope that it will be useful for anyone still requiring QIIME 1.

Features and Enhancements#

This image is based on the latest LTS version of Ubuntu 16.04.3 with Ubuntu MATE - a simpler desktop environment with minimal applications and desktop effects disabled for better performance. The default user is qiime with password qiime.

VirtualBox guest additions is pre-installed so features like shared folders, shared clipboard and better video support should all work immediately.

QIIME 1.9.1 and some dependencies (BLAST, Mothur, Clearcut, Infernal, Muscle, Cdbtools, Swarm, ea-utils, SeqPrep, SortmeRNA, BWA, BLAT) are installed using the Conda package manager and an environment is setup for the qiime user called qiime1. This environment is activated automatically on login.

R is installed along with the required packages (ape, biom, optparse, RColorBrewer, randomForest, vegan, DESeq2 and metagenomeSeq).

Additional dependencies installed - RDP Classifier, cd-hit, ChimeraSlayer, rtax, AmpliconNoise, Cytoscape and SourceTracker, thanks to qiime-deploy.

Search and install additional software easily from the Ubuntu and Debian repositories using the inclued Synaptic Package Manager. Alternatively, packages can also be installed using the Conda package manager that is configured for the qiime user. Additional software like USEARCH can be made available by downloading the executable to /home/qiime/Software/bin and then creating a symlink or renaming the executable to the desired name. See tips section below for an example.

Storage space can be increased, if necessary. LVM has been configured for the base installation so hard disk size can be increased if necessary by adding additional disks to the VM in VirtualBox.

Setup and Usage#

Install VirtualBox and the extension pack.
Download the QIIME_1.9.1_UM.ova file.
Click on the downloaded file. If that does not launch VirtualBox, open VirtualBox and then use the File --> Import Appliance option and select the downloaded file.
VirtualBox will offer the possibility to configure the number of CPU’s, the amount of RAM and other options in the next step.

Tips#

Passwords#

Password for the QIIME user qiime is qiime. You can change it using the passwd command.

Becoming root#

The qiime user has admin privileges (sudo) and so logging in to the root account is not normally required. If you do need to become root, use sudo su.

How do I exit the qiime1 conda environment?#

Use the command source deactivate or dt (alias).

Increasing storage space#

Shutdown the VM.
VirtualBox -> Settings - Storage -> Click on the “Adds Harddisk” button next to Controller: SATA. From there, select “Create new disk”, use defaults and select a suitable disk size depending on the space available on the hard disk. Click OK when done.
Start VM, open terminal, use command lsblk to check if the new hard disk is listed.
Use LVM commands to extend and resize the file system:

pvcreate /dev/sdb
vgextend ubuntu-mate-vg /dev/sdb
lvresize -L +10G -r /dev/ubuntu-mate-vg/root

The root (/) filesystem should have now been resized. To check disk space usage, use the command df -h.

Testing#

Here is the output of the print_qiime_config.py command to test the QIIME installation:

(qiime1) qiime@ubuntu-vm:~$ print_qiime_config.py -tvf

System information
==================
Platform:   linux2
Python version: 2.7.13 | packaged by conda-forge | (default, May  2 2017, 12:48:11)  [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
Python executable:  /home/qiime/Software/miniconda3/envs/qiime1/bin/python

QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:
https://github.com/biocore/qiime-default-reference/releases/tag/0.1.3

Dependency versions
===================
        QIIME library version:  1.9.1
         QIIME script version:  1.9.1
  qiime-default-reference version:  0.1.3
                NumPy version:  1.10.4
                SciPy version:  0.17.1
               pandas version:  0.20.3
           matplotlib version:  1.4.3
          biom-format version:  2.1.5
                 h5py version:  2.6.0 (HDF5 version: 1.8.17)
                 qcli version:  0.1.1
                 pyqi version:  0.3.2
           scikit-bio version:  0.2.3
               PyNAST version:  1.2.2
              Emperor version:  0.9.51
              burrito version:  0.9.1
     burrito-fillings version:  0.1.1
            sortmerna version:  SortMeRNA version 2.0, 29/11/2014
            sumaclust version:  SUMACLUST Version 1.0.00
                swarm version:  Swarm 1.2.19 [Jun  1 2017 11:47:02]
                        gdata:  Installed.
RDP Classifier version (if installed):  rdp_classifier-2.2.jar
      Java version (if installed):  Not installed.

QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:
http://qiime.org/install/qiime_config.html
http://qiime.org/tutorials/parallel_qiime.html

             blastmat_dir:  None
  pick_otus_reference_seqs_fp:  /home/qiime/Software/miniconda3/envs/qiime1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                 sc_queue:  all.q
  topiaryexplorer_project_dir:  None
 pynast_template_alignment_fp:  /home/qiime/Software/miniconda3/envs/qiime1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
          cluster_jobs_fp:  start_parallel_jobs.py
pynast_template_alignment_blastdb:  None
assign_taxonomy_reference_seqs_fp:  /home/qiime/Software/miniconda3/envs/qiime1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
             torque_queue:  friendlyq
            jobs_to_start:  1
               slurm_time:  None
    denoiser_min_per_core:  50
assign_taxonomy_id_to_taxonomy_fp:  /home/qiime/Software/miniconda3/envs/qiime1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                 temp_dir:  /tmp/
             slurm_memory:  None
              slurm_queue:  None
              blastall_fp:  blastall
         seconds_to_sleep:  1

QIIME full install test results
===============================
test_FastTree_supported_version (__main__.QIIMEDependencyFull)
FastTree is in path and version is supported ... ok
test_INFERNAL_supported_version (__main__.QIIMEDependencyFull)
INFERNAL is in path and version is supported ... ok
test_R_supported_version (__main__.QIIMEDependencyFull)
R is in path and version is supported ... ok
test_ampliconnoise_install (__main__.QIIMEDependencyFull)
AmpliconNoise install looks sane. ... ok
test_blast_supported_version (__main__.QIIMEDependencyFull)
blast is in path and version is supported ... ok
test_blastall_fp (__main__.QIIMEDependencyFull)
blastall_fp is set to a valid path ... ok
test_blastmat_dir (__main__.QIIMEDependencyFull)
blastmat_dir is set to a valid path. ... ok
test_cdbtools_supported_version (__main__.QIIMEDependencyFull)
cdbtools is in path and version is supported ... ok
test_cdhit_supported_version (__main__.QIIMEDependencyFull)
cd-hit is in path and version is supported ... ok
test_chimeraSlayer_install (__main__.QIIMEDependencyFull)
no obvious problems with ChimeraSlayer install ... ok
test_clearcut_supported_version (__main__.QIIMEDependencyFull)
clearcut is in path and version is supported ... ok
test_cluster_jobs_fp (__main__.QIIMEDependencyFull)
cluster_jobs_fp is set to a valid path and is executable ... ok
test_denoiser_supported_version (__main__.QIIMEDependencyFull)
denoiser aligner is ready to use ... ok
test_for_unrecognized_values (__main__.QIIMEDependencyFull)
qiime_config has no extra values ... ok
test_gdata_install (__main__.QIIMEDependencyFull)
gdata is installed ... ok
test_get_qiime_scripts_dir (__main__.QIIMEDependencyFull)
Test that we can find the directory containing QIIME scripts. ... ok
test_h5py (__main__.QIIMEDependencyFull)
h5py is installed ... ok
test_mothur_supported_version (__main__.QIIMEDependencyFull)
mothur is in path and version is supported ... ok
test_muscle_supported_version (__main__.QIIMEDependencyFull)
muscle is in path and version is supported ... ok
test_pynast_template_alignment_blastdb_fp (__main__.QIIMEDependencyFull)
pynast_template_alignment_blastdb, if set, is set to a valid path ... ok
test_pynast_template_alignment_fp (__main__.QIIMEDependencyFull)
pynast_template_alignment, if set, is set to a valid path ... ok
test_raxmlHPC_supported_version (__main__.QIIMEDependencyFull)
raxmlHPC is in path and version is supported ... ok
test_rtax_supported_version (__main__.QIIMEDependencyFull)
rtax is in path and version is supported ... ok
test_sourcetracker_installed (__main__.QIIMEDependencyFull)
sourcetracker is installed ... ok
test_temp_dir (__main__.QIIMEDependencyFull)
temp_dir is set to a valid path ... ok
test_uclust_supported_version (__main__.QIIMEDependencyFull)
uclust is in path and version is supported ... ok
test_usearch_supported_version (__main__.QIIMEDependencyFull)
usearch is in path and version is supported ... ok

----------------------------------------------------------------------
Ran 27 tests in 0.276s

OK

A Python Script to Update NCBI BLAST Databases

2017-03-05T00:00:00+00:00

The following command will download and/or update the swissprot protein database in the current directory:

update_blastdb --decompress --passive swissprot

Output:

Connected to NCBI
Downloading swissprot.tar.gz... [OK]

Here is the list of downloaded files:

ls
swissprot.tar.gz  swissprot.tar.gz.md5

One issue with this approach is that any long running BLAST jobs currently accessing the database will be aborted. To overcome this problem, I wrote a wrapper around the update_blastdb command - blastdb_updater.py.

It uses a symbolic link to the latest version of the database and only updates the link if the database is not being used. If the database is being used, the script adds a message to the log after the database download is complete. The link can then be updated manually later.

Note
This script will only work on Linux/Unix-like systems due to its dependence on the lsof command to check if a directory is being accessed.

Download python script#

From vimalkvn/sysadminbio repository on GitLab:
Link: blastdb_updater.py

Save this script as blastdb_updater.py under /home/user/programs/blastdb_updater.py (only used for the purpose of the examples below). It can be saved somewhere else.

Usage#

Assuming you would like to download the swissprot database to /home/user/blast, use:

python /home/user/programs/blastdb_updater.py \
  -d swissprot -p /home/user/blast

A log file will be available under
/home/user/blast/log/blastdb_updater.log.

To use the database in your BLAST search, you can use:

blastp -db /home/user/blast/swissprot/swissprot \
  -query sample.fasta

Other databases (supported by update_blastdb) can be downloaded in the same manner.

Automated update#

An automated update can be setup using cron:

MAILTO=email@domain
0 0 1 * * /home/user/programs/blastdb_updater.py \
-d swissprot -p /home/user/blast

The above cron job will update the database on the 1st of every month.

PyQt Interfaces in BioNumerics 6

2010-08-02T21:48:21+01:00

PyQt is a set of Python bindings to Qt — Nokia’s cross-platform GUI toolkit. Qt is mature, versatile and is released under an Open Source license. It is used in software like Google Earth, KDE, Opera, Skype, VLC media player and VirtualBox (Source: Wikipedia).

One of the advantages of using PyQt for graphical user interface (GUI) development is the bundled Qt Designer makes GUI design very easy and efficient. Widgets can be dragged and dropped onto dialogs, main windows and widgets. The resulting XML user interface (.ui) files can be converted to Python code using the scripts included with PyQt.

Using PyQt from within Bionumerics scripts opens up a number of possibilities — use simple message boxes with error or informative texts, create dialogs and display windows for working with databases and so on. Bionumerics includes some modules including BioPython but PyQt will need to be included manually. It is fairly simple to use an existing PyQt installation.

Requirements#

Python 2.6
The latest stable version as of this writing is 2.6.5. It can be downloaded from http://python.org/download/releases/
PyQt
Latest version can be downloaded and installed from http://www.riverbankcomputing.co.uk/software/pyqt/download [under Binary packages]. PyQt version as of this writing is PyQt-Py2.6-gpl-4.7.3-2.exe.
To be able to import PyQt modules from within Bionumerics, the module directory:
C:\Python26\Lib\site-packages
should be added to PYTHONPATH or appended to sys.path before other import statements.
The Qt DLL’s must be available or added to the system PATH. This will be done automatically by the PyQt installer. If the DLL’s are not added to the system path, it will lead to error messages like “DLL load failed” and the script will fail.

Sample Application#

To demonstrate, I am going to use a simple dialog application as an example. This application is used to set the value of a given field of entries selected in the BioNumerics main window. When run, this is how it will appear:

Step 1 — Creating the dialog using Qt Designer#

The dialog displayed above was created in Qt Designer and the file was saved as setfielddlg.ui. The text at the top, Field and Value labels are all of type QLabel. The combo box used to display existing field names from the database is of type QComboBox. The input field corresponding to Value is of type QLineEdit. The Apply, Close and Deselect All buttons are all of type QPushButton.

A list of tutorials on PyQt can be found here. The book Rapid GUI Programming with Python and Qt is an excellent reference on the topic. Also useful is Riverbank’s documentation of PyQt’s classes.

Step 2 — Generating Python code for the Dialog#

The setfielddlg.ui file generated from Qt Designer in the previous step is an XML file describing all the elements of the dialog. This can be converted to Python code using the pyuic4 command included with PyQt4:

pyuic4 -o ui_setfielddlg.py setfielddlg.ui

The -o option specifies the name of the output file. The dialog can now be called from other scripts.

Note

C:\Python26\Lib\site-packages\PyQt4\bin
should be in the system PATH for the pyuic4 command to work.

Step 3 — The main script#

set_field.py

"""A simple dialog application to set the field of selected entries in
Bionumerics 6

"""
import sys
import bns

#append paths to PyQt4 and the current directory to sys.path
moduledir = [r'C:\Python26\Lib\site-packages', r'G:\Python\Bionumerics_scripts\src']

for moddir in moduledir:
        if moddir not in sys.path:
                sys.path.append(moddir)

import dbf
from PyQt4.QtCore import SIGNAL
from PyQt4.QtGui import QApplication, QDialog, QMessageBox
import ui_setfielddlg

class Dlg(QDialog, ui_setfielddlg.Ui_Dialog):

    def __init__(self, fieldnames, parent=None):
        super(Dlg, self).__init__(parent)
        self.setupUi(self)

        self.connect(self.close_button, SIGNAL('clicked()'), self.accept)
        self.connect(self.apply_button, SIGNAL('clicked()'), self.setfield)
        self.connect(self.deselect_button, SIGNAL('clicked()'), self.deselect)

        if fieldnames:
            self.field_combo_box.addItems(fieldnames)
        else:
            QMessageBox.critical(None, 'Error', 'Could not get fieldnames')
            return

    def deselect(self):
        '''Deselect all entries if selected'''

        if len(bns.Database.Db.Selection):
            bns.Database.Db.Selection.Clear()
        else:
            return

    def setfield(self):
                '''Sets the field value of selected entries'''

        field = unicode(self.field_combo_box.currentText())
        value = unicode(self.value_line_edit.text())

        selected = dbf.getSelected()

        if selected:
            for item in selected:
                key = item.Key
                try:
                    bns.Database.EntryField(key, field).Content = value
                except Exception, e:
                    QMessageBox.critical(None, 'Error', str(sys.exc_info()))
                bns.Database.Db.Fields.Save()
                else:
                    QMessageBox.information(None,'No entries selected',
                       'Please select entries before running script')

if __name__ == '__main__':
    sys.__dict__['argv'] = ['argv']
    app = QApplication(sys.argv)
    fields = dbf.getFieldNames()
    dialog = Dlg(fields)
    dialog.show()
    __bnscontext__.Stop(app.exec_())

Step 4 — Running the script#

Source code of all the scripts used here can be downloaded from my GitHub repository.

The scripts set_field.py and ui_setfielddlg.py and the module dbf.py should be in the same directory. In my case, this was G:\Python\Bionumerics_scripts\src and also specified in the moduledir list in the set_field.py script.

The dialog can be called by using the Scripts -> Run script from file option in Bionumerics and selecting set_field.py.

For this to work, database entries must be selected in the main window of BioNumerics or else the dialog displays a message and does nothing. Select the field from the Field combo box, type in the value that needs to be set for the field and hit Apply. The value field has a maximum length of 80 characters, which is the maxiumum length of a field in BioNumerics.

The script in detail#

Some additional notes

Imports#

bns is the BioNumerics Python module
PyQt4 module directories are appended to sys.path, as is the path to the directory of the running script. This is done as to import any additional modules used by the script.
dbf is a module I wrote for general database functions. In this script, it is used for getting a list of selected entries (get_selected) and for getting the list of field names in the database - get_field_names.

ui_setfielddlg contains code for the dialog:

import sys
import bns

#append paths to PyQt4 and the current directory to sys.path
moduledir = [r'C:\Python26\Lib\site-packages', r'G:\Python\Bionumerics_scripts\src']

for moddir in moduledir:
    if moddir not in sys.path:
        sys.path.append(moddir)

import dbf
from PyQt4.QtCore import SIGNAL
from PyQt4.QtGui import QApplication, QDialog, QMessageBox
import ui_setfielddlg

The dialog class — Dlg#

Takes the fieldnames as an argument and populates the Field combo box if it is not empty. The clicked() signal of the close button is connected to the accept slot, which closes the dialog. This is the same signal that is emitted by when the window is closed from the menubar. The other buttons are connected to their respective slots which are defined within the class.

main#

The: sys.__dict__['argv'] = ['argv'] does nothing. For some reason, sys.argv is empty or not defined when running the script from BioNumerics, and the line of the code QApplication(sys.argv) refuses to work without it.

As no commandline arguments are used in the code above, it should cause no harm: __bnscontext__.Stop(app.exec_()) stops the program once the event loop — app.exec_() returns. This is similar to sys.exit(app.exec_()) used in standard PyQt4 programs.

Troubleshooting#

Some issues I am aware of:

Only one instance of the script should run. Calling it twice causes a crash.
The dialog must be closed before exiting Bionumerics or else the program crashes.

Vimalkumar Velayudhan

Installing Conda Using the Miniforge Installer

About Conda#

Step 1: Downloading Miniforge Installer#

Step 2: Running the Installer Script#

Q1: Miniforge License Agreement#

Q2: Where to Install Miniforge#

Q3: Do You Wish to Initialize Conda?#

Check if Conda is Installed Correctly#

Step 3: (Optional) Disabling Auto-Activation of base Environment#

Getting the Lineage of an Organism Using Python, BioSQL, and Taxonomy Data

Introduction#

Step 1: Install Required Software#

Step 2: Create a Database for Storing Taxonomy Data#

Create Database#

Create Database User#

Initialise Database With BioSQL Schema#

Step 3: Import NCBI Taxonomy Data Into Database#

Step 4: Run Python Script to Get Lineage#

Running Galaxy Project on Ubuntu 16.04 LTS

Step 1: Create the virtual server (droplet)#

Step 2: Configure droplet#

2.1 Create user with administrative access#

2.2 Add swap space#

2.3 Install required software#

2.4 Set PostgreSQL administrator password#

2.5 Enable required Apache modules#

2.6 Create Galaxy user accounts and database#

Step 2: Configure the Apache web server#

Step 3: Download, configure and start Galaxy#

Step 4: Improving security and performance, updating Galaxy#

4.1 Secure SSH access#

Change default port#

Disable root login#

Disable password authentication#

Restrict users with SSH access#

4.2 Configure firewall#

4.3 Enable HTTPS#

4.4 Keep your Galaxy instance up to date#

Docker Images for Bioinformatics Software

Wrapper script to run containers#

VIGA#

QIIME#

DADA2#

PyFBA#

OpenLava-LSA#

Cutadapt#

An Unofficial VirtualBox Image for QIIME 1

Features and Enhancements#

Setup and Usage#

Tips#

Passwords#

Becoming root#

How do I exit the qiime1 conda environment?#

Increasing storage space#

Testing#

A Python Script to Update NCBI BLAST Databases

Download python script#

Usage#

Automated update#

PyQt Interfaces in BioNumerics 6

Requirements#

Sample Application#

Step 1 — Creating the dialog using Qt Designer#

Step 2 — Generating Python code for the Dialog#

Step 3 — The main script#

Step 4 — Running the script#

The script in detail#

Imports#

The dialog class — Dlg#

main#

Troubleshooting#