Installing MPI-AMRVAC
This is a step-by step instruction manual on how to install the MPI-AMRVAC code on your computer.

This manual is based on several assumptions:
  1. That you are working on a Linux machine or any Unix-based system.
  2. That you have some experience with programming languages and that you have worked on a Unix/Linux type operating system before.


Step 1
First of all, you will need to install on your system:
  1. A Fortran compiler capable of processing Fortran 90/95 code (E.g. ifort).
  2. A C-compiler capable of processing C++ code (E.g. icc).
  3. An MPI library, e.g. Open MPI (Ensuring that it is set to use the compilers mentioned under 1 and 2)
  4. Perl
  5. Subversion

Do not forget to make sure that the compilers are added to your PATH and that the compiler libraries are added to your LD_LIBRARY_PATH.
You should also make sure that your current directory is part of your PATH.
E.g. If you are working in a Bash shell, you should add the following lines to your '~/.bashrc' file (version numbers and locations must be checked for your system):

PATH="/opt/intel/cc/10.1.018/bin:$PATH"
PATH="/opt/intel/fc/10.1.018/bin:$PATH"
PATH="/opt/openmpi/bin:$PATH"
PATH=".:$PATH"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/cc/10.1.018/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/fc/10.1.018/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/mkl/10.0.4.023/lib

Keep in mind that the exact paths may be different on your system!
(If you are working on a large computer using a module system, setting paths is done automatically when the relevant modules are loaded.)

To make sure that Open MPI uses the correct compilers go to your Open MPI directory and type:

configure CC=icc CXX=icpc F77=ifort FC=ifort

A little ASCII text file which contains some notes on how to properly install the combination ifort/openMPI on several systems is found here, with no garantuee that it will solve all your problems.

Step 2
In order to download the MPI-AMRVAC you need a username/password combination for getting the code from the subversion repository. You can use the following combination studCMFA09 and cpa9amrvac. You may wish to let any of the developers know that you downloaded the code. The mentioned combination will just let you check out and update your code version, and does not give you any rights to make changes to the software. In case you want to contribute, contact us, we may then consider creating a username/password combination with more priviliges.

Step 3
Open a terminal on your computer, go to your home directory and type:

svn co https://svn.esat.kuleuven.be/amrvac/trunk --username myusername mpiamrvac

The myusername is the username mentioned above, and this will prompt you for the password mentioned above.
Once the system is satisfied as to your authorization, this command will create a directory 'mpiamrvac' and use Subversion to install the latest version of the mpi-amrvac code in that directory.

Step 4
Go to the source-directory of the newly downloaded mpi-amrvac code by typing

cd mpiamrvac/src

In this directory create a local file called:

'amrvacsettings.t'

This file should consist of two lines:

! the following are set by amrvac!!!
INTEGER, PARAMETER :: ixGlo^D=1, ixGhi1=16, ixGhi2=16, ixGhi3=16

Step 5
Check and, if necessary, modify the makefile for the Fortran 90/MPI compiler used on your system. This means providing correct settings for:

F90=
FFLAGS=
F90FLAGS=

E.g. For the ifort/OpenMPI combination suggested under Step 1 this should be:

F90=mpif90
FFLAGS= -c
F90FLAGS= -FR -implicitnone -O3 -Wl, -rpath -Wl,${LD_LIBRARY_PATH}

For some simulations it may be necessary to switch from -O3 to -O2, although we do not recommend this. For debugging purposes the flags -DB and -traceback can be added.


Step 6
Now configure the code to run the default test: 2D advection of the VAC logo in a slab geometry by typing in the 'src' directory:

setamrvac -d=22 -phi=0 -z=0 -p=rho -u=testrho -g=16,16 -cp=openmpi -s

The command setamrvac has a number of options:
  1. -d sets the dimensionality and is followed by two integers. The first integer gives the number of dimensions of the grid, the second gives the number of dimensions of the vector quantities
  2. -phi is only relevant in cylindrical symmetry. It is followed by a single integer and sets whether 'phi'  is the second or the third component of the geometry. Set to zero as it is here, it is a dummy argument
  3. -z is similar to -phi, but for the z-component of a cylindrical grid.
  4. -p chooses what physics is used in the simulation. It is followed by a string. In this case rho which means that the problem is treated as purely advection.
  5. -u chooses the name of the userfile that contains the initial conditions of the simulation. It is followed by a string. In this case testrho, which indicates the file './usr/amrvacusr.t.testrho'
  6. -g sets the size of each grid (16x16 in this case). It is followed by a series of integers, separated by commas. The number of integers has to be equal to the number of dimensions. Each integer (best divisible by 4 for some of the error estimates) is the sum of the number of actual gridcells along an axis and the number of ghostcells along each axis (two on each side of the grid). It is not necessary for the grids to have the same number of cells in each direction.
  7. -s tells the code to report the options chosen on the screen.
  8. -cp informs the code on which MPI implementation you are using. Actually, this flag only distinguishes between -cp=mpt and settings different from this (e.g. -cp=openmpi) in a manner which circumvents a known bug for the SGI MPI implementation called MPT.
Step 7
Still in the 'src' directory you should now compile the code by typing:

make clean amrvac

The clean option deletes the '.f', '.o' and '.mod' files from the previous setup. The amrvac option tells the 'makefile' to compile the code in its current settings and create the executable file 'amrvac' in the 'mpiamrvac' directory.

Step 8

Now go to the 'mpiamrvac' directory (cd ..). Here you have to create a symbolic link  called 'amrvac.par' to the par-file that is associated with your particular problem. This is necessary because mpi-amrvac will automatically read its input parameters from 'amrvac.par'.
The command to create this link is (for the 2D testrho simulation that we have just compiled in steps 6 and 7):

ln -s par/testrho/testrho_vac22 amrvac.par

Step 9
Now the code can be run with the command:

mpirun -np 1 amrvac

The option -np followed by a space and then an integer specifies the number of processors on which the code will be run (In this example only one, and it is in principle also possible to then run directly using the executable amrvac. Further note that depending on your MPI implementation, the mpirun command may need to be replaced by mpiexec). You should now check your result against the description of this example (and start with the data analysis and conversion).