# Mpirun and ff-mpirun works in Mac but not in Ubuntu

**URL:** <https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889>\
**Category:** General Discussion\
**Created:** [July 12, 2022, 8:37am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889 "2022-07-12T08:37:50Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 12, 2022, 8:37am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/1 "2022-07-12T08:37:50Z")

</div>

I have the following python code `solveMPI.py` to run an embarrasingly parallel job

```auto
import numpy as np
from mpi4py import MPI
# import os
import subprocess

comm = MPI.COMM_WORLD
size = comm.Get_size()

def loop(rank):
    # ffcmd = "ff-mpirun random_source.edp -nProcess {} -process {}".format(size, rank)
    # print(ffcmd)
    # os.system(ffcmd)
    subprocess.run(["ff-mpirun", "random_source.edp", "-nProcess", str(size), "-process", str(rank)], shell=False)

def parallel():
    rank = comm.Get_rank()
    loop(rank)

if __name__ == ' __main__':
    parallel()

```

Then I run the python code by `mpirun -np 10 python3 solveMPI.py`.

It works perfectly well in Mac OS. But in Ubuntu, the command does nothing but just print the following text

```auto
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 1
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 4
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 6
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 3
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 8
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 9
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 5
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 0
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 2
'/usr/bin/mpiexec' --oversubscribe /usr/local/bin/FreeFem++-mpi -nw 'random_source.edp' -nProcess 10 -process 7

```

If I replace the command in `subprocess.run` by simple ones such as `date`, then it produces expected output. I searched the internet for answers but coudn’t solve the problem. Hope experts here can help me!

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 12, 2022, 8:41am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/2 "2022-07-12T08:41:16Z")

</div>

BTW, If I copy one of the 10 commands and paste it into the terminal, then it runs ok.

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [July 12, 2022, 9:16am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/3 "2022-07-12T09:16:48Z")

</div>

Then it probably means that those binaries are not in your path.

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 12, 2022, 9:23am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/4 "2022-07-12T09:23:08Z")

</div>

I checked and they are in my path.

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 12, 2022, 10:12am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/5 "2022-07-12T10:12:58Z")

</div>

If I change the `subprocess` line to

```auto
subprocess.run(["ff-mpirun", "--version"], shell=False)

```

Then I get the expected output

```auto
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
'/usr/bin/mpiexec' --oversubscribe --version
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/
mpiexec (OpenRTE) 4.1.2

Report bugs to http://www.open-mpi.org/community/help/

```

Once the `--version` is replaced by an `edp` file, the command will not be executed.

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [July 12, 2022, 4:04pm UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/6 "2022-07-12T16:04:17Z")

</div>

You are missing the parameter `-np 1`.

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 13, 2022, 4:16am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/7 "2022-07-13T04:16:38Z")

</div>

Adding `-np 1` makes no difference. Pasting the command in terminal works but not the Python script. Exactly the same script works in Mac. I’ll check again to see if it’s the Python problem.

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 13, 2022, 5:29am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/8 "2022-07-13T05:29:03Z")

</div>

I replaced the Python script with a C++ code:

```auto
#include<stdlib.h>
#include<string.h>
#include <mpi.h>

using namespace std;

int main(int argc, char** argv) {
  int rank, n_ranks;

  // First call MPI_Init
  MPI_Init(&argc, &argv);
  // Get my rank and the number of ranks
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &n_ranks);

  string cmd = "ff-mpirun random_source.edp -nProcess " + to_string(n_ranks) + " -process " + to_string(rank) + " -ne 0";
  cout << cmd << endl;
  system(cmd.c_str());

  // Call finalize at the end
  return MPI_Finalize();
}

```

Again the command is print but not executed. I have no idea now.

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [July 13, 2022, 6:25am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/9 "2022-07-13T06:25:54Z")

</div>

No problem whatsoever on my end, I just tried with a dummy `random_source.edp` that just prints to screen.

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 13, 2022, 6:41am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/10 "2022-07-13T06:41:11Z")

</div>

What is your linux distribution and FreeFem version?

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [July 13, 2022, 6:50am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/11 "2022-07-13T06:50:44Z")

</div>

Debian Sid with FreeFEM develop branch.

---

<div class="post-metadata">

**Author:** ![Jade](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@Jade](https://community.freefem.org/u/Jade)\
**Post date:** [July 14, 2022, 11:43am UTC](https://community.freefem.org/t/mpirun-and-ff-mpirun-works-in-mac-but-not-in-ubuntu/1889/12 "2022-07-14T11:43:04Z")

</div>

I also tried the following dummy `test.edp` file:

```auto
cout << "This is a test!" << endl << endl;

```

Then

```auto
os.system("FreeFem++ test.edp")

```

gives the correct output. But

```auto
os.system("ff-mpirun -np 1 test.edp")

```

just prints the command itself. Since `ff-mpirun` calls `FreeFem++-mpi`, I test the code

```auto
os.system("FreeFem++-mpi -nw test.edp")

```

This time it gives a long error message

```auto
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  getting local rank failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)

```

Finally, all 3 commands give correct output if they are executed directly in the terminal.