Printer-related segfault

[anderkve]

@idamarie1303 and I have run some GAMBIT-light tests using a python scanner, the hdf5 v2 printer and a very quick likelihood function, and during these tests we sometimes encounter a segfault. Some observations:

• The segfault more frequently appears when running with many MPI processes
• The segfault goes away when setting printer: none
• The segfault seems become less likely (or go away completely?) when we increase the buffer_length setting for the hdf5 printer

Based on this, our current best guess is that this is some sort of race condition problem.

This was using hdf5 version 1.10.10.

[ChrisJChang]

Which python scanner is this? Is this happening only with Ida's code running through gambit light? If it is an issue with fast likelihood functions, we should hopefully be able to reproduce it with the example yamls.

I quickly tried to see if this happened with some of the scipy scanners, and I am finding that they are not working when running with multiple mpi processes. There is an error that it cannot find that scanner. This does not happen when running the random scanner with multiple MPI processes, it the scipy scanners with a single process.

[idamarie1303]

The issue was first discovered when I worked on a new scanner, and used the standard Rosenbrock example in gambit-light.

I tested other python scanners to reproduce the issue, and found an additional warning when using the Nautilus scanner that allowed me to find a different solution. The warning was the following:

UserWarning: h5py is running against HDF5 1.10.10 when it was built against 1.14.4, this may cause problems _warn(("h5py is running against HDF5 {0} when it was built against {1}, "

After uninstalling and reinstalling h5py with the correct HDF5 version I rediscovered a previous issue with the HDF5 file locking: GambitBSM/gambit#398 due to this error:

ERROR: A problem has occurred in the printer utilities. Failed to open existing HDF5 file, then failed to create new one! (runs/gambit_light_example_rosenbrock_scan/samples//results.hdf5). The file may exist but be unreadable. You can check this by trying to inspect it with the 'h5ls' command line tool.

Both the nautilus scanner and the new scanner works as expected after using export HDF5_USE_FILE_LOCKING=FALSE as proposed in GambitBSM/gambit#398.

[anderkve]

I quickly tried to see if this happened with some of the scipy scanners, and I am finding that they are not working when running with multiple mpi processes. There is an error that it cannot find that scanner.

Hmm, strange! Apparently the scipy scanners work in MPI mode for @idamarie1303 , and they have been working for me to. Are you sure you built GAMBIT-light with -DWITH_MPI=True ? Note that what the scipy scanners are supposed to do in MPI mode with N processes is just to run N repetitions of the scan (but not N independent GAMBIT runs, of course -- that I think would happen if you have -DWITH_MPI=False )

[ChrisJChang]

I can confirm that I built with -DWITH_MPI=ON because mpi seems to behave as expected when running the random scanner.

[anderkve]

Strange. I've just confirmed that the scipy scanners (at least scipy_differential_evolution) work as expected in MPI mode also for me. Let us know if find any more details about what is going wrong in your case.

[ChrisJChang]

I confirmed that the problem for me was because I did not have mpi4py installed. This was shown in the logs, but not obvious from the error that gets printed to screen. Now using scipy scanners in mpi mode works as expected.

I might try again to see if I can reproduce the main error that this issue is for.

[anderkve]

Thanks!

We should also look into why you didn't get a sensible error/warning message about missing mpi4py. In our utils library for the Python scanners there is a check that is supposed to inform you about this, either as an error or a warning:

https://github.com/GambitBSM/gambit/blob/72b51ca10a71cc86f5594f97467f1fdd96bb26fe/ScannerBit/src/scanners/python/plugins/utils/mpi.py#L3-L24

But I guess that check isn't being run, so we need to fix that. (Probably a general issue across all the Python scanner plugins.) We should fix this on GAMBIT proper and then sync the fix to GAMBIT-light.

[anderkve]

Ah, I remember now -- I think the error/warning messagees from importing the python scanner plugins are caught and redirected to the logs, since we don't want GAMBIT to stop at startup just because one of the scanner plugins (which may not be in use) is having some problem with some dependencies or something.

But we probably should have some sort of standard MPI check function that is performed just when an MPI-dependent python scanner is actually used, so the relevant error/warning is thrown then. We can probably call this function from the constructor of the given scanner plugin. Anyway, something we can discuss in the Scanner WG meetings. (cc: @gregorydavidmartinez)

[anderkve]

Just to follow up on this one: there is now a PR in GAMBIT that sets HDF5_USE_FILE_LOCKING=FALSE at GAMBIT startup: GambitBSM/gambit#536

So once that PR is merged in I will sync this change to GAMBIT-light.