Extreme slowness when running with PoCL 7.1

[majosm]

grudge has recently started picking up PoCL 7.1 when installing for the downstream tests in pytato, and some of the CI tests have since started running very slowly or hanging. Aside from the CI, I've also been able to reproduce this on my Macbook and on Tuolumne (CPUs; the GPUs there don't use PoCL). mirgecom's CI runs out of time when using 7.1 as well. When I revert to PoCL 6, all of these work fine.

From what I can tell it's only tests that use inverse_metric_derivative or inverse_surface_metric_derivative that encounter the issue (but not all of them do). I took one of the slow/hanging grudge tests (test_dt_utils::test_wave_dt_estimate with dim == 2 and degree == 2) and tried to create a minimal reproducer. Here's what I have so far: reproducer

Some observations:

1. It seems related to calling these inverse metric functions multiple times; when I do that, device operations after it (like the x = actx.from_numpy(...)) end up getting stuck (specifically, they get stuck somewhere inside _enqueue_write_buffer on the C++ side). If I add a self.queue.finish() inside actx.freeze() just after pt_prg(...), then it gets stuck there instead.
2. In the reproducer, if I freeze the calls separately, it runs fast.
3. Most of the tests that get stuck sit there seemingly forever, but the reproducer does eventually finish in 5-10 minutes. And on subsequent runs it's fast until I blow the PoCL cache away.

It sounds like maybe something is slow in the kernel compilation? Though I don't know if that happens inside pt_prg(...) or before.

Here is the generated loopy kernel: code. (I would include the generated OpenCL too if I remembered how to retrieve that...)

I'm not really sure where to go from here. Any ideas @inducer?

[inducer]

Thank you for doing this investigation! This is super helpful. Figuring out why the grudge tests have started dying was on my list of things to do, and this is a huge help getting me started. Here's a slight simplification of the reproducer (mainly strips away pytest stuff):

https://gist.github.com/inducer/c3431739fd84dd13f0daf9b5d0e0b2ba

Running perf top -z shows that all the "hang" time is spent in LLVM, so it's definitely something on the pocl end.

Mildly related: I am tracking another regression in pocl 7 that's making my life difficult.

Ultimately, I think the path forward is fairly clear:

• We'll limit grudge to "pocl<7" with reference to this issue. (I'll file a PR.)
• I will make an OpenCL-only reproducer and take it to the pocl people.

We can also see if there's something straightforward we can do in code generation to work around this issue, but based on what I'm seeing, I'm not a huge fan of that. It appears that it is pocl's regression and should be fixed there.

[inducer]

#397 has the pocl bound, which seems to work around this for now. I'll work on a CL reproducer soon.

[inducer]

I've sent an OpenCL reproducer to the pocl people. We'll see what we hear.