For info I'm on Windows 10 64 bit. NVIDIA Quadro 4000 on DirectX 12.0, Direct3D API 11.2
If I go into my NVIDIA control panel and set:
Use advanced 3D image settings, AntiAliasing - FXAA to ON
then it works. But sceneAntialiasingNumPasses is still 1. It looks way better but not quite as good as my old setup under Java3D 1.5.x.
I'm also worried I've made this work on my machine but it's not a general solution for other users.
By the way, JogAmp's Java3D doesn't use DirectX, the obsolete versions of Java3D don't have a consistent behavior and they have plenty of unfixed bugs.
Thank you for the hint about the advanced 3D image settings. It would be nice to know whether the problems comes from Java3D or from JOGL. I advise you to run an existing JOGL unit test about antialiasing.