Java3D doesn't directly override JOGL canvases and windows unlike JogAmp's Ardor3D Continuation, it's a fundamental difference between them. Therefore, don't expect to find a trivial workaround except if you succeed in disabling HiDPI within AWT or Java2D:
But I noticed that Java3D works fine in Mac OS X in Retina display. Doesn't that mean that a similar technique to canvas.setSurfaceScale is already implemented in Java3D? Looks like it's just missing in Windows.
No, no similar technique is implemented in Java3D because it can't be implemented in Java3D. Your supposition is completely absurd, sorry to be harsh. Canvas3D and JCanvas3D don't extend JOGL canvases and have remained mostly unchanged for years, they both were written lots of years before the release of the first hardware supporting HiDPI and a long time before it became supported by OpenJDK and Oracle Java.
Keep in mind that the support of HiDPI depends on the version of the operation system and on the version of Java. Moreover, it depends on Java2D too, it can fail to detect it, the stability of this feature depends on the maturity of the JRE.
As far as I know, the JEP 263 concerns HiDPI support in Windows and Linux, HiDPI was already supported under OS X in Java 1.8. I can't be 100% sure but I think that either HiDPI is disabled when you run Java3D on your Mac or HiDPI is enabled and AWT returns the right information only under OS X but not under Windows.
In Windows I'm tried both the latest version and older version from 2 years ago.
The latest version I used is: Java3D 1.7 and JOGL 2.3.2
In Mac I didn't upgrade to the latest version because the old version was fine. So I'm not sure if the latest version still works in Mac. I can try it when I return home (I'm away in Canada right now).
Sorry for not jumping in earlier on this topic, but I've never had a need to use HiDPi scaling.
An intial look at the code doesn't show me anything that make OSX work with this scale.
I would expect it to be in Screen3D or maybe something related to DrawingSurfaceObject or GraphicsConfigInfo.
Possibly it's already in GraphicsConfiguration and just needs to be used when creating the context.
I'll look through the commits on github and see if I can see anything about this.
If I can't find anything existing now I'll have to examine Ardor3D and try to retro fit that solution into Java3D. However which version number it will be released under will be a problem. 1.6 is fixed now, but 1.7 requires major changes to package names to adopt...