But it can be done, although you will probably have to spend more time integrating them than writing the UI yourself :-)
There is for example an integration effort going on in the jMonkeyEngine: http://hub.jmonkeyengine.org/forum/topic/javafx-embedded-in-jme3/
Many projects/engines use Nifty to write UIs in games: http://nifty-gui.lessvoid.com so that may be an alternative to look at.