Hi Tiofred,
I've run tests on my Core2 Quad @2.4GHz, 8GB RAM, GeForce 8800 GTX, Windows 7 64, and I get about 15 million colored, non-textured triangles per second when I use vertex buffers as described at
http://wadeawalker.wordpress.com/2010/10/17/tutorial-faster-rendering-with-vertex-buffer-objects/. This is re-copying the whole vertex buffer to the card every frame, so if the geometry wasn't changing, it would be much faster.
For your scene on my machine, 10,000 boxes means 6 * 2 * 10,000 = 120,000 triangles, so with one camera I'd expect 15M/120K = 125 frames/second. With four cameras, 125/4 = 31.25 FPS.
The fact that you're getting many-second delays from a scene with only 120K triangles in it seems to imply some algorithm or data structure problem, rather than an OpenGL API or hardware problem

Have you tried profiling your application to see where the time is being spent?