Complexity determines how much memory is required to render it. In very rough terms, the ray tracer requires a minimum of 2 GB per million triangles rendered, although much more RAM can be required depending on current settings.
To find out how many primitives your scene contains, issue:
ray renderer=2
On a 32-bit Windows XP-based PC, processes are limited to 2 GB of address space regardless of how much memory your computer has. Thus, it is difficult to render a scene with more than a million triangles on such a system.
On 32-bit Linux or Macintosh systems, provided that you have at least 4 GB of RAM, you can usually render scenes up to about 2.5 million triangles.
On 64-bit systems, provided that you use a 64-bit executable, it is the amount of physical memory you have that will typically be limiting. Note that 32-bit executables are limited to 4 GB or less, even when running on 64-bit systems.
The easiest way to reduce scene complexity is to simplify the representations used. Spheres are the simplest representation and require the least memory to render.
set sphere_mode, 1 as spheres ray