When developing an editor it's sometimes hard to focus on performance, as the 'user is king' ethos means a dedication to the user experience that's not always prevalent in the engine that the editor is coupled to. But to meet the requirements to the user, we need to deliver a product that's not a chore to use (and ideally pleasurable); which inherently means it must perform at a requisite level.
The problem though, is that the editor environment is always in a state of flux. New entities are being added, some are being deleted while the rest are potentially being updated. Properties are changing at a high rate of knots - how do we manage all this data and still keep the user experience as slick as possible?
One area in particular that can grind the performance level to its knees is the maintaining of user created geometry. Primitives are added, clipped, merged, materials applied and in the region of many thousands of individual pieces. Brushes, in the classical Quake-sense, are still prevalent today. Most high-end engines support the manipulation of these convex objects, for a myriad of reasons - graphics geometry that makes it into the gold-master, or as a tool to drive visibility, or as a means to reduce physics-mesh complexity, and many more. Granted, a lot of engines target a DCC tool, and simply import data from there, but there's such strong arguments for allowing on-the-fly geometry creation within the editor, particularly if it's a WYSIWYG environment with the implied production benefits.
In our latest project, we have scenes that contain upwards of 5000 brushes. As an average, we can assume each brush is a cubioid with around half the faces assigned to different materials. With our previous method of simply rendering each visible face individually, we can incur the cost of many, many thousands of draw calls - even as much as 20K+. This is simply too high and will make any well meaning system despair.
The idea here is to create a vertex buffer on request that can hold up to 64k verts - the 16bit index limit. It is then cut up into a series of chunks, that are cuboid-sized - 6*4=24 verts. If a brush is like a prism, and covers more than a few chunks there may be some wasted memory space, but having fixed sized chunks makes the allocations and deallocations much quicker. A bit array is then used to determine if a chunk is free or used, with the system creating a new vertex buffer carved up in the same manner if no free space can be found. When a chunk is released, the entire buffer will be discarded if it is empty. A dynamic index buffer is then populated to point to the verts of the visible brush faces - even performing classic back-face removal on the CPU for major wins, while also batching by material. As the system matures, I'll likely add options to defrag the vertex buffers to ensure maximum memory use, but on a resource-rich developer PC, I think this current method is acceptable.
As a result on modest hardware, we now have scenes that totalled 23k draw calls at 6fps, now running with 283 draw calls at 50+fps - much more editing friendly, with full lighting and shadowing enabled.