overlaying large, complex datasets in Geoprocessing improved at 10.1

Be successful overlaying large, complex datasets in Geoprocessing

Spatial datasets often have large numbers of features.  Furthermore, these datasets may also contain individual features such as road casings that are very large and complex.

As a result of such large datasets, massive amounts of feature overlap (number of features and complexity of the overlapping features) within feature classes can result in a geoprocessing tool failure or poor performance.  These failures are often due to system resources limitations, mismanagement of resources, “bad” data, or massive overlap in an area that cannot be processed within the available resources.  Failure messages such as “Invalid Topology [4GB_FILE_LIMIT] “, or “Out of memory” can occur.  In rare cases, an application crash is possible.

At 10.1, significant work has been done for managing the processing of large complex data.  We have removed internal limitations and overhauled our overlay engine’s memory management.  Some overlay processes that previously failed can now complete successfully, and with reasonable performance.  However, following these guidelines and best practices should ensure your large overlay processes complete successfully. Continue reading →