Large Data Sets: Processing Guidelines for Economists

This paper lists guidelines for analysts working with large data sets intended for computer storage and manipulation. Particularly when processing very large data sets, organization and planning of data collection, data preparation, and loading are extremely important. Considerable loss of time and money may result from oversights such as inconsistency in naming conventions or variable scales. Even a handful of aberrant record formats in a data set of 2000 variables could later require extra time for set merging. Careful preparation helps avoid such errors and inconvenience.