Skip to main content

Creating a Batch API operation to parse a large CSV file

In this code snippet, I'll show how you can parse a (large) CSV file using Drupal's Batch API. The purpose of batching an operation is to avoid PHP memory limits and time outs. Before you begin, I recommend reviewing the following two articles. Be sure to review the additional batch parameters outlined in the documentation, you might need to use them.

function MYMODULE_callback_csv_import() { // define path to CSV file $csv_file_path = file_directory_path() . '/import_path/myfile.csv'; // define a redirect path upon batch completion $redirect_path = 'admin/import-csv'; // define batch array structure // NOTE: minimal parameters defined to simplify code $batch = array( 'title' => t('Reading File'), 'operations' => array( array( '_MYMODULE_batch_read', array($csv_file_path), ), ), ); // set batch batch_set($batch); // process batch batch_process($redirect_path);
}

Next, we'll define the batch callback function. This function will be called repeatedly until the $context['finished'] variable is set to "1".

function _MYMODULE_batch_read($csv_file_path, &$context) { // define batch limit   $batch_limit = 100; // assume the batch process has not completed $context['finished'] = 0; // open the file for reading $file_handle = fopen($csv_file_path, 'r'); // check if file pointer position exists in the sandbox, and jump to location in file if ($context['sandbox']['file_pointer_position']) { fseek($file_handle, $context['sandbox']['file_pointer_position']); } // loop through the file and stop at batch limit for ($i = 0; $i < $batch_limit; $i++) { // get file line as csv $csv_line = fgetcsv($file_handle); // NOTE: at this point, do what ever you'd like with the CSV array data! if (is_array($csv_line)) { // db_query(), etc } // retain current file pointer position $context['sandbox']['file_pointer_position'] = ftell($file_handle); // check for EOF if (feof($file_handle)) { // complete the batch process $context['finished'] = 1; // end loop break; } }
}