CSV import - record limit

[nkendrick]

I am trying to import about 8000 records in csv format but only about 2000-2500 seem to go in at a time. I am aware of the file size and max upload limits in php.ini but as the server is also processing email these are set to 50MB, and memory size limit is 30MB. I have tons more data to process so any chance of a fix?

What gives?

Thanks

[jmaxwell]

Does it give any error?

Or does it just end up at a blank page?

I have a similar issue.

[longreach]

It is apparently some kind of memory leak. All you can do is max out the memory available to the session, and then identify the biggest chunk size you can process, and then chunk it up. Pretty ugly, but I know of no one who has it working any better than this.

[jcoates]

Has there been any change or modification made to address this limitation?

[nkendrick]

Does it give any error?

Or does it just end up at a blank page?

I have a similar issue.


Nope the import 'finishes' without any error, it's just that not all the records have been brought in.

[jmaxwell]

That is odd.

I have heard that you need to cut up the import.

I would look at using phpMyAdmin to bulk load into the database.

[longreach]

Nope - you can't do that for a lot of the data - as the record ID fields that refer to other tables need to be generated by the Sugar code.

[longreach]

With maximum memory allocations, you can often import about 2-300 objects at a time - that seems to be about the limit. 700 is a more normal limit with regular memory allocation.

[johnol]

Ran into the same problem with 30,000 records.

Sugar's import is a real memory hog and it needs time to process. You need to bump up your max_execution_time, max_input_time, and memory_limit settings *significantly*...

So, in your php.ini try this:
max_execution_time = 3000 ; Maximum execution time of each script, in seconds
max_input_time = 6000 ; Maximum amount of time each script may spend parsing request data
memory_limit = 386M

Even with *these* settings - I still had to break up my import into 5,000 record chunks...

Memory is a major player in this. In other words, once I put my execution_time and input_time to these levels, any further increase didn't help. However, memory *did*. So, if you can allocate a gig of ram, I'll be you'll be able to import your 8,000 records in one shot.

-John

www.progresspays.com

[tcleu]

Hi!

I'm encoutering similar problems. I noticed an entry of:

'import_max_execution_time' => 3600

in the config.php file for sugar. What is this config value for?

and how does this interact with the 'max_' values in php.ini?

Thanks!

-TC

[tcleu]

Hi!

Just wanted to make this complete as I discover stuff. Not meaning to bring back a 1-1/2 yr old issue.

My connection timeout setting in httpd.conf also played a huge part:

Timeout 3000

The default was 300 second which caused the import of opportunities to "halt" after 5 minutes.

- TC