Memory errors and list limits in python

I need to produce large and big (very) matrices (Markov chains) for scientific purposes. I perform calculus that I put in a list of 20301 elements (=one row of my matrix). I need all those data in memory to proceed next Markov step but i can store them elsewhere (eg file) if needed evenif it will slow my Markov chain walkthrough. My computer (scientific lab): Bi-xenon 6 cores/12threads each, 12GB memory, Os: win64

   Traceback (most recent call last):
  File "my_file.py", line 247, in <module>
    ListTemp.append(calculus)
MemoryError

exemple of calculus results: 9.233747520008198e-102 (yes, it’s over 1/9000)

The error is raised when storing the 19766th element

ListTemp[19766]
1.4509421012263216e-103

if I go further

Traceback (most recent call last):
  File "<pyshell#21>", line 1, in <module>
    ListTemp[19767]
IndexError: list index out of range

so this list had a memory error at the 19767 loop.

Questions:

  1. Is there a memory limit to a list? Is it a “by-list limit” or a “global-per-script limit”?

  2. How to bypass those limits? Any lead in mind?

  3. Will it help to use numpy, python64? What are in this case memory limits? What about other languages?

Thanks for your incoming help and advices.

python memory limits

I would like to generate a list of the following: L = list(itertools.product(‘…’, repeat=8)) the call itertools.product(…) generates about 1’099’511’627’776 elements of length 8 (I know its a b

various memory errors, python

I am using Python, but recently I am running a lot into the memory errors. One is related to saving the plots in .png format. As soon as I try to save them in .pdf format I don’t have this problem any

python list memory usage

I am trying to improve the memory usage of my script in python, therefore I need to know what’s RAM usage of my list. I measure the memory usage with print str(sys.getsizeof(my_list)/1024/1024) which

Memory Limits

I have an app which I have created to copy users (10,000+) from one domain and create the user in another domain with custom properties. The app seems to run fine until it hits a 1.7gb of memory, I kn

Specifying memory limits with hadoop

I am trying to run a high-memory job on a Hadoop cluster (0.20.203). I modified the mapred-site.xml to enforce some memory limits. <property> <name>mapred.cluster.max.map.memory.mb</na

how to check memory limits

I am trying to check if the memory limits in my OS works. So I am using this is simple c program: #include <iostream> #include <cstdlib> int main() { const size_t GB = 1024 * 1024 * 1024;

mathematical limits in python?

I am trying to do mathematical limits in python. I have defined a function for smoke import turtle t = turtle.Pen() def drawsmoke(y): i = 0 while i < ((2 * y) – 1): t.seth(i * 5) t.circle((10 + i),

Understanding Python fork and memory allocation errors

I have a memory intensive Python application (between hundreds of MB to several GB). I have a couple of VERY SMALL Linux executables the main application needs to run, e.g. child = Popen(make html,

shared memory in windows limits (c++)

Currently I’m working on a solution for the memory limits per process. So I came to shared memory. First, I’m using windows 7 with visual studio as developer platform, the software will run on a moder

How GAE emulator limits list of available Python modules?

I installed Python Mock module using PIP. When I try to import mock running under ‘dev_appserver’, GAE says that it can’t find module ‘mock’. import mock works perfectly in Python interpreter. I under

Answers

There is no memory limit imposed by Python. However, you will get a MemoryError if you run out of RAM. you say you have 20301 elements in the list. This seems too small to cause a memory error for simple data types (e.g. int), but if each element itself is an object that takes up a lot of memory, you may well be running out of memory.

The IndexError however is probably caused because your ListTemp has got only 19767 elements (indexed 0 to 19766), and you are trying to access past the last element.

It is hard to say what you can do to avoid hitting the limit without knowing exactly what it is that you are trying to do. Using numpy might help. It looks like you are storing a huge amount of data. It may be that you don’t need to store all of it at every stage. But it is impossible to say without knowing.

The MemoryError exception that you are seeing is the direct result of running out of available RAM. This could be caused by either the 2GB per program limit imposed my Windows (32bit programs), or lack of available RAM on your computer. (This link is to a previous question).

You should be able to extend the 2GB by using 64bit copy of Python, provided you are using a 64bit copy of windows.

The IndexError would be caused because Python hit the MemoryError exception before calculating the entire array. Again this is a memory issue.

To get around this problem you could try to use a 64bit copy of Python or better still find a way to write you results to file. To this end look at numpy’s memory mapped arrays.

You should be able to run you entire set of calculation into one of these arrays as the actual data will be written disk, and only a small portion of it held in memory.

First off, see How Big can a Python Array Get? and Numpy, problem with long arrays.

Second, the only real limit comes from the amount of memory you have and how your system stores memory references. There is no per-list limit, so Python will go until it runs out of memory. Two possibilities:

  1. If you are running on an older OS or one that forces processes to use a limited amount of memory, you may need to increase the amount of memory the Python process has access to.
  2. Break the list apart using chunking. For example, do the first 1000 elements of the list, pickle and save them to disk, and then do the next 1000. To work with them, unpickle one chunk at a time so that you don’t run out of memory. This is essentially the same technique that databases use to work with more data than will fit in RAM.