64 bit computing, Python, Numpy and Pandas

I have been using 64 bit Windows 7, on my work laptop. It seems a robust system. Better than Vista or buggy XP 64. As my work machine a Dell Precision M90 is about 5 years old. There isn’t much of a performance improvement as the motherboard chipset limits the memory use to 3.25GB.

I have been doing a lot of work on time series data, particularly half hourly files and iterating over these and processing them. Using Pandas for this has been invaluable. If I batch process the time series data ( it is structured into one folder a month ) and run process several months worth of files, I end up hitting memory problems once the commit size of the process nears 2GB on Python 32 bit, we have some more powerful workstations at work ( one with 32GB ) and I ran the process again and hit the same problem, I realized it must be a cap in 32 bit Python, so I installed 64 bit python and got the unofficial Windows 64 bit binaries of the libraries I needed and no memory issues.

64 bit computing is a revelation when using a proper 64 bit stack and decent hardware. It is invaluable when working with large data sets.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s