Large Pages and Locked Pages

By Alexander Yee

 

(Last updated: April 13, 2016)

 

 

y-cruncher v0.7.1 and later supports the use of large pages and locked pages for memory allocation.

Both of these features were added as part of a refactoring of the memory subsystem between v0.6.9 and v0.7.1. They are optional features that are not required for the program to function properly. But y-cruncher will attempt to use them automatically and fall back to normal pages upon failure.

 

Both features can be explicitly disabled from the Custom Compute menu.

 

 

Large Pages:

 

Large page support was added to see if there was any performance that could be squeezed out. Unfortunately, the situation for y-cruncher is not the same as for database applications. For y-cruncher, the speedup is negligible to 2% at most. But these were tested on small desktops. The situation may be different for large workstations that have hundreds of gigabytes of memory.

 

 

Locked Pages:

 

Locked page support was added to solve the problem of overzealous paging on Linux when running swap mode computations. At some point between Ubuntu 14.10 and Ubuntu 15.10, there was a change to the kernel that makes the OS (by default) aggressively swap out processes even when there is enough memory to hold it all in ram. This happens only in swap mode - possibly a consequence of the OS trying to increase the disk cache.

 

The problem is that y-cruncher's performance can degrade by orders of magnitude if even a small part of its memory is paged out to disk. When y-cruncher allocates memory, it expects it to stay in memory, and it uses it like it is physical memory. When such access patterns are applied to disk as a result of paging, only bad things can happen. If the page file is located on the same mechanical hard drive that the system resides on, it usually leads to a system freeze which can only be fixed by a hard shutdown. Things are better for SSDs, but that's just treating the symptoms rather than the cause.

 

It's possible to configure the OS to not swap by either disabling the pagefile or setting the swappiness to zero. y-cruncher attempts to solve it transparently by automatically locking pages into memory if possible. Ultimately, the OS is trying to be too smart. While this is beneficial for "normal" programs, it gets in the way of specialized applications like y-cruncher which manually handle disk swapping.

 

This problem currently does not exist on Windows. Unlike Windows, the Linux kernel does not respect the raw I/O flag (O_DIRECT) with regards to completely disabling the disk cache for a particular file handle/descriptor.

 

Why bother with swap mode at all? Why not just let the OS do it for you via paging? This is answered in the swap mode FAQ.

 

 

Support:

 

Support for large pages and locked pages varies between Windows and Linux. On Linux, large and locked pages are independent from each other. On Windows, large pages are always locked, but not vice versa. Furthermore, y-cruncher currently only supports them together. It cannot lock pages that are not also large pages.

 

Large page support requires manual configuration of the OS to enable. In Windows, you need to give yourself a user privilege. On Linux, you need to install a package.

y-cruncher does not attempt to do either of these automatically - since that'd be a bit shady for a program that only crunches numbers.

 

For both Windows and Linux, locking pages in memory is a privileged operation that requires elevation. Since large pages are also locked in Windows, this means that large pages also require privilege elevation. On Linux, large pages do not require elevation to use, but they do require elevation to enable.

 

 

Windows

 

On Windows, large and locked pages are tied together. You get either both or none. While it's possible to get locked pages that are not large pages, y-cruncher currently does not support it.

 

There are 3 requirements for large (and locked) pages to work on Windows:

  1. You need to have the "SeLockMemoryPrivilege" token. This privilege is disabled for all users by default. You will need to enable it manually.
  2. y-cruncher needs to be running as an administrator.
  3. There needs to be enough contiguous physical memory in the system.

 

Enabling SeLockMemoryPrivilege:

 

"SeLockMemoryPrivilege" is disabled for all users by default. So you will likely need to enable it manually.

 

Instructions for enabling it using the Group Policy Editor can be found here: https://msdn.microsoft.com/en-us/library/ms190730.aspx

If your version of Windows is a Home Edition and does not have the Group Policy Editor, you can also enable it by running the following program:

 

Enable "SeLockMemoryPrivilege" on Windows:

 

Run y-cruncher as an Administrator:

 

Right-click the program and select "Run as administrator". You may get a UAC warning depending on your settings.

 

 

 

Make sure there is sufficient contiguous physical memory:

 

There's no way to tell other than to try and run the desired computation. If it works, then it works. If not, then you may need to try rebooting the system.

 

Windows can only allocate large pages when there is sufficient contiguous memory. But it does not appear to be able to defragment the memory on the fly. This means that the only way to defragment the memory is to reboot the system.

 

Support for large pages in Windows is unfortunately very flaky and there is no guarantee that it will ever work.

 

 

When large and locked pages are working, it will look like the image to the right. Note the presence of the "(locked, 2.00 MiB pages)" tag next to the memory sizes.

 

When it doesn't work, the tag will be missing. If you are missing the privilege, there will be a red warning that reads,

Unable to acquire the permission, "SeLockMemoryPrivilege".

Large pages and page locking will not be possible.

This will happen if you don't run y-cruncher as administrator or if the user account doesn't have the "SeLockMemoryPrivilege" token.

 

If you don't see the warning, and the tag is missing, then there isn't enough contiguous physical memory and you will need to try rebooting.

 

 

 

Linux

 

On Linux, there are more options. Large pages and locked pages are independent and can be enabled/disabled separately.

 

Requirements for Locked Pages:

  1. y-cruncher needs to be running with privilege elevation.
  2. There needs to be sufficient memory available.

Requirements for Large Pages:

  1. Large pages are either reserved or enabled transparently in the OS.
  2. There needs to be sufficient memory available.

 

Run y-cruncher with privilege elevation:

sudo ./y-cruncher

 

Enable large pages in the OS:

 

Install the hugepages package:

sudo apt-get install hugepages

Enable large pages:

sudo hugeadm --pool-pages-max=DEFAULT:30G

 

(Replace "30G" with however much you want. Don't worry about setting it too high, it will clip down to what is available.)

At this point, y-cruncher will be able to use large pages provided that the physical memory space isn't too fragmented. This behavior is the same as in Windows, except that it's more likely to succeed in Linux. Perhaps Linux attempts to defragment the physical memory.

 

These instructions enable what is called "Transparent Huge Pages". If it ever fails, y-cruncher will gracefully fallback to normal pages. There are other ways to enable large pages. For example, they can be reserved at startup which prevents them from being fragmented. Should you decide to go this route, be aware that y-cruncher does not properly detect memory when huge pages are reserved this way. So you may get warnings.

 

For example, suppose you have 32 GB of ram and you reserve 28 GB of it for large pages. This can be done either at startup or by running:

sudo hugeadm --pool-pages-min=DEFAULT:28G

 

(Note the bold "min" to distinguish it from the earlier command.)

Now when you run y-cruncher, it will think that you have less than 4 GB of memory left since it can only detect memory that's available for normal pages. So if you try compute 5 billion digits of Pi (which requires 24GB of memory), it will spit warnings at you saying there isn't enough memory. But if you ignore all those warnings and try to run it anyway, it will work fine.

 

 

Just as with Windows, there will be a tag indicating that large and/or locked pages are working. Likewise, there will be a warning when y-cruncher doesn't have sufficient privileges to lock pages in memory.