LebGeeks

user

Member

OS cache/memory management question

I was not sure if this belongs in the software section or here. It has little to nothing to do with actual code. I have a complicated question about the operating system caching.
I will start with what I know: The operating system caches any file that was opened at least once if any free memory exists, to avoid going to hard disk and perform io which is expensive. Memory cache has no drawbacks because it can be emptied extremely quickly to make room for data that need it more.

Here is my hypothetical question: assume you have duplicate files, lets say you set up several instances of teamspeak server lets say or you just copied an entire game twice on your pc. You opened that software once from one directory and closed it. The files are in the OS cache now(or some of them anyway) you then opened that same software from the other directory. Does the os cache recognize that these are the same files with the same content that it already has cached or does it open them again from the new directory?

And another question about memory in general. If I open the game twice from different directories, do all the files get loaded twice in memory or can it load certain files once and use them for both?

rolf

Member

Re: OS cache/memory management question

I am no specialist but I will answer based on my knowledge about computers in General.
I have a book called "Inside Windows 2000" which goes into great details about such processes, but I never got far in the lecture of this book. It is very technical (you can borrow it if you want).
So my answers are just what I think the OS probably does.

For the memory cache, I don't think the OS would recognize that these are the same file, because AFAIK it doesn't look at the contents of the file, so it would not know if it is the same or not. Plus it would not make any sense for it to analyze files in such way, because it is a performance penalty and introduces added complexity and would only help for rare cases.

For your second question: same thing, I guess. The exception would be shared libraries which are only loaded once. Also if the software (the game in your example) is a little "smart", and many software do that nowadays, it will load only once, and if you try to start a second instance it would not accept to do that.

user

Member

Re: OS cache/memory management question

There are quick ways to recognize the file as being the same. One very quick way I can think of is compare the filename+ the size(down to the last byte)+modification date(up to the second) a quick 4 KB read could potentially save the memory from loading a 100 MB file(more or less) There is a risk of error in this though...

I will test this and find out. Best way to test is use slow flash usb and a 4 GB program, any memory cache will be very noticeable.

Last edited by user (June 18 2014)

Joe

Member

Re: OS cache/memory management question

And another question about memory in general. If I open the game twice from different directories, do all the files get loaded twice in memory or can it load certain files once and use them for both?

I can answer this one to the best of my knowledge (which is not that much). It basically depends on the app, and how it's coded.

If a program is using shared libraries, they will be loaded only once in memory. Unfortunately, too many apps (especially on Windows and OSX) tend to avoid shared libraries because they can be very difficult to get right. Most users care more about ease of installation over performance, and as a result programmers tend to bundle all the libs in the executable, so the user gets all the binaries she needs at once and the app becomes easier to run.

Now I read that modern OS are able to deal with duplicate static libraries in an intelligent manner. I cannot find the link I read, if I do, I'll copy it here. OS don't rely on file names or file size to determine duplicates, but rather on some sort of signature comparison. I think.

rolf

Member

Re: OS cache/memory management question

Thing is, let's say two clones of the same apps are run at the same time, and they load the same file. Now let's image for a moment that the OS is "smart" and detects that it's two exact copies of the same file, and loads only one, with the other one just being a pointer.
Now let's say one of the two instances *writes* to this file. No the OS has to separate the files again into two copies.
This complexity alone would be, in my opinion, enough reason to avoid implementing such a system.

nuclearcat

Member

Re: OS cache/memory management question

http://www.linux-kvm.org/page/KSM
might be interesting for you.

hussam

Member

Re: OS cache/memory management question

http://www.linux-kvm.org/page/KSM
might be interesting for you.

NC, my kernel (compiled from source) is configured with CONFIG_KSM=y
I did

echo 1 > /sys/kernel/mm/ksm/run

and I set a systemd tmpfile to automate this at next boot.

ps aux | grep ksmd
root        22  0.0  0.0      0     0 ?        SN   Jul15   0:00 [ksmd]
root      7922  0.0  0.0  13140  2272 pts/2    S+   00:30   0:00 grep ksmd

Now after a few hours:

grep -H "" /sys/kernel/mm/ksm/*
/sys/kernel/mm/ksm/full_scans:0
/sys/kernel/mm/ksm/merge_across_nodes:1
/sys/kernel/mm/ksm/pages_shared:0
/sys/kernel/mm/ksm/pages_sharing:0
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/pages_unshared:0
/sys/kernel/mm/ksm/pages_volatile:0
/sys/kernel/mm/ksm/run:1
/sys/kernel/mm/ksm/sleep_millisecs:20

pages_shared is 0

Does this mean it is not working?

Last edited by hussam (July 17 2014)

rolf

Member

Re: OS cache/memory management question

The doc says that it will only merge pages when explicitely called by an application on a memory area. My guess is very few apps bother using this, so who knows? The only way of being sure is running a test binary (creating one yourself, possibly) and checking these stats again.

hussam

Member

Re: OS cache/memory management question

It seems to be working on my ancient android phone.

root@ace:/ # grep -H "" /sys/kernel/mm/ksm/*
/sys/kernel/mm/ksm/deferred_timer:1
/sys/kernel/mm/ksm/full_scans:7
/sys/kernel/mm/ksm/pages_shared:791
/sys/kernel/mm/ksm/pages_sharing:7508
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/pages_unshared:30078
/sys/kernel/mm/ksm/pages_volatile:89597
/sys/kernel/mm/ksm/run:1
/sys/kernel/mm/ksm/sleep_millisecs:500

uname -a
Linux localhost 3.0.101-ge1e0e31 #1 PREEMPT Wed Jul 16 01:04:38 PDT 2014 armv7l GNU/Linux

Ok, thank you rolf :)