Home > linux > Centos 7 systemd temp files

Centos 7 systemd temp files

So this entry is not written to discuss about cons and/or pros of systemd. Apparently systemd owns complete linux its everywhere and you can not get rid of it easily. Here is what happened 2 days ago. I installed centos 7 on my main machine. While debugging one of php project i was working on, i needed to check my debug file which normally i write with “error_log” function:

return error_log($log, 3, "/tmp/ErrorLog")

it was writing to /tmp/ directory but when i tried to tail the file system coulnd’t find it. No entry in nginx or php-fpm log files , no errors but i’m not able to find the file. So i tried to read the file :

$filename="/tmp/ErrorLog";
$myfile = fopen($filename, "r") or die("Unable to open file!");
echo fread($myfile,filesize($filename));

and boom its reading the contents. So i wasted ~45 minutes try to debug my application and then nginx and php-fpm (bottom line this was a fresh install centos 7 so first time im checking this log file)

then i tried to search the file (first time i cancelled the search as it could take long time and decided to check my source code which was mistake)

[root@marko tmp]# find / -name ErrorLog
/tmp/systemd-private-kJHrTu/tmp/ErrorLog

What the fuck?????

So seems like systemd is doing something shady behind my back. Till today i was never curious about systemd as im not a network administrator anymore. As long as i can run my system with my past experiences i do not care about the changes. Instead i focus on software development.

So i started digging into systemd stuff. After 10 sec search i hit to this page

http://fedoraproject.org/wiki/Features/ServicesPrivateTmp

Seems like this proposal is reality now. With one difference it says default value is FALSE which is not the case with centos 7.

So another post mentions

> Also you can get which directories are used by which process with the
> following command:
> 
> $ sudo grep systemd-private /proc/*/mountinfo

So lets try it

[root@marko tmp]# grep systemd-private /proc/*/mountinfo
/proc/20991/mountinfo:271 201 253:0 /tmp/systemd-private-nunEbg/tmp /tmp rw,relatime shared:213 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/20991/mountinfo:273 201 253:0 /var/tmp/systemd-private-JCyn7Y/tmp /var/tmp rw,relatime shared:214 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/21148/mountinfo:271 201 253:0 /tmp/systemd-private-nunEbg/tmp /tmp rw,relatime shared:213 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/21148/mountinfo:273 201 253:0 /var/tmp/systemd-private-JCyn7Y/tmp /var/tmp rw,relatime shared:214 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/2970/mountinfo:110 80 253:0 /tmp/systemd-private-gBrQ6F/tmp /tmp rw,relatime shared:93 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/2970/mountinfo:111 80 253:0 /var/tmp/systemd-private-8kjTJV/tmp /var/tmp rw,relatime shared:94 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3486/mountinfo:151 119 253:0 /tmp/systemd-private-tAXBDZ/tmp /tmp rw,relatime shared:133 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3486/mountinfo:152 119 253:0 /var/tmp/systemd-private-BPzvqo/tmp /var/tmp rw,relatime shared:134 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3689/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3689/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3691/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3691/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3692/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3692/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3693/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3693/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3694/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3694/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3695/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/3695/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/4210/mountinfo:199 164 253:0 /tmp/systemd-private-q1NXco/tmp /tmp rw,relatime shared:178 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/4210/mountinfo:236 164 253:0 /var/tmp/systemd-private-UbiRCk/tmp /var/tmp rw,relatime shared:216 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/4211/mountinfo:199 164 253:0 /tmp/systemd-private-q1NXco/tmp /tmp rw,relatime shared:178 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/4211/mountinfo:236 164 253:0 /var/tmp/systemd-private-UbiRCk/tmp /var/tmp rw,relatime shared:216 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/5000/mountinfo:279 242 253:0 /tmp/systemd-private-kJHrTu/tmp /tmp rw,relatime shared:256 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered
/proc/5000/mountinfo:280 242 253:0 /var/tmp/systemd-private-oHMLFF/tmp /var/tmp rw,relatime shared:257 master:1 - ext4 /dev/mapper/centos-root rw,data=ordered

Not that useful, lets try to find php-fpm pid and try to see the details with findmnt. At least we’ll know what we are looking for:

[root@marko tmp]# ps -ax | grep php-fpm
3689 ?        Ss     0:00 php-fpm: master process (/etc/php-fpm.conf)
3691 ?        S      0:00 php-fpm: pool www
3692 ?        S      0:00 php-fpm: pool www
3693 ?        S      0:00 php-fpm: pool www
3694 ?        S      0:00 php-fpm: pool www
3695 ?        S      0:00 php-fpm: pool www
5000 ?        S      0:00 php-fpm: pool www
5188 pts/3    S+     0:00 grep --color=auto php-fpm

 

[root@marko tmp]# findmnt -N 3689
TARGET                           SOURCE                FSTYPE          OPTIONS
/                                /dev/mapper/centos-root
ext4            rw,relatime,data=ordered
├─/dev                           devtmpfs              devtmpfs        rw,nosuid,size=3950776k,nr_inodes=987694,mode=755
│ ├─/dev/shm                     tmpfs                 tmpfs           rw,nosuid,nodev
│ ├─/dev/pts                     devpts                devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/mqueue                  mqueue                mqueue          rw,relatime
│ └─/dev/hugepages               hugetlbfs             hugetlbfs       rw,relatime
├─/proc                          proc                  proc            rw,nosuid,nodev,noexec,relatime
│ ├─/proc/sys/fs/binfmt_misc     systemd-1             autofs          rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
│ │ └─/proc/sys/fs/binfmt_misc   binfmt_misc           binfmt_misc     rw,relatime
│ └─/proc/fs/nfsd                sunrpc                nfsd            rw,relatime
├─/sys                           sysfs                 sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security         securityfs            securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup               tmpfs                 tmpfs           rw,nosuid,nodev,noexec,mode=755
│ │ ├─/sys/fs/cgroup/systemd     cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
│ │ ├─/sys/fs/cgroup/cpuset      cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,cpuset
│ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,cpuacct,cpu
│ │ ├─/sys/fs/cgroup/memory      cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/devices     cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/freezer     cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,freezer
│ │ ├─/sys/fs/cgroup/net_cls     cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,net_cls
│ │ ├─/sys/fs/cgroup/blkio       cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/perf_event  cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,perf_event
│ │ └─/sys/fs/cgroup/hugetlb     cgroup                cgroup          rw,nosuid,nodev,noexec,relatime,hugetlb
│ ├─/sys/fs/pstore               pstore                pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config           configfs              configfs        rw,relatime
│ ├─/sys/kernel/debug            debugfs               debugfs         rw,relatime
│ └─/sys/fs/fuse/connections     fusectl               fusectl         rw,relatime
├─/run                           tmpfs                 tmpfs           rw,nosuid,nodev,mode=755
│ └─/run/user/1000/gvfs          gvfsd-fuse            fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000

/dev/sda6             ext4            rw,nosuid,nodev,relatime,data=ordered
├─/var/lib/nfs/rpc_pipefs        sunrpc                rpc_pipefs      rw,relatime
├─/boot                          /dev/sda1             ext4            rw,relatime,stripe=4,data=ordered
├─/mnt/media                     //192.168.1.xx/media/ cifs            rw,relatime,vers=1.0,cache=strict,username=admin,domain=LS-W,uid=1000,forceuid,gid=0,noforcegid,addr=192.168.1.xx,file_mode=0644,dir_mode=0755,nounix,serverino,r
├─/mnt/backup1               //192.168.1.xx/backup1/
cifs            rw,relatime,vers=1.0,cache=strict,username=admin,domain=LS-W,uid=1000,forceuid,gid=0,noforcegid,addr=192.168.1.xx,file_mode=0644,dir_mode=0755,nounix,serverino,r
├─/mnt/backup2                     //192.168.1.xx/backup2/ cifs            rw,relatime,vers=1.0,cache=strict,username=admin,domain=LS-W,uid=1000,forceuid,gid=0,noforcegid,addr=192.168.1.xx,file_mode=0644,dir_mode=0755,nounix,serverino,r
├─/tmp                           /dev/mapper/centos-root[/tmp/systemd-private-kJHrTu/tmp]
ext4            rw,relatime,data=ordered
└─/var/tmp                       /dev/mapper/centos-root[/var/tmp/systemd-private-oHMLFF/tmp]
ext4            rw,relatime,data=ordered

 
Now we know its definitely system as last 4 line indicates its writing my file to “/var/tmp/systemd-private-oHMLFF/tmp” without my knowledge. i have to find where are these systemd unit files (basicly service based conf files)

ok they are store “/usr/lib/systemd/system/”

[root@marko tmp]# ls /usr/lib/systemd/system/
[root@marko tmp]# ls /usr/lib/systemd/system/nginx.service
/usr/lib/systemd/system/nginx.service
.
.

And the content of the file:

[root@marko tmp]# cat /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Directive we are looking for is there “PrivateTmp” I wont argue if this is a good feature or not. And do not tell me i have to check change logs etc… basically i should be comfortable with a linux system whenever i use it either its 2000 or 2014.

One last tip files under /tmp/ will be removed if related service is stopped and/or restarted. So you can not store your log files under /tmp/ anymore.

Also during this time (5 days on centos 7) system crashed 1 time pretty interestingly but that story is for some other time

Linux is changing…Its time to move back to BSD system i guess….

Advertisements
  1. Wes
    May 26, 2015 at 1:48 am

    Thanks for the write-up! I just spent two hours pulling my hair out over that one. Apache has the same PrivateTmp setting in CentOS 7, so I copied: /usr/lib/systemd/system/httpd.service => /etc/systemd/system/httpd.service and then set PrivateTmp=false. This was breaking several scripts that used /tmp (the real, system level one) for storage between processes that ran every couple of minutes. Really?!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: