[PUP-8548] File isn't synced to disk Created: 2018/03/14  Updated: 2018/05/15  Resolved: 2018/05/15

Status: Closed
Project: Puppet
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: Matthias Baur Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 1
Labels: container, triaged, type_and_provider
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:
  • OpenStack instance on KVM (SDN: Midonet/SDS: Quobyte)
    • Ubuntu 16.04.1
    • Docker 1.13.1
    • Puppet 5.4.0

Template: PUP Bug Template
Team: Platform OS
Method Found: Needs Assessment
QA Risk Assessment: Needs Assessment

 Description   

We're running our Puppet tests with `apply` in docker containers. From time to time we're seeing strange errors which seem to come from apt source list files not being on disk as apt::update is triggered.

Apt doesn't know anything about the additional repository:

root@9af454802315:/# ls -la /etc/apt/sources.list.d/zabbix.list
-rw-r--r-- 1 root root 114 Mar 14 05:49 /etc/apt/sources.list.d/zabbix.list
root@9af454802315:/# apt-cache policy zabbix-server-mysql
zabbix-server-mysql:
  Installed: 1:2.4.7+dfsg-2ubuntu2.1
  Candidate: 1:2.4.7+dfsg-2ubuntu2.1
  Version table:
 *** 1:2.4.7+dfsg-2ubuntu2.1 500
        500 https://mirror.syseleven.de/ubuntu xenial-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     1:2.4.7+dfsg-2ubuntu2 500
        500 https://mirror.syseleven.de/ubuntu xenial/universe amd64 Packages
root@9af454802315:/# apt-get update
Ign:1 http://apt.puppetlabs.com xenial InRelease
Hit:2 http://apt.puppetlabs.com xenial Release
Hit:3 https://mirror.syseleven.de/ubuntu xenial-backports InRelease
Hit:4 https://mirror.syseleven.de/ubuntu xenial-security InRelease
Hit:5 https://mirror.syseleven.de/ubuntu xenial-updates InRelease
Hit:6 https://mirror.syseleven.de/ubuntu xenial InRelease
Hit:8 http://repo.percona.com/apt xenial InRelease
Get:9 http://repo.zabbix.com/zabbix/3.0/ubuntu xenial InRelease [7093 B]
Get:10 http://repo.zabbix.com/zabbix/3.0/ubuntu xenial/main amd64 Packages [2444 B]
Fetched 9537 B in 1s (8328 B/s)
Reading package lists... Done
root@9af454802315:/# apt-cache policy zabbix-server-mysql
zabbix-server-mysql:
  Installed: 1:2.4.7+dfsg-2ubuntu2.1
  Candidate: 1:3.0.15-1+xenial
  Version table:
     1:3.0.15-1+xenial 500
        500 http://repo.zabbix.com/zabbix/3.0/ubuntu xenial/main amd64 Packages
 *** 1:2.4.7+dfsg-2ubuntu2.1 500
        500 https://mirror.syseleven.de/ubuntu xenial-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     1:2.4.7+dfsg-2ubuntu2 500
        500 https://mirror.syseleven.de/ubuntu xenial/universe amd64 Packages

But Puppet tells us that the file got placed before triggering apt::update, so apt should know about the repository:

Notice: /Stage[main]/Zabbix::Repo/Apt::Source[zabbix]/Apt::Key[Add key: FBABD5FB20255ECAB22EE194D13D58E479EA5ED4 from Apt::Source zabbix]/Apt_key[Add key: FBABD5FB20255ECAB22
3D58E479EA5ED4 from Apt::Source zabbix]/ensure: created
Notice: /Stage[main]/Zabbix::Repo/Apt::Source[zabbix]/Apt::Setting[list-zabbix]/File[/etc/apt/sources.list.d/zabbix.list]/ensure: defined content as '{md5}f5c0f867946b96c7706d35083c118733'
Notice: /Stage[main]/Sys11mysql::Repo/Apt::Source[percona]/Apt::Key[Add key: 4D1BB29D63D98E422B2113B19334A25F8507EFA5 from Apt::Source percona]/Apt_key[Add key: 4D1BB29D63D98E422B2113B19334A25F8507EFA5 from Apt::Source percona]/ensure: created
Notice: /Stage[main]/Sys11mysql::Repo/Apt::Source[percona]/Apt::Setting[list-percona]/File[/etc/apt/sources.list.d/percona.list]/ensure: defined content as '{md5}fbbedc4eba5a288aa968a3d20e94bb36'
Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Nagios::Nrpe/File[/etc/nagios/nrpe.cfg]/owner: owner changed 'root' to 'nagios'
Notice: /Stage[main]/Nagios::Nrpe/File[/etc/nagios/nrpe.cfg]/group: group changed 'root' to 'nagios'
Notice: /Stage[main]/Nagios::Nrpe/File[/var/cache/nagios_results]/ensure: created
Notice: /Stage[main]/Nagios::Nrpe/File[/usr/local/bin/sys11-nrpe]/ensure: defined content as '{md5}2deb57ccac742ed57bdd421266ef7a9b'
Notice: /Stage[main]/Zabbix::Server/Package[zabbix-server-mysql]/ensure: created

This seems to be a sync to disk problem. Futher debugging in https://github.com/puppetlabs/puppet/blob/master/lib/puppet/type/file/content.rb#L124 showed that the `sync` (https://ruby-doc.org/core-2.2.2/IO.html#method-i-sync) parameter of the file object is set to 'false'. Should this be set to true?



 Comments   
Comment by Matthias Baur [ 2018/03/14 ]

Hmm just found https://github.com/puppetlabs/puppet/blob/master/lib/puppet/type/file.rb#L860-L863 so seem like the flush to the disk really is happening. Still we see this error from time to time, do you have any idea where this might come from?

Comment by Branan Riley [ 2018/05/14 ]

Is your `/etc/` mounted through any sort of indirection? I wonder if this is actually something in the underlying filesystem that's reporting stale data back.

Comment by Matthias Baur [ 2018/05/15 ]

Not exactly, '/' is on one partition on the Docker Host and in the Docker Container. We did a lot of changes on our CI lately including changing the Docker storage driver from aufs to overlay2, switching from Jenkins to Gitlab CI, ...
After all those changes we haven't seem the error in the last couple of weeks.

 

Still, the issue seems kind of odd.

Comment by Branan Riley [ 2018/05/15 ]

Doing a bit of googling I see several old and closed issues against Docker related to filesystem issues (mostly at build time) - if there was some sort of FS issue under the hood in Docker, it could definitely also affect puppet runs.

Since you can't reproduce this in your current set of deployments, I'm going to go ahead and close for now. If you can come up with any new data that makes it look like a Puppet issue again, feel free to reopen.

Generated at Mon Oct 14 06:01:20 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.