Seeing these vulnerability issues on Dogfood every night

[getvictor]

@lucasmrod : It does seem benign, so we should log error as debug and continue the for loop.

fleet/server/vulnerabilities/msrc/sync.go

Line 106 in 7ac39e2

if err := fsClient.Delete(d); err != nil {

Repro/QA

1. Enroll two hosts with different builds of the same version of Windows.
2. Run the vulnerabilities cron to pull the appropriate MSRC file.
3. Rename the MSRC file to match yesterday's date.
4. Run the vulnerabilities cron again.

Repro will show the above error. Fixed version won't.

[iansltx]

Digging deeper on this, as the deletion should only be picking up files that exist, so if we're attempting to delete files that we can't find when deleting something else may be at play. Still troubleshooting this.

[iansltx]

@rfairburn Why might these files be getting deleted between the time we calculate a delta between files to download/delete and actually deleting the files? We're consistently seeing this once per day, with one file failing to delete on one hour (~1:22a UTC) and one failing to delete the next hour (~2:22a UTC). Given that we run the vulns cron hourly, this seems odd. Before downgrading the error (see the associated PR), I want to know why we're seeing this.

[rfairburn]

Does Dogfood already have the fix that prevents alerts from repeating every time a cron runs until the service is restarted? I'm pretty sure that made it into the RC, but not sure if that version of the RC has been applied to Dogfood yet.

This could have been a one-off thing for any number of reasons (we don't have persistent or shared storage at all for example as containers are intended to be stateless as much as possible), but would alert every cron interval with the same error if the alerting fix has not been deployed yet.

[iansltx]

The info above was from CloudWatch Logs, not alerts, and it only happens 2x per day (but happens 2x every day), so I don't think it's one-off, nor is it related to the the repeating alerts thing...I think?

[iansltx]

@rfairburn Your theory on this being due to multiple matches to the same MSRC file to delete was a sound one. Different builds of the same version of Windows is the culprit here (which is why you didn't see this in every environment). The included PR fixes that issue; thanks for the assist here!