Log recursion detection feature

[ktff]

Kubernetes and Docker share the same issue of collecting their own log which could cause a positive feedback loop of logs. Currently both are using different ways of detecting vector logs. The way we are currently doing it in Kubernetes was exposed to have a hole #2171.

We could potentially solve this problem for both sources by implementing a recursion detection based on logs. We would log a unique string of characters with println!, and once we detect a log with the same string of characters from a container, we would start filtering out it's logs.

Ref #2188.

[LucioFranco]

I'd actually like us to explore the namespace solution as we talked about here #2188 (comment)

[lukesteensen]

Is this still an issue if we limit ourselves to k8s 1.14+?

This seems like it would require significant complexity to do well, and be potentially impossible to do perfectly. I'd prefer we avoid situations where it becomes a problem if possible.

[ktff]

It isn't, and I agree that it's probably not worth it.

[Hoverbear]

@lukesteensen shall we close this with a wontfix then?

[lukesteensen]

@Hoverbear That sounds good to me! Unless @ktff had another reason for reopening?

[ktff]

@lukesteensen I wanted to leave this decision to @MOZGIII . But I am for closing it.

[MOZGIII]

Sorry I took too long with the feedback on this matter. I wanted to think about it for a bit.

So, my view on this is some generic way of detecting recursively consumed stream would be great. Not just in the context of k8s or docker, and just for vector-generated logs. With log aggregation, there are situations where misconfiguration leads to the same logs being processed in a cycle, resulting in extreme traffic and flood. If would be very cool if we could detect that and react accordingly.

Speaking of vector reading logs of itself, it's on its own is not a problem - it's the situation I outlined above is what we would really want to prevent. So, if the rate of logging that vector does is not directly connected to the volume of input it gets - we should be fine, as it won't result it cyclic flood. However, that invariant doesn't hold currently, and there's no accurate way to guarantee that, so we won't be able to offer it as a reliable property of vector.

I went a bit off, but back to the case at hand.

I do think the approach proposed here has its rationale, but I don't think we should rely on it in docker and k8s to exclude vector logs from processing. We should use a different, more explicit approach for those scenarios.

If considered independently of docker or k8s, this should probably have a composability capabilities of a transform, rather than be baked into the sources. This brings me to an idea of sources composed of modular pieces - but more on that another time.

Considering all of the above, I'm going to close this issue, with the following considerations:

• we don't want it for k8s and/or docker integration, as we can use more explicit approaches that would prevent this from happening in the first place;
• we probably want to explore ways we can deal with cycles in log processing; even though we're pretty good at preventing cycles via our topology design, we can't completely prevent cycles caused by external parties; but we can investigate what can we do to prevent vector from going down under flood when that happens.

Thanks @ktff for creating this issue.