Hey @Dafftt
I love your question and Dr. Jack Daniels was right in highlighting this. Back in the days when we did not have any wearables, coaches and scientists tried to find a way to systemize training. They were trying to figure out what kind of training works. They had many different philosophies, ideas and anecdotal evidence on what had worked with their athletes well before. So in come scientists, and they try to come up with models that could help coaches, athletes and scientists create training programs to elevate athletes - and bring home medals. 
Decades later, we have all kinds of gadgets, tracking methods, and models to predict Fitness, Fatigue, and Form. Are we smarter now? Perhaps.
What Dr. Daniels is saying in the video, is that Fitness is essentially a combo of physiological aspects like cardiovascular, pulmonary, cellular functions. Some of them may lose their best ability to circulate blood, oxygenate and deoxygenate blood, ability to fire fast, etc etc while other aspects of fitness may take longer to lose “fitness”. Any model of human physiology has many degrees of errors because it’s nearly impossible to give a universal rate of decline to all these systems even in a single HUMAN being, yet alone groups of individuals. Humans are not robots. We have to remember that.
My take from he video is that Dr. Daniels is relaying a message that you should not freak out about having a day, two, or ten days off. Go have that vacation and don’t stress about it! Don’t hurry back after an illness. Put in the work when you feel good, train, YES! of course we need to put in the work. But when life happens, especially as recreational athletes - learn to not stress about it. (This is a note to myself as much as for you). If overtraining taught me anything it is this:
Sometimes, the missing piece of the puzzle is recovery. No matter how much more you could train, no matter how much harder you could train, but if you don’t give recovery enough time - you’re just digging yourself deeper into the muddy pit that is Overtraining.
Don’t fear rest.
Happy resting.
Marjaana