Case study
Why your tool calls a long job stuck when it is still running
Twice in one week a video platform told a team an ad had stopped while it was still being made. The fix was to ask the step holding the job before calling it lost, instead of deciding from a clock.
Key takeaways
- A stuck warning based on elapsed time alone will be wrong on the longest and most queued jobs.
- Ask whether the tool checks with the step holding the job before it calls the job lost.
- A restart can add the waiting time again, and the earlier attempt's finished output may be thrown away.
- A full queue can look like a broken one, especially on busy days.
- When one step is fixed, ask whether the other steps that handle long work decide the same way.
A job marked stuck but still running is worse than one that has really stopped, because the obvious response is to start it again. Twice in one week a client's video platform told the team an ad had stopped while it was still being made. Both faults were found and fixed quickly: one was traced the day after it appeared, the other the same day.
Why a job marked stuck but still running costs a business
The cost of a false stuck warning is paid twice in waiting time. The person running the job does the sensible thing and starts it again, so the business waits for the second attempt while the earlier one is still working. On a busy day a restart also takes a place in the line, and a job closed while still running holds that place while doing nothing useful, so the work behind it waits for a slot that will not free. After one or two false alarms, people stop trusting the warnings that are real, and a genuine failure can sit unseen.
What the team saw: two false alarms in one week
The earlier warning came on an unusually long ad, close to twenty minutes, whose final step draws every frame into the finished video. That step ran for more than three hours. After two and a half hours the platform said the video had stopped moving. The person running it started the ad again. The second attempt took over two hours; the earlier one finished after all, and its video was thrown away as an outdated attempt. The ad reached the team about an hour and a half later than it would have without the warning.
Two days later the same kind of false alarm came at a different step, where the presenter's clips are made. An ad waited there for most of a working day before the platform closed it as lost. The clips for that ad were still in line to be made, now for nobody. The ad behind it was held back, because the line still counted the closed one as running. And the team could not start the closed ad again for the rest of the day, because the step still listed it as in hand.
What our engineer found: a clock deciding instead of the work
The warning came from the clock, not from the step saying it had lost the job. A rule looked only at elapsed time: if nothing had touched the video for two and a half hours, it was presumed lost. A safeguard existed that should have kept a long job marked alive while the step drawing it was busy. Our engineer found when that safeguard ran: only while other videos were waiting in the line behind. That afternoon nothing else was waiting, so it never ran, and the clock rule fired on a job that was nearly done.
The second case had the same root. The presenter step's time limit had been worked out from how long an ad takes when the service making the clips has room. That week its capacity was used up and every clip waited its turn, so the step took many times longer than the estimate. The clock said lost; the step itself still had the ad in its hands and kept it there, which is why the line behaved as though the ad was running.
What we changed
We made both steps report what they hold, and made the platform ask before it gives up on an ad. The drawing step now says, from the moment it takes a video until it hands the finished file back, whether the video is waiting its turn or being drawn, and how far it has got. Every minute the platform looks at any video that has been quiet for about ten minutes and asks. If the video is held, however long it takes, it is marked alive, so the team's screen does not call it stalled. If nothing holds it and it is still quiet a moment later, it is marked lost straight away with a clear reason, and can be started again at once.
At the presenter step, once the quiet time passes its limit, the platform asks the step whether the ad is still among its jobs. While it is, the ad is waiting in line, not lost, and it stays open, so no clips are made for nobody and the line does not count a closed job as still running. Where a step cannot answer, the old rule still applies, so nothing is left without a safety net, and a missing reading of how busy a step is now shows as unknown, never as idle.
What the business has now
Both changes are live on the client's platform. A long video is no longer called stuck while it is still being drawn, and one that really was lost is reported as lost within minutes instead of after a long wait. An ad waiting for its presenter's clips on a busy day stays open until those clips are made, so the team does not lose the ad, its place in line, or the rest of the day to a false alarm. From the moment the drawing step takes a long video, the team's screen shows that the video is held and whether it is waiting its turn or being drawn, so a quiet spell on screen no longer looks like a stop.
What you can check yourself
- When a warning says a job has stopped, look for signs it is still waiting its turn: a queued status, a step that still lists it, or progress that moved recently.
- Ask whoever looks after the tool what the warning rests on: a fixed time limit, or a direct answer from the step doing the work.
- Check whether a step that has taken a job tells the rest of the system it is holding it, or goes quiet until it finishes.
- Look at a job closed as lost: are its remaining tasks still queued, and is the job behind it held back?
- Compare the stuck warnings on a busy day with a quiet one; a full line can look like a broken one.
What to do next
If a stuck warning has already led to a restart, keep the history of both attempts before anyone clears it, so you can see which one finished and when. Ask your developer or the tool's support whether the warning comes from a time limit alone, and whether the step doing the work is asked before its job is called lost. Then ask the same question of every step that handles long work. Tracing a warning back to the rule behind it, and changing the rule to ask before it gives up, is the kind of work our engineers do for clients.
Questions people ask
Why does my video tool say a job is stuck when it is still processing?
Often because the software is judging silence rather than asking the step that holds the job. A long drawing step, or one waiting for capacity, may not report anything for a while, and a time limit then calls it lost. The work can still be running while the screen has stopped changing.
Should I restart a job as soon as my software says it has stopped?
Not straight away. First see whether the job is queued, held, or actually failed. Starting a second attempt can add the same waiting time again, and the earlier attempt's result may be thrown away as outdated when it arrives. Restart when the tool gives a clear reason that the job is lost.
Why do more jobs look stuck on my busiest days?
Busy days fill the line, so each step waits longer than it would on a quiet day, and a time limit set from a quiet day fires on jobs that are still moving slowly. A queue can look broken when it is only full.
What should a stuck warning be based on?
A direct answer from the step that holds the job, rather than a fixed time limit on its own. The tool should ask whether the step still has the job before calling it lost. If a step cannot answer, a fallback rule is sensible, as long as it is treated as a fallback.
What happens to the earlier attempt if I start a job again?
It depends on the tool. Some let the earlier attempt run and discard its output when the second finishes; others cancel it at once. Before you restart, ask what becomes of the earlier attempt and of any work it has already done.