Case study
Why jobs finish out of order and older ones wait all day
Ads queued in the evening were still running the next day while newer ones finished. There was no queue: a free slot went to whichever job asked first. What we found, what we changed, and the questions to ask about any shared tool.
Key takeaways
- A job that waits all day while newer ones finish points to a missing queue, not automatically to too little capacity.
- A first come, first served order has to be built; without it, a free slot goes to whichever job asks first.
- Showing in line separately from generating answers most of a team's questions before they are asked.
- Finished or stuck jobs can keep holding capacity if nothing releases it when they stop needing it.
- Alert on long waits as well as failures, because a waiting job never raises an error.
A job marked running looks busy. When work queued that morning is already finished and work queued the night before is still going, the status tells the team nothing, and nobody knows whether to wait, start again or give up. A client's in-house video team came to us with exactly that: ads queued in the evening were still marked as running half a day to a whole day later, while ads queued later the same night, and even that morning, had already finished.
What the team saw: running all day, while newer work finished
Every ad showed the same word, running, whether it was being made or only waiting its turn. Half a day to a whole day after queueing, some ads still showed it.
The screen gave no hint of who was next, so the order looked random. Ads queued later were done; ads queued earlier were not. The only way for the team to find out anything was to keep looking.
Why waiting work goes wrong in any business
When people cannot see where their work stands, they stop trusting the tool: they start again, ask around or give up. That holds for a video tool, a booking slot, a machine on a shop floor or a shared account anyone on the team can use.
The order matters as much as the speed. A job that waits a day while newer jobs finish makes every delivery time the team has promised worthless, and the waiting is only half of it. The not knowing is the rest.
A first come, first served queue does not happen by itself. Something has to remember the order, hold a place for each job while it waits, and give the next free slot to the job that has waited longest. Where nobody built that, capacity goes to whichever waiting job asks first, and an old job can wait indefinitely.
What our engineers found: plenty of waiting, no line at all
Our engineers went through the history of each ad that had waited, and found two separate problems: too little capacity overnight, and no order in how that capacity was handed out.
The capacity part was simple. Overnight, only one of the accounts that make the AI video clips could work, so only one ad at a time could be made. That explained slow progress, but not why newer ads overtook older ones.
The order was the real finding. Every waiting ad checked about once a minute whether a slot had opened, and whichever happened to ask first after a slot freed up took it. There was no line. One editor's ads queued just after midnight were finished by the morning, while two ads the same editor had queued a few minutes earlier still waited past lunch.
The same history showed slots held by ads that no longer needed them. An ad with all its clips in, waiting only for the final assembly, still counted as busy. So did a run that had been restarted. And one ad with a single failed clip held its slot for most of a day, sitting in running status while others queued behind it.
What we changed
We built a real line, and made the status say what is actually happening.
- Each ad takes its place when it is first queued and keeps it while it waits, even if it is restarted, and a free slot always goes to the ad that has waited longest.
- An ad leaves the line when it starts, finishes, fails or is stopped.
- Ads waiting only for their final assembly, or with all their clips in, no longer hold a slot.
- The team can see whether an ad is in line or generating, how many ads are ahead of it, how many accounts are working right now and how long it has waited. Once its turn comes, it shows how many clips are made so far.
- The ads list says in line or generating instead of running for both.
- When any ad has waited more than three hours, the team gets one alert saying how many are waiting and why: no account can work right now, or every account that can work is busy.
What the business has now
The change is live on the client's platform. Ads are made in the order they were queued, nobody's ad is overtaken by a newer one, and capacity is no longer held by ads that do not need it.
The team sees in plain words where each ad stands, and hears about a long wait within hours rather than finding it the next morning. We traced and fixed it within a day of the team reporting it, and the same day we nudged the few ads that had stopped checking for a slot on their own.
What you can check yourself
- Ask your team or your supplier: when two jobs wait for one free slot, which one gets it? If nobody can answer, there is likely no line.
- Look at a running job's status. Does it say whether the job is being worked on, or only that it is waiting?
- Find a job you know has finished and see whether it is still counted against capacity.
- Ask what happens to a job's place if it is restarted: does it keep its position or go to the back of the line?
- Ask whether anyone is told when a job has waited a long time, or only when it fails.
What to do next
Start with the two questions worth asking: what order are jobs made in, and what does the status actually mean. Ask them of your team if the tool is in-house, or of the supplier if you rent it.
If jobs are being overtaken and capacity is short, fix the order first. Adding capacity to a tool with no line usually means newer jobs still jump ahead, only faster.
Giving a shared tool a real order, and a status that says where each job stands, is the kind of work our engineers do for clients. Whether you build, buy or inherit the next tool, ask the ordering question early: it is far easier to answer before a team is waiting.
Questions people ask
Why does my job show as running when nothing is happening?
Because one status often covers two different things: a job being worked on, and a job waiting for a free slot. If the tool has a single word for both, a job that has waited all day looks exactly like one that started a minute ago. Ask what the status actually means, and whether the tool can show waiting separately from working.
How can I tell a slow job from a stuck one?
Two things help: whether the tool shows progress that changes over time, and whether you can see what the job is waiting for. A job that has made no progress for a long stretch while newer jobs finish usually points to an ordering problem rather than a slow job. Ask your supplier what a waiting job should look like on screen.
Should I add more capacity or fix the order first?
In our experience, fix the order first. Extra capacity with no line means the newest job can still ask first and still overtake one that has waited for hours. Once the line is real, you can see whether capacity is the remaining problem.
How long should a job be allowed to wait before someone is told?
There is no single right answer; it depends on the work and on what the team expects. What matters is that someone is told at all, because a job that waits all day never raises an error and so never appears as a failure. Pick a wait longer than normal for your work, and have the team alerted when it is crossed.
Can a finished job still be blocking others?
Yes. A job that has done its main work but is still waiting for a final step, or one that was restarted, can keep counting as busy and hold capacity it no longer needs. Ask whether the tool releases capacity as soon as a job stops needing it.