A Step-by-Step ServerlessOps Drill: The Duplicate Storm

A Step-by-Step ServerlessOps Drill: The Duplicate Storm

Flight students learn stalls and engine-out landings in the practice area, with an instructor beside them, long before they ever meet one by surprise. The reason is simple: rehearsed responses hold up under pressure, and improvised ones do not. Software teaches a parallel lesson, and this how-to is about it. In ServerlessOps, the Ops Atlas calls one rehearsal the duplicate storm, a deliberate flood of repeated messages into a test copy of your workflow to see whether anything happens twice. It sits at the third rung of the Atlas’s own maturity scale, the one it names “Safe,” so passing it is a genuine milestone.

Expertise is optional here. What you need is a test environment, a workflow that does something you could count, and an hour or two of quiet. Let us go through it step by step, in the order I would teach it.

Before you start: what you are testing

Quick background, because the drill makes more sense with it. Many function platforms deliver events “at least once.” That means a message is guaranteed to show up, and occasionally it will show up more than once. The Atlas explains that repeats come from retries after timeouts, from a slow first worker letting the queue hand the message to someone else, and from operators sending held messages back through for another try. The workflow must therefore be idempotent, a long word for “doing it again changes nothing the first run did not already change.”

The drill proves, with a count, that your workflow has that property. Not an argument, not a code review: a count.

Step 1: pick one workflow with a countable effect

Choose something that leaves a trace you can tally, such as “send a lesson reminder,” “record a booking” or “add a credit to a student’s account.” Avoid workflows whose effect cannot be counted. If your first thought is “everything,” narrow it down. A drill on one workflow teaches more than a vague one on twenty. If you want a mental model, think of a single manoeuvre practised until it is clean, instead of a whole flight attempted in one go.

Step 2: set up a safe copy

Create a test version of the workflow that points to harmless targets. Emails should go to a mailbox you control, payments to a sandbox, and records to a scratch table. The test must never touch real customers. Real money or real messages turn a learning exercise into an incident, which is the opposite of the point.

Step 3: write down the expected count

Before you send anything, decide what the correct result is. If you will submit fifty distinct orders, you expect fifty reminders, fifty records, and fifty credits. Writing it down first stops you from rationalising whatever you observe afterwards. Instructors do the same thing before a lesson: they state what a good approach looks like before you fly it.

Step 4: choose your business key

Every duplicate check needs an identifier that stays the same across repeats. The Atlas recommends a business key taken from the real event, such as the order reference or the booking number, and cautions against the platform’s own message identifier, because held messages sent back for another attempt receive fresh ones. Write your key down. If your workflow does not have one yet, adding it is the first improvement the drill will tell you to make.

While you are at it, note how long your system remembers that it has handled a key. The Atlas gives two reference points: a widely used helper library keeps such records for one hour by default, while the Atlas’s own example sets fifteen days. A payment provider it mentions discards its own keys after twenty-four hours. The memory must outlast the longest delay a repeated message could suffer, so pick a figure on purpose.

Step 5: fire the storm

Now the fun part. Send your fifty distinct messages, and then send them again. And again. A simple script is enough: loop over the same fifty messages three to five times, with some in quick bursts and some spaced out by several minutes. If your queue offers a de-duplication feature, test with a gap longer than its window, because the Atlas notes that the ordered queue type it examines stops recognising repeats after just five minutes. Mix in a handful of deliberately sluggish messages, so the function overruns its limit and the platform launches a retry. Finally, if you can, send some held messages back through a redrive, so they arrive with new identifiers.

Resist the urge to tidy up while it runs. You want the messy, overlapping behaviour that real life produces.

Step 6: count, compare, and write it down

Once everything has settled, count the effects. How many reminders arrived? How many records exist? What do the credits add up to? Compare each to the expected count from Step 3.

  • Counts match exactly: the workflow passed this drill. Record the date, the key, and the memory period you used.
  • Some effects doubled: find which repeat type caused them. Timeout-then-retry and redrive are the usual suspects.
  • Some effects went missing: check the failure queue and the logs for messages that exhausted their attempts.

The Atlas defines success at this level as zero double effects. Anything else is a finding to fix, which is a good result, because you found it on a quiet afternoon.

A worked example of reading the results

Suppose your imaginary flying-club app records lesson bookings and emails a confirmation. You send fifty distinct bookings, three times each. After the storm you find fifty booking records, which is right, but seventy-one confirmation emails, which is not. The bookings were protected by the business key. The email step was not, because it sits after the record is written and has its own retry. The fix is to record “confirmation sent” against the same key and check it before sending. Rerun, and the email count should drop to fifty.

Notice how the drill told you precisely where the gap was, with no argument about whose code was at fault. That clarity is why the Atlas puts this exercise on the way to its third maturity level, and why instructors love rehearsals: they turn opinions into observations.

Step 7: fix, then rerun until it is boring

Fix one thing at a time and repeat the drill after each change. A typical sequence is: add the business key, record handled keys before the side effect and not after, extend the memory period, and make the batch handler report only the failed items instead of failing the whole batch. After every change, run the identical storm and compare the counts. When two consecutive runs produce the expected numbers, you are done for now.

Then schedule it again. Any time you change the code on that path, run the drill again, and put a recurring reminder in the calendar so it happens even when nothing seems to have changed.

What to watch while you build the habit

A drill is a snapshot, and you will want everyday instruments as well. Two are worth setting up right away. The first is the age of the oldest message waiting on a queue; if that climbs, something is stuck. The second is a count of items arriving in your failure queue. The Atlas recommends giving that queue an owner and an alarm, and keeping its contents longer than the source queue does. If you are curious where the drill fits in a larger plan, the Atlas offers a month-by-month rollout in which the first duplicate-storm test lands in the middle stretch, after triggers have been catalogued and the top few side-effecting functions have been made safe.

A word on staying calm

The first time your drill produces doubles, it can feel discouraging. It should not. Instructors will tell you that mistakes made in practice are the cheap ones, and our article on staying composed under pressure in flight training makes the same argument for pilots. If budget is on your mind as you learn, our tips for trimming the cost of flight training show that careful planning pays off in both worlds, and a drill that catches a duplicate-charge bug before launch is a very cheap lesson indeed.

Keep learning

When you have completed the drill once, read how the discipline frames the next steps in the Ops Atlas entry on ServerlessOps. It covers measurement, monthly reviews and the stages beyond “Safe.” One rehearsal will not make a system perfect, but it turns a vague worry into a number, and a number is something you can improve.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *