CallEvent
CallEvent is a readonly struct passed to the Resilience.OnEvent listener. Raising an event is allocation-free, though the Result member boxes value-type results. Result is populated only when a listener is attached.
| Member | Description |
|---|---|
Kind | The type of event that occurred. |
PolicyName | The name of the policy (Resilience.Name). This allows a single listener to monitor multiple policies. |
AttemptNumber | The 1-based index of the attempt. For Attempt events, it is the attempt that just finished. For Retrying and pre-attempt events, it is the attempt about to run. |
Verdict | The classification of the most recent attempt. Defaults to Ok before any attempt has run. |
Duration | For Attempt events, this is the duration of that attempt. For all other event kinds, this is the total time the call has been running. |
Delay | The duration of the pause about to be served. This is the backoff delay for Retrying events or the rejection pause for the two rejection events. This is null for all other event kinds. |
Exception | The exception thrown by the most recent attempt, or null if none was thrown. |
Result | The value returned by the most recent attempt, as an object. This is null if the attempt threw an exception or returned nothing. |
Reason | The StopReason indicating why the call stopped. This is populated only for terminal event kinds. |
IsRejection | true for RejectedByBreaker and RejectedByBudget, for a listener that treats the two refusals alike. false for RejectedByQuota, which refuses one attempt rather than the call. |
IsTerminal | true for the kinds that end a call. Exactly one of these is raised per call. |
ToString() | Returns a formatted summary of the event, omitting absent segments. |
Create(kind, ...) | Static. Builds a CallEvent for testing a listener without the executor. Every parameter but kind is defaulted. |
CallEventKind
The CallEventKind enum defines the event types raised during a call.
| Kind | Terminal | Delay Populated | Reason Populated |
|---|---|---|---|
Attempt | No | No | No |
Retrying | No | Yes (Backoff) | No |
Succeeded | Yes | No | Yes (Succeeded) |
NotRetried | Yes | No | Yes (Permanent) |
Exhausted | Yes | No | Yes (AttemptsExhausted) |
RejectedByBreaker | Yes | Yes (Rejection pause) | Yes (DependencyUnavailable) |
RejectedByBudget | Yes | Yes (Rejection pause) | Yes (BudgetExhausted) |
DeadlineExceeded | Yes | No | Yes (DeadlineExceeded) |
OrphanedWork | No | No | No |
BreakerOpened | No | No | No |
BreakerClosed | No | No | No |
BreakerHalfOpened | No | No | No |
NestedRetry | No | No | No |
HedgeStarted | No | Yes (the latency threshold) | No |
HedgeWon | No | No | No |
HedgeDiscarded | No | No | No |
HedgeSuppressed | No | Yes (the latency threshold) | No |
AttemptCeilingAdapted | No | Yes (the measured ceiling) | No |
BackoffBaseAdapted | No | Yes (the measured base) | No |
Stalled | No | Yes (the stall bound) | No |
SaturationDetected | No | Yes (the queue delay) | No |
RejectedByQuota | No | Yes (the time until the window resets) | No |
StreamResumed | No | No | No |
Draining | Yes | No | Yes (Draining) |
Event invariants and behavior
- Terminal events: Every call ends with exactly one terminal event, which is what makes logical operations countable. The
IsTerminalproperty identifies them.Stalledis the one event that can arrive after the terminal event of the call it belongs to, and it is not itself terminal, so the count still holds - see below. - Rejections: A refusal names the guard that made it:
RejectedByBreakerindicates that the dependency is unavailable;RejectedByBudgetindicates that the client is retrying too hard.IsRejectioncovers both.RejectedByQuotarefuses one attempt rather than the call, so it is neither terminal nor anIsRejection- see below. - Attempt events: Exactly one
Attemptevent fires per attempt. - Retrying events:
Retryingfires before the backoff delay is served, so listeners can report the expected idle time. - Orphaned work:
OrphanedWorkfires when an attempt exceeds its ceiling by more than one second, raised retrospectively the moment the work finally returns. - Nested retries:
NestedRetryevents are raised only by the HTTP handler. - Hedging:
HedgeStartedcarries the live latency quantile that triggered it onDelay.HedgeDiscardedfires when a leg is cancelled because a sibling answered first; itsDurationis how long that leg ran. A discarded leg raises noAttemptevent, because nothing classified it.HedgeSuppressedfires when a call got slow enough to hedge and the hedge was held back - bySuppressAtor byWinRate- and carries the same threshold onDelaythatHedgeStarteddoes, so the two count against each other. A hedge the retry budget refused, one held back because the call isSheddable, and one that was never armed at all, raise nothing. See Hedging. - Stalls:
Stalledfires when a transfer is cut off for lack of progress, and carries the bound that fired onDelayand theAttemptStalledExceptiononException. For a response body the caller is reading itself, it arrives after the call has already raisedSucceeded- the retry loop was over before the stall existed - which makes it the only event that follows a terminal one. For a stream between two elements, and for a body read underBufferResponses, the stall is inside the attempt and is followed byRetryingor by a terminal event of its own. See progress bounds. - Measured backoff bases:
BackoffBaseAdaptedcarries the new base onDelay, after theSpreadclamp - which is what the curve actually uses. It is raised on the retry decision, and only when the number differs from the last one raised for that policy instance. A policy whose previous attempt was throttled rather than transient raises nothing, because a throttled retry does not use the measured base. See Retry. - Measured attempt ceilings:
AttemptCeilingAdaptedcarries the new ceiling onDelay. It is raised only when the measured term is what bounds the attempt, and only when the number differs from the last one raised for that policy instance - so the rate follows how much the estimate moves rather than how much traffic there is. A policy whose ceiling has been clamped back toAttemptTimeoutraises nothing. See Deadlines. - Local saturation:
SaturationDetectedcarries the thread-pool queue delay that was measured onDelay. It is raised at the onset of an episode - once when the process crossesMultipletimes its own normal queue delay, and nothing more until the queue has drained and filled again - so a count of these is a count of local incidents. The episode is tracked per policy instance, because each policy independently stops feeding its own estimates: a listener attached to five policies sees five events per episode. Nothing is refused and no bound moves, so it is neither a failure nor terminal. See Local saturation. - Published quota:
RejectedByQuotafires when the allowance the dependency publishes is spent and the HTTP handler refused an attempt without sending it, and carries the time until the published window resets onDelay- which is also the pushback the retry honors. The call still ends withSucceeded,ExhaustedorDeadlineExceeded.Durationis zero, because the handler does not hold the call's start. Raised only by the HTTP handler. See the published quota. - Breaker transitions: Breaker state transitions are raised on the call that triggered the transition, outside the breaker's internal lock.
- Checkpointed resume:
StreamResumedfires when a stream that failed part-way through was restarted from the caller's last checkpoint, and when an HTTP body that stalled while the caller was reading it resumed with aRangerequest.AttemptNumberis the restart, counting from one - not the underlying attempt count, which each restart's own retry sequence resets. See Checkpointed resume and resuming a stalled download. - Draining:
Drainingfires when a call stops because the process is shutting down, and carriesStopReason.Draining. It replaces the retry the call would otherwise have made, so it is terminal, and the failure reported is the dependency's own rather than a refusal of the library's.Durationis how long the call ran, andDelayis null, because nothing is going to wait. A hedge is not armed while draining either, and raises nothing. See Drain-aware shutdown.
Listener contract
The OnEvent listener runs synchronously on the executor's thread.
- Blocking: A listener that blocks blocks the whole call.
- Exceptions: Any exception a listener throws is swallowed, so telemetry cannot crash the application.
- Multiple listeners:
policy.WithListener(listener)adds one to whatever is already attached. AssigningOnEventin awithexpression replaces it instead, which drops the telemetry and logging a registration attached. - Order: Listeners run in the order they were added.
Log event IDs
These are the ILogger records a registered policy writes. An event ID is a contract the moment an alert is built on it, so the numbers below are stable and gated by a test. See Logging in DI for how to filter them.
Every record is written every time unless you opt into sampling, which thins IDs 1000-1005 and 1022-1024 while the policy is healthy and leaves the rest alone.
| ID | Name | Default | Verbose | Message |
|---|---|---|---|---|
| 1000 | AttemptSucceeded | Trace | Information | {Policy} attempt {Attempt} succeeded in {ElapsedMs} ms |
| 1001 | AttemptFailed | Debug | Information | {Policy} attempt {Attempt} failed in {ElapsedMs} ms: {Verdict} {ErrorType} |
| 1002 | AttemptLimited | Debug | Information | {Policy} attempt {Attempt} was refused by a local limiter before it left the process |
| 1003 | Retrying | Debug | Information | {Policy} waiting {DelayMs} ms before attempt {Attempt} after a {Verdict} outcome |
| 1004 | CallSucceeded | Trace | Information | {Policy} succeeded in {ElapsedMs} ms |
| 1005 | CallSucceededAfterRetries | Debug | Information | {Policy} succeeded on attempt {Attempt} after {ElapsedMs} ms |
| 1006 | NotRetried | Debug | Information | {Policy} stopped after attempt {Attempt}: the outcome was classified Permanent |
| 1007 | NotRetriedFirstSighting | Warning | Warning | {Policy} did not retry {ErrorType} on attempt {Attempt} because the classifier called it Permanent. |
| 1008 | Exhausted | Debug | Information | {Policy} used all {Attempt} attempts in {ElapsedMs} ms and failed with {ErrorType} |
| 1009 | DeadlineExceeded | Debug | Information | {Policy} ran out of deadline after {ElapsedMs} ms on attempt {Attempt} |
| 1010 | RejectedDependencyUnavailable | Warning | Warning | {Policy} refused a call because its circuit breaker is open. Rejections logged quietly since the previous warning: {Suppressed}. |
| 1011 | RejectedBudgetExhausted | Warning | Warning | {Policy} refused a retry because the retry budget is exhausted. |
| 1012 | RejectedRepeat | Debug | Information | {Policy} refused a call: {Reason} |
| 1013 | BreakerOpened | Warning | Warning | {Policy} opened its circuit breaker on attempt {Attempt}. |
| 1014 | BreakerHalfOpened | Information | Information | {Policy} is probing its dependency: the break duration elapsed and this call is the probe |
| 1015 | BreakerClosed | Information | Information | {Policy} closed its circuit breaker and is taking traffic again |
| 1016 | OrphanedWork | Warning | Warning | {Policy} attempt {Attempt} kept running after its timeout, so that work is still going unobserved. |
| 1017 | OrphanedWorkRepeat | Debug | Information | {Policy} attempt {Attempt} kept running after its timeout |
| 1018 | NestedRetry | Warning | Warning | {Policy} is retrying a request that is already inside a retrying client. |
| 1019 | NestedRetryRepeat | Trace | Information | {Policy} is retrying inside another retrying client |
| 1020 | PolicyResolved | Debug | Information | {Policy} resolved: {Effective} |
| 1021 | PolicyClassifier | Trace | Debug | {Policy} classifier: {Rules} |
| 1022 | HedgeStarted | Trace | Information | {Policy} started hedge attempt {Attempt}: the call has been running longer than {ThresholdMs} ms |
| 1023 | HedgeWon | Trace | Information | {Policy} answered from hedge attempt {Attempt} after {ElapsedMs} ms |
| 1024 | HedgeDiscarded | Trace | Information | {Policy} discarded attempt {Attempt} after {ElapsedMs} ms because a sibling answered first |
| 1025 | AttemptCeilingAdapted | Debug | Information | {Policy} measured a new per-attempt ceiling of {CeilingMs} ms from recent latency |
| 1026 | BackoffBaseAdapted | Debug | Information | {Policy} measured a new backoff base of {BaseMs} ms from recent latency |
| 1027 | HedgeSuppressed | Debug | Information | {Policy} held back hedge attempt {Attempt} after {ThresholdMs} ms |
| 1028 | Stalled | Warning | Warning | {Policy} cut off a transfer after {TransferredCount} byte(s) or element(s): nothing arrived for {StallMs} ms |
| 1029 | SaturationDetected | Debug | Information | {Policy} stopped measuring: this process's thread pool is queueing for {QueueDelayMs} ms |
| 1030 | RejectedByQuota | Warning | Warning | {Policy} refused an attempt because the allowance the dependency publishes is spent, and it resets in {ResetMs} ms. |
| 1031 | StreamResumed | Debug | Information | {Policy} resumed a stream on attempt {Attempt} from the caller's last checkpoint |
| 1032 | Draining | Debug | Information | {Policy} stopped after attempt {Attempt} in {ElapsedMs} ms without retrying: this process is draining, and failed with {ErrorType} |
Event 1032 is written once per drained call, so its rate during a rollout is a count of calls the shutdown cut short.
Field names are shared with the metric tag vocabulary wherever both exist (Policy, Verdict, Reason), so a structured record and a metric describe the same call with the same words.
Events 1010, 1011, 1016, 1018 and 1030 are rate-limited per policy - see flood control. Events 1007, 1012, 1017 and 1019 are the quiet forms the suppressed occurrences take.
