HexVM Engineering Practices

Reversible Network Shaping on a Cloud Mac

Reversible Network Shaping on a Cloud Mac

When remotely debugging an iOS client, the hardest condition to reproduce is usually not a complete loss of connectivity, but a connection that still works while responding very slowly: handshakes take longer, upload speeds drop, and a small number of requests are lost. These conditions can eventually trigger duplicate submissions, endless retries, or interfaces that remain stuck in a loading state. When running these tests on a HexVM Cloud Mac, the first priority is not to throttle traffic, but to isolate application test traffic from management traffic such as SSH and VNC. Otherwise, an overly broad rule can lock you out of the machine.

Define Testable Degraded-Network Scenarios First

Do not use “poor network conditions” as a test condition. It is neither reproducible nor specific enough to determine whether a fix works. Define each scenario with fixed parameters and document the expected client behavior first.

Scenario Download and upload limit Added latency Packet loss Primary acceptance criteria
Baseline Unlimited 0 ms 0 Normal request duration and success rate
High latency 8 Mbit/s 120 ms 0 Loading states, cancellation, and timeouts
Low bandwidth 1 Mbit/s 80 ms 0 Upload progress and background transitions
Unstable link 4 Mbit/s 100 ms 2% Retry limits and idempotent handling

These parameters are test inputs, not a description of the node’s network quality. Teams should adjust the values to account for API timeouts, file sizes, and the networks their users rely on. However, the parameters must remain unchanged throughout a single regression run.

The purpose of degraded-network testing is not to prove that a request can eventually succeed. It is to verify that, when a failure occurs, the client stops waiting promptly, presents an understandable status, and does not create duplicate data.

Capture an Unshaped Baseline

Before loading any shaping rules, record DNS, routing, and endpoint response data. The team should control the test endpoint and provide a probe path that cannot write production data.

export TEST_URL="https://test-endpoint.invalid/health"
scutil --dns | grep 'nameserver\[[0-9]*\]'
route -n get default
networkQuality
curl -sS -o /dev/null \
  -w 'code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}
' \
  "$TEST_URL"

Replace the example domain with a real, controlled test address. Record the median of at least three requests rather than keeping only the best result. If the baseline already shows DNS failures, certificate errors, or routing problems, resolve those underlying issues first instead of masking them with additional retries.

Also determine whether the destination is served from multiple IP addresses. If only one IP is shaped, DNS may direct a request to another address, producing inconsistent results within the same test. A more reliable approach is to provide a fixed entry point for degraded-network acceptance testing and save its resolved address before the run begins.

Shape Only the Target Endpoint’s Traffic

On macOS, dnctl creates pipes for bandwidth limits and latency, while pf directs matching traffic into those pipes. Every rule must constrain the destination IP, protocol, and port. Never use a wildcard condition that covers all outbound connections.

The script below creates a test pipe with a bandwidth limit of 8 Mbit/s, 120 ms of latency, and 2% packet loss. Set TARGET_IP to the fixed address of the controlled endpoint.

set -euo pipefail

TARGET_IP="${TARGET_IP:?set TARGET_IP first}"
PIPE_ID=310
ANCHOR="com.apple/hexvm-nettest"

cleanup() {
  sudo pfctl -a "$ANCHOR" -F all >/dev/null 2>&1 || true
  sudo dnctl delete "$PIPE_ID" >/dev/null 2>&1 || true
}

trap cleanup EXIT INT TERM

sudo dnctl pipe "$PIPE_ID" config bw 8Mbit/s delay 120 plr 0.02
printf 'dummynet out quick proto tcp from any to %s port 443 pipe %s
' \
  "$TARGET_IP" "$PIPE_ID" |
  sudo pfctl -a "$ANCHOR" -f -

sudo pfctl -E >/dev/null
sudo pfctl -a "$ANCHOR" -sr -v

curl -sS -o /dev/null \
  -w 'code=%{http_code} connect=%{time_connect} total=%{time_total}
' \
  "$TEST_URL"

First use a read-only health check to verify that the rule matches. Only then should you run business tests that write data. Do not use pfctl -d for cleanup, because the machine may already have other firewall rules. The script clears only its own anchor and deletes only its own pipe, avoiding disruption to parallel workloads.

Protect the Remote Management Session

Keep a second management terminal open before applying the rules. Confirm that they do not include management ports, the default gateway, or wildcard destinations. If the test endpoint and management connection share the same IP address, use a separate test entry point instead of relying on rule order.

Turn Client Behavior into Automated Assertions

Degraded-network acceptance testing must evaluate more than whether a request eventually returns. Test code should record the number of attempts, the final error type, the total wait time, and the request identifier. For write operations, both the server and client should use stable idempotency keys. After a timeout, the client should query the result before deciding whether to retry.

Wrap the network layer with observable instrumentation and verify the following boundaries:

  1. A connection can be canceled after exceeding its threshold instead of waiting indefinitely.
  2. Retries have a fixed upper limit and use backoff intervals without creating a request storm.
  3. After the user cancels an operation, a background task does not silently submit it again.
  4. An interrupted upload retains an explicit status and is not shown as completed.
  5. Operations can resume after connectivity returns without creating duplicate records.

Simulator tests can target an explicit device identifier instead of depending on whatever device name happens to exist on the machine.

export SIMULATOR_UDID="replace-with-booted-simulator-udid"

xcodebuild test \
  -scheme NetworkBehaviorTests \
  -destination "platform=iOS Simulator,id=${SIMULATOR_UDID}" \
  -resultBundlePath build/NetworkBehavior.xcresult

Save a separate result bundle for each network scenario and include the scenario parameters in the test log. A failure record can then show the exact latency and packet-loss conditions under which the failure occurred, rather than leaving only an irreproducible red status.

Remove the Rules and Complete the Regression Run

The script performs cleanup when it exits normally or receives an interrupt signal. However, if the remote session closes unexpectedly, verify the state manually. After reconnecting, run:

sudo pfctl -a com.apple/hexvm-nettest -sr
sudo dnctl list
curl -sS -o /dev/null \
  -w 'code=%{http_code} connect=%{time_connect} total=%{time_total}
' \
  "$TEST_URL"

The dedicated anchor should be empty, and pipe 310 should no longer appear in the list. Then repeat the baseline request and confirm that the connection time and total duration have returned to their normal pre-test ranges.

Finally, run one regression without traffic shaping. Pay particular attention to whether the degraded-network fixes changed request ordering, cache hits, or interface responsiveness under normal network conditions. Store the scenario parameters, client version, target endpoint version, result-bundle path, and cleanup outcome in the same run record. This turns degraded-network testing from a one-off manual demonstration into an auditable, reproducible engineering process that can be exited safely.

Frequently asked questions

Why should I avoid shaping all outbound traffic on a remote Mac?

A global rule can degrade SSH, VNC, dependency downloads, and the application test at the same time. Match only the test endpoint's IP address, protocol, and port.

How do I verify that the temporary network rules are gone?

Flush the dedicated pf anchor, delete its dnctl pipe, confirm that the anchor has no remaining rules, and rerun the unshaped baseline request.

Which client behaviors belong in a degraded-network test?

Check bounded retries, connection timeouts, cancellation, offline feedback, duplicate-submit protection, and clean recovery without duplicated data.

Dedicated Apple Silicon

Choose a dedicated cloud Mac for your next build

Choose a Mac mini model, rental term, and region based on your workload. Each order maps to an independent physical node; actual availability is based on the real-time response from the console.

Choose a cloud Mac plan