A/B experiments

An experiment runs on one parameter. Units (signed-in users, otherwise devices) that match its condition and fall in its exposure share are assigned a variant by weight. Assignments are stored, so changing weights only moves people who weren't assigned yet. The SDK logs one $exposure event the first time the app reads the value, and results compare each variant with the control.

sov experiments create checkout_test --param new_checkout \
  --variant control=false:50 --variant on=true:50 --goal purchase
sov experiments start checkout_test
sov experiments results checkout_test
sov experiments rollout checkout_test --yes        # the winner becomes the default

#How results are computed

• Conversion goals use a two-proportion z-test; mean goals use Welch's t-test. Both show a 95% interval for the change against the control.
• A sample-ratio check (chi-square) flags a split that doesn't match the weights; fix logging or assignment before trusting the numbers.
• No winner is named until every variant has the minimum sample (100 by default) and p < 0.05.
• Roll out winner makes the winning value the parameter's default for everyone, ends the experiment and publishes.

#Plans and limits