Identify the nature of the change and the metric to consider to decide which version of the site to choose
Next, decide the number of samples/visits necessary to hit the necessary statistical significance (e.g. 95%). This can be done by using a chi-squared test (if we are using a binomial random variable of clicking vs. not clicking) or a ztest (if we are using a normally distributed random variable). You can then evaluate the p-value to identify whether the metric of the B test is statistically significantly different than the metric of the baseline A test. If it is and the metric is better than the baseline, then the alternative site is the better way to go.
Some other issues you should consider in this answer:
1) Identify potential biases due to interactions across pages. Talk to the product manager and see if there are ways that a random sampling may not work to test the nature of the change you’re proposing for a web page.
2) Perform a A/A test which implies testing two random samples of visitors, and check if the distribution and metric of choice does not have a statistically significant difference. This will ensure the fairness of the A/B test. An A/A test ensures that your audience doesn’t have a particular skew or bias and a randomized selection for an A/B test will be statistically relevant