What it means
It is the probability that the difference you are seeing between two variants is not down to chance. Most teams use a 95% confidence threshold — a 1-in-20 chance the result is a fluke.
Why it matters
Acting on insignificant results is how teams ship "winners" that quietly lose money. Significance depends on three things: the size of the effect, the amount of traffic, and how long the test runs. Small effects need lots of traffic to prove. Use a sample-size calculator before you launch, and do not stop early just because the line looks good.