What does Hamlet mean when he says "The play's the thing/Wherein I'll catch the conscience of the king."?

This rhymed couplet sums up the plan that Hamlet decides to pursue at the end of Act II.

Hamlet isn't entirely certain that Claudius killed Hamlet's father, so he is going to test Claudius. He is going to stage a play for Claudius to watch, a thinly-disguised dramatization of Claudius's supposed murder of Hamlet's father. If, after watching the play, Claudius betrays emotions of guilt or otherwise behaves suspiciously, Hamlet will have confirmed that his uncle did, in fact, kill his father.

If you read the approximately 18 lines leading up to these words, you can observe Hamlet hatch this plan. The others have left; Hamlet is thinking aloud. And he recalls something:

"Fie upon't! foh! About, my brain! I have heard
That guilty creatures sitting at a play
Have by the very cunning of the scene
Been struck so to the soul that presently
They have proclaim'd their malefactions;
For murder, though it have no tongue, will speak
With most miraculous organ."

In other words, he's heard that people guilty of crimes have confessed their guilt after watching a play that portrays situations highly reminiscent of their own, criminal acts. Seeing their crimes acted out by others reminds them of their guilt, and so they give themselves away.

Having remembered this, Hamlet determines that he will write his own little drama -- one recreating the peculiar circumstances of the crimes he believes Claudius committed -- and have the actor's troupe perform it for the royal household. Then, to determine if Claudius is guilty, Hamlet says:

"… I'll observe his looks;
I'll tent him to the quick: if he but blench,
I know my course."

If Claudius reacts with so much as a flinch, Hamlet says, he will have the evidence he needs to confirm Claudius's guilt. This is crucial because Hamlet isn't sure he can trust what he was told by the spirit that had appeared to be his father's ghost. He's concerned that "the devil" might have taken the shape of his father and told him lies.