An unauthorized, bogus court.
Kangaroo courts are sham legal proceedings which are set-up in order to give the impression of a fair legal process. In fact, they offer no impartial justice as the verdict, invariably to the detriment of the accused, is decided in advance. Such courts are associated with groups who have found a need to dispense a rough and ready form of justice but are, temporarily at least, outside the bounds of formal judicial processes; for example, inmates in jail, soldiers at war, settlers of lands where no jurisdiction has yet been established.
The origin of 'kangaroo court' is unknown, although, given that kangaroos are native nowhere else, we might expect the term to have originated in Australia. As always, a lack of a definite origin encourages speculative claims, which may be an appropriate word in this context as one frequently repeated supposed derivation relates to 'claim jumping' in the California Gold Rush - hence the allusion to kangaroos. That's quite a plausible notion. Kangaroos and their claim to fame, so to speak, i.e. jumping, were known in the USA by the early 1800s, so there's no reason to limit the derivation to Australia. Also, the earliest known citation of the term is American and appears in a collection of magazine articles by Philip Paxton (the pen name of Samuel Adams Hammett), which were published in 1853 under the title of A stray Yankee in Texas:
"By a unanimous vote, Judge G-- was elected to the bench and the 'Mestang' or 'Kangaroo Court' regularly organized."
The natural inclination to want to base the phrase in Australia has led to suggestions that the vacant stares of kangaroos when meeting humans for the first time were mimicked by jury members in court. There's no documentary evidence to support this, or any other Australian derivation, and it seems highly speculative.
The claim jumping derivation though has the feel of a 'trying to hard' explanation that is the stamp of folk etymology. The supposed wordplay of linking kangaroos and jumping is appealing but isn't really necessary to explain this phrase. Kangaroo courts courts were also called 'mustang courts' in the USA (see above). Allusions to the unsophisticated natures of wild animals are frequent in the metaphorical coinage of phrases that apply to things that are considered inferior or ersatz. We have dog Latin, dog's breakfast, horse-faced and many others. It seems probable that the reference to mustangs (half-wild horses) and kangaroos came about by that same route.
See other phrases that were coined in the USA.