Preferences

Given states (prizes) $A$ and $B$, an Agent can express preferences of the form

For deterministic environments we know all possible states so we can always state preferences in the following way.

$A$ is preffered over $B$ $A\succ B$ $A$ and $B$ are the same $A\sim B$ $B$ is not preferred over $A$ but it would not matter when $A$ is preffered $A⪰B$

In a Stochastic Environment however we do not have full information about the states. So we use the states Priors in a lottery. $\left[p_1,A_1;\dots ;p_n,A_n\right].$

A set of preferences is called rational iff the following constraints hold:

Orderability

$A\succ B\vee B\succ A\vee A\sim B$

Transitivity

$A\succ B\wedge B\succ C\Rightarrow A\succ C$ also see Transitivity

Continuity

$\mathbf{A}\succ B\succ C\Rightarrow (\exists \mathbf{p}\cdot [\mathbf{p},\mathbf{A};1-\mathbf{p},C]\sim B)$

Substitutability

$A\sim B\Rightarrow [p,A;1-p,C]\sim [p,B;1-p,C]$

Monotonicity

$A\succ B\Rightarrow (p>q)\iff [p,A;1-p,B]\succ [q,A;1-q,B]$

Decomposability

Lotteries in a lottery can be decomposed into a big lottery via their probabilities.

$[p,A;1-p,[q,B;1-q,C]]\sim [p,A;((1-p)q),B;((1-p)(1-q)),C]$

For all constraints we could construct an example where an agent would act irrational if the constraint would not hold.

Also see Preferences on Reward Sequences for MDP.