Selberg sieve

Remarks

The inequality holds termwise: if $\gcd(a, P(z)) = 1$, the inner sum is $\lambda_1 = 1$; otherwise the square is $\ge 0$. Example: take $\mathcal{A} = \set{1, \ldots, 10}$, $P(z) = 6$, $D = 3$, $\lambda_1 = 1$, $\lambda_2 = \lambda_3 = -1/2$ and $\lambda_6 = 0$. The inner sum is $1$ for the three $a$ coprime to $6$, and $1/2$ for the six $a$ divisible by exactly one of $2$ and $3$. It is $0$ for $a = 6$. So the bound is $3 + 6 \cdot \tfrac14 = 4.5$, and indeed $S(\mathcal{A}, \mathcal{P}, z) = 3 \le 4.5$. Non-example: weights with $\lambda_1 = 0$ fail clause 1, and then $a = 1$ contributes $0$ instead of $1$, so the inequality can fail. With $0 < g(p) < 1$ for $p \mid P(z)$, the optimal weights give the main term $X / G$ with $$G = \sum_{\substack{d \mid P(z) \\ d \le D}} \prod_{p \mid d} \frac{g(p)}{1 - g(p)},$$ and they satisfy $\abs{\lambda_d} \le 1$.