In this paper, they systematically study different ways of learning concept bottleneck models. Let LcjL_{c_j} be a loss function that measures the discrepancy between the predicted and true j-th concept, and let LYL_Y measure the discrepancy between predicted and true targets. They consider the following ways to learn a concept bottleneck model (f^,g^)(\hat{f}, \hat{g}):

  1. The independent bottleneck learns f^\hat{f} and g^\hat{g} independently: f^=arg min⁡f∑iLY(f(ci);yi)\hat{f} = \argmin_{f}\sum_{i}L_{Y}(f(c^{i}); y^i), and g^=arg min⁡g∑i,jLCj(gj(xi);cji)\hat{g} = \argmin_{g}\sum_{i,j}L_{C_j}(g_j(x^i);c_j^i). While f^\hat{f} is trained using the true cc, at test time it still takes g^(x)\hat{g}(x) as input.

  2. The sequential bottleneck first learns g^\hat{g} in the same way as above. But it uses the concept predictions g^(x)\hat{g}(x) to learn f^=arg min⁡f∑iLY(f(g^(xi));yi)\hat{f}=\argmin_{f}\sum_{i}L_Y(f(\hat{g}(x^i)); y^i).

  3. The joint bottleneck minimizes the weighted sum f^,g^=arg min⁡f,g∑i[LY(f(g(xi));yi)+λ∑jLCj(g(xi);ci)]\hat{f}, \hat{g} = \argmin_{f,g}\sum_{i}\left[L_Y(f(g(x^i)); y^i) + \lambda \sum_{j} L_{C_j}(g(x^i); c^i)\right]

  4. The standard model ignores concepts and directly minimizes f^,g^=arg min⁡f,g∑iLY(f(g(xi));yi)\hat{f}, \hat{g} = \argmin_{f,g}\sum_{i}L_{Y}(f(g(x^i)); y^i).

The hyperparameter λ\lambda in the joint bottleneck controls the tradeoff between concept vs. task loss. They also consider that the standard model is equivalent to taking λ→0\lambda \rightarrow 0, while the sequential bottleneck can be viewed as taking λ→∞\lambda \rightarrow \infty.

References

Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020, November). Concept bottleneck models. In International conference on machine learning (pp. 5338-5348). PMLR.