Effects Analysis Framework: Keep the Cases That Challenge Your Idea

A striking example can start a research idea. A public statement appears, attention moves and the earlier controversy becomes less visible. The next step is to ask whether the pattern survives a more systematic look.

In developing my proposed Effects Analysis Framework, I want to examine observable changes in attention, framing, accountability and amplification. That requires keeping evidence that complicates the explanation alongside evidence that seems to support it. The framework is a developing proposal, not an established or validated research instrument.

Define the effect before choosing the examples

If the question concerns attention, specify what will count: perhaps the proportion of relevant headlines devoted to each topic within a defined set of outlets. If it concerns framing, describe the categories used to classify the language. Decide the time window in advance where possible.

Without those decisions, “everyone stopped talking about it” can become a description of one person’s feed. A personalised feed is an observation point, but it cannot by itself establish the behaviour of the entire media environment.

Look for the pattern that did not happen

Include comparable occasions when a prominent statement was followed by little change in coverage. Also examine occasions when attention moved without such a statement. These cases help test how much explanatory weight the proposed event can carry.

Imagine a hypothetical dataset in which coverage of topic A falls after statement B. A major unrelated event also begins that afternoon. The observed decline remains worth reporting, but the competing explanation matters. Timing alone cannot tell us which cause produced the change.

Keep effects and intentions in separate columns

A measured shift in coverage and an allegation of deliberate distraction are different claims. The first might be supported by a defined sample and transparent counts. The second requires evidence about intention beyond the shift itself.

Record uncertainty where it belongs. A small sample limits generalisation. Ambiguous headlines may need a second reader. A change to the category definitions should be disclosed rather than quietly applied only to inconvenient examples.

The value of a framework grows when another person can inspect its decisions and challenge its conclusions. For EAF, the most useful question may become: what observation would make us revise this interpretation? Keeping that question open turns a compelling first impression into a research programme that can learn.