What a great question! I ran into this the other day reading Dark Heresy 1e, which has specific rules for becoming invisible, but no rules on what it means to be invisible.
I find D&D 5e’s solution to be one of the simplest, if not the most elegant: you are assumed to be able to determine location by sound, such that you can attack at disadvantage. However, character cannot be seen so doesn’t trigger a bunch of “one character you can see” conditions for spells and the like. It’s easy, it’s conceptual. It’s baller. You can break it down into skill checks and passive perception with Hiding mechanics. There’s a real simplicity for it. Of course, that’s also a massive problem because there’s no granularity (for example if you already have disadvantage on your roll, the invisibility means nothing). It also doesn’t “feel” like invisibility in a game like that. It also responds INCREDIBLY poorly to the tactical minis part of the game. All of that is an honest mess. But it’s entry level, so its fine.
Negative modifiers, stepping back outcomes, extra rolls as you pointed out. It’s truly interesting. This kind of thing that tactical wargames work around with fog-of-war and especially smoke (WW2 settings). There are entire armies in Flames of War built around smoke. The benefit of smoke there is because it’s a thin line of line-of-sight-blocking obstacles, the point is to force movement. The movement then reduces tank rate-of-fire and provides manoeuvring opportunity.
In the kind of FKR, Kriegspiel methodology, darkness/blindness/smoke are all both strategic and tactical limiters. Strategically, the GM passes less information on to the players (in a dungeon crawl, imagine this as taking damage or making dex rolls without really knowing the context behind it). Tactically, it reduces the impact of the crews involved: A slower advancing speed, no long-range shots on a charging enemy. And I think both avenues are valuable.
The Interesting, Challenging, Fair triangle is a cool question here because usually interest comes from a question left unanswered: roll 1d6, take that much damage. “what from?” Great question, your arm feels hot and wet, but why, you don’t know.
Challenge comes from providing an adversary with axes of attack and defence, which means you need a way to nullify the darkness’ effect. Perhaps that’s with a spell of light, or perhaps it’s by throwing a bomb forward and just hoping you hit whatever is threatening you over there. This is where the “challenge” of locating by sound can be really interesting: Rolling perception rolls for the group as you attempt to narrow down a location for return fire. I imagine this in a military/spec ops setting as a skill challenge to overcome the darkness (cf @TheBeardedBelgian and their comments about FATE, in which negatively charge aspects/characters exist to be both Overcome and overcome). HP-for-Darkness is a good way to think about it too, engaging with the extant mechanics to overcome it.
Fair depends on the scope of your game and the economies in play. In FATE, are there ways for players to Overcome, and spend and receive fate points? In an OSR game, are there tactics in which they can avoid or reduce the impact of darkness? In D&D, have you considered the darkness within the challenge rating of the encounter (some monsters that can produce darkness or have their own invisibility have this inherent, some do not).
Encounter design is a really interesting pattern of planning, and I don’t think it’s as simple as “is darkness interesting”. In a kobold v adventurers slug-fest, or a runners v prisec shootout, darkness with extra miss rolls just extends the whole combat symmetrically in a very boring way. But if the challenge is “do you assassinate the high value target”, a Prisec team can cut the lights to create an opportunity to evac. If the challenge is “do you escape uncaptured”, adventurers can cast Darkness to give themselves to provide a clean out with the conflict that they also can’t see.
In wargames, smoke encounter design is about providing counterplay tactically (as I said, in FoW, smoke is a thin line that can be avoided or waited out), and building the cost into the unit strategically (as both a point cost for balancing, and an opportunity cost for use).
I think as a standard, consider how it makes DIFFERENT things difficult, as opposed to how it makes the regular challenges MORE DIFFICULT. ie it’s not about “can you beat the armour class of this monster”, it’s about “can you move through the difficult and dangerous room, in order to make that one-hit-kill at the end”.
Also “A Blind Legend” on steam. It’s an adventuring story where directional hearing is used to direct you.