But if you tell me there are invisible yellow unicorns, what am I to do with that? That's not how we use color words. — Srap Tasmaner
It is hard to give up the commonsense-seeming notion that words refer to things. So the way you talk about this philosophically looks to presume the two realms of the mental and the physical. And then we can point to objects in both realms - real qualia and real things. And that reality then allows true correspondence relations. There is the experience of yellow in my head. There is the yellowness or wavelength energy out there in the world. Words can then safely refer - ostensively point to - some thing that is a fact of the matter. Whether out there or in my head.
So commonsense defends a dualistic paradigm of realms with real objects - both mental and physical. Then words are simply labels or tokens. All they do is add a tag for talking about the real.
But another approach - pragmatic or semiotic - would be to give words a properly causal role in reality. So now rather than merely pointing - an uninvolved position that changes no real facts - words are a habit of constraint. Speaking is part of the shaping of reality - mental or physical (to the degree that divide still exists).
So to speak of an invisible yellow unicorn is to create constraints on possibility (physical or mental). It restricts interpretation in a way that is meaningful. We are saying this unicorn, if it were visible, would be yellow (calling anything yellow itself being a general interpretive constraint on experience).
Of course unicorns are fictions. Invisible colour a contradiction. So the actual set of invisible yellow unicorns would be very empty indeed.
But that is not the point. It is how words actually function. And their role - as signs - is not to point from a mental idea to a material instance, as is "commonsense". Their role is to place pragmatic limits on existence. Or rather, restrict the interpretive relation we have with "the world" in some useful or meaningful goal achieving fashion.
So with yellow, it doesn't matter what we each have privately in our heads. What matters is that there is some reliable social habit of communication where "yellow" is a sign that acts to constrain all our mental activity in a fashion where we are most likely to respond to the world in sufficiently similar ways.
The big change here is from demanding the need for absolute truth or certainty - the pointer that points correctly - to a more relaxed view of word use where word meaning is only as constraining as useful. The fact that there is irreducible uncertainty - like do we all have the same qualia when we agree we are seeing "yellow" - becomes thankfully a non-issue. This kind of fundamentally unknowability is accepted because we can always tighten shared definitions if wanted. But more importantly, not having to sweat such detail is a huge semantic saving of effort - and the basic source of language creativity. You want slippery words otherwise you would be as uninventive as a machine or computer.
So it is a paradigm shift. Words work by restricting states of interpretation or experience. They don't have to point from an idea to a world, or connect every physical object with its mental equivalent object in "true referential" fashion.
But words do have to be effective as encoding habits of thought. They have to produce the kinds of relational states of which they attempt to speak. An "invisible yellow unicorn" is an example of the kind of word combination that is perfectly possible, but which could have no meaning as either something we could physically encounter or properly imagine.