What "non-standard" katakana are commonly used?

I know about the "standard" special katakana, like ファ fa, ディ di, ヴ vu, etc. However, I keep running into others I can't find on any list, including the ones in What special kana are used to write foreign words?

For example, on this Wikipedia page, one character has the name:

Quark (クォーク Kwōku)

I've never seen クォ kwo in any katakana list.

Another example is フュ fyu, as in the new movie X-MEN: フューチャー&パスト.

So, what are all the "non-standard" katakana in regular use?

Answer

Here is a list of extended katakana color coded based on usage. In my personal experience, I have seen the orange ones and most of the blue ones in actual use multiple times. The beige and purple ones I have never seen used before (except maybe スィ), so don't bother learning them.

Here's a list with example words I made based on the link, roughly ordered by commonality based on my experience and the amount of noteworthy dictionary hits:

Common extended katakana:

• ティ (ti)　―　パーティー (party), セキュリティー (security)


• ディ (di)　―　ディスク (disk), メロディー (melody)


• ファ (fa)　―　ファッション (fashion), アルファベット (alphabet)


• フィ (fi)　―　フィクション (fiction), プロフィール (profile, pronounced more like profeel)


• シェ (she)　―　シェルター (shelter), ポルシェ (Porshe)


• ジェ (je)　―　ジェスチャー (gesture), ジェット (jet)

Uncommon extended katakana:

• ウィ (wi) ―　ハロウィーン (Halloween), ウィルス (virus, pronounced more like weerus)


• ウェ (we) ―　ソフトウェア (software), ウェイトレス (waitress)


• フォ (fo)　―　フォーク (fork), イヤフォン (ear phone)


• フェ (fe)　―　フェア (fair), カフェ (café, coffee house)


• チェ (che)　―　チェス (chess), チェック (check, cheque)


• デュ (dyu)　―　デュエット (duet), プロデューサー (producer)


• ウォ (wo)　―　ストップウォッチ (stopwatch), キャットウォーク (catwalk)

Rare extended katakana:

• イェ (ye)　―　イェイ (yay!)


• トゥ (tu)　―　タトゥー (tattoo)


• クォ (kwo) ―　クォーツ (quartz), クォーク (quark)


• ツァ (tsa)　―　モーツァルト (Mozart), ピッツァ (pizza - more commonly ピザ)


• ツェ (tse)　―　プレッツェル (pretzel), ヘルツェゴビナ (Herzegovina)


• ツォ (tso)　―　インテルメッツォ (intermezzo)


• 
  
  

  フュ (fyu)　―　フューチャー (future)

  
  


• 
  

  ヴァ (va) ―　ヴァイオリン (violin), ヴァンパイア (vampire) (more commonly バイオリン and バンパイア)

  
  


• ヴィ (vi) ―　ヴィーナス (venus) (more commonly spelled ビーナス)


• ヴ (vu) ―　ラヴ (love) (more commonly spelled ラブ)


• ヴェ (ve) ― ベートーヴェン (Beethoven) (more commonly spelled ベートーベン)


• 
  

  ヴォ (vo) ― ヴォーカリスト (vocalist) (more commonly spelled ボーカリスト)

  
  
  


• 
  

  ドゥ (du) ― シルク・ドゥ・ソレイユ (Cirque du Soleil), ドゥーイットユアセルフ (do-it-yourself)

  
  

(even when spelled with ヴァ, ヴィ, ヴ, ヴェ or ヴォ, the words are often still pronounced with a B sound rather than a V. They just look more fancy or foreign or whatever.)

Almost exclusively in foreign names:

• クァ (kwa) ― エスクァイア (Esquire)


• クィ (kwi) ― クィントゥス (Quintus)



• クェ (kwe) ― クェンティン (Quentin)


• グァ (gwa) ― グァンタナモ (Guantanamo)


• ツィ (tsi) ― ツィンメルマン (Zimmermann)


• テュ (tyu) ― テューリンゲン (Thüringen)

Other combinations such as スィ (si) and ホゥ (hu) also exist, but unlike the ones listed above they are often substituted for other katakana. For example, simple is シンプル and hood is フード. But they can at times be used for portraying foreign sounds more accurately. Most often in names of people and places, or transcriptions of foreign words that are not used as loan words in a Japanese sentence. (Very similar to the ヴァ row, except those are much more commonly used for alternate spellings of loan words)

Lastly, as mentioned by OP in the comments, it's worth mentioning small vowel kana (ぁ, ぃ, ぅ, ぇ, ぉ) can be used for long vowels, a practice that I believe is actually more common in hiragana. It's almost a different matter entirely, but for completion's sake I'll explain it anyway.

If a small vowel kana is used after a kana with the same vowel (e.g. きぃ, ねぇ, ハァ, ルゥ) it means it's a long vowel (similar to きい, ねえ, ハー, ルー). I have a hard time finding a source that explains the nuances implied by use of a small vowel kana, but from personal experience I know four uses:

1. Just a long vowel, nothing more. i.e. ああ, あー and あぁ are completely identical, and a long vowel is just used for variation or looks. (Or there is something behind it, but I just can't figure out what. That's another explanation.) For example using ワァド as a variation of ワード. I believe this use is most common in popular fiction, where a character name in a fantasy novel might be spelled with a small vowel kana instead of a ー to make it look more unique or foreign or whatever. I believe エル・プサイ・コングルゥ (L-Ψ-congrue) mentioned by OP falls under this category.



2. At the end of a sentence or interjection a small vowel kana is sometimes used to imply a long vowel that decreases in volume and/or pitch. This is because as opposed to ああ, あぁ looks like the vowel is getting "smaller". For example, a sigh is often transcribed as はぁ. I believe this usage is most common in manga


3. For lengthening the vowel in a word without changing its meaning. A ー or a regular double vowel (e.g. それはなーに or それはなあに) could also be used, but as opposed to a small vowel these are most often used for words with a completely different meaning (e.g. すじ means line, but すうじ means number). A small vowel (それはなぁに) can be used to avoid this ambiguity. This usage is also very common in dialects. For example, in Kansai dialect they say せぇへん and けぇへん (pronounced せーへん and けーへん) instead of しない and こない, and [気]{き}ぃつけて (pronounced きーつけて) instead of [気]{き}をつけて. These are used because etymologically they used to be words with a short vowel. E.g. せぇへん comes from せ (an Old Japanese mizenkei conjugation of the verb する) + へん (Kansai variant of ない). So logically speaking the word should be せへん, but it's pronounced せえへん. Thus you could say せぇへん is a compromise between grammatically correct spelling and actual pronunciation. The same is true for 気　pronounced きい but spelled 気ぃ, and, for that matter, also それはなぁに in standard Japanese.


4. In manga and other popular fiction ですぅ and ますぅ is used for characters that pronounce です and ます without a silent vowel at the end. It can be pronounced with a short vowel, a long vowel, or somewhere in between, depending on what the voice actor deems appropriate for the character or situation. For your enjoyment (or annoyance), here is a compilation of Souseiseki from Rozen Maiden saying ですぅ about 200 times.