Character Encoding Issues: Why “ö” sometimes displays as “ö”

A few weeks ago I got an email which read (in german):

“Unser Versandpartner meldet sich telefonisch oder per Mail zur Ankündigung der Lieferung. Aufgrund des großen Bestellaufkommens können wir dies aktuell jedoch nicht garantieren.”

This is a very common encoding phenomenon, which happens all the time. But why does this happen? To understand the problem, we first need to talk about what encoding means.

How computers represent text: ASCII

Computers generally only work with numbers, so there needs to be some way to interpret a number as a character. Once you know how a character is represented you can form any text as a sequence of numbers, also known as “strings”. This representation of numbers as characters is known as the character encoding.

One of the simplest encodings is known as ASCII and assigns numbers to unique characters. Although ASCII characters are stored in a byte, which can represent 256 possible values, ASCII only consists of 128, as originally one of the bits was used for error checking. This was later replaced by an extended ASCII version (ISO 8859-1), which makes use of all 256 possible characters. As there are only a few of them, the ASCII characters can easily be shown in a table:

No.	Char	No.	Char	No.	Char	No.	Char	No.	Char	No.	Char	No.	Char	No.	Char
0	Ctrl-@	16	Ctrl-P	32		48	0	64	@	80	P	96	`	112	p
1	Ctrl-A	17	Ctrl-Q	33	!	49	1	65	A	81	Q	97	a	113	q
2	Ctrl-B	18	Ctrl-R	34	"	50	2	66	B	82	R	98	b	114	r
3	Ctrl-C	19	Ctrl-S	35	#	51	3	67	C	83	S	99	c	115	s
4	Ctrl-D	20	Ctrl-T	36	$	52	4	68	D	84	T	100	d	116	t
5	Ctrl-E	21	Ctrl-U	37	%	53	5	69	E	85	U	101	e	117	u
6	Ctrl-F	22	Ctrl-V	38	&	54	6	70	F	86	V	102	f	118	v
7	Ctrl-G	23	Ctrl-W	39	'	55	7	71	G	87	W	103	g	119	w
8	Ctrl-H	24	Ctrl-X	40	(	56	8	72	H	88	X	104	h	120	x
9	Ctrl-I	25	Ctrl-Y	41	)	57	9	73	I	89	Y	105	i	121	y
10	Ctrl-J	26	Ctrl-Z	42	*	58	:	74	J	90	Z	106	j	122	z
11	Ctrl-K	27	Ctrl-[	43	+	59	;	75	K	91	[	107	k	123	{
12	Ctrl-L	28	Ctrl-\	44	,	60	<	76	L	92	\	108	l	124	`
13	Ctrl-M	29	Ctrl-]	45	-	61	=	77	M	93	]	109	m	125	}
14	Ctrl-N	30	Ctrl-^	46	.	62	>	78	N	94	^	110	n	126	~
15	Ctrl-O	31	Ctrl-_	47	/	63	?	79	O	95	_	111	o	127	DEL

This worked well with basic english text, however, there is one obvious flaw. You can only represent strings including these 128 characters. If you need a special character, like “ö”, it cannot be represented. Although there were local variants of ASCII, there was no uniform encoding everyone agreed on. This is where our next encoding comes into place, which is trying to fix this problem.

The UTF-8 encoding

UTF-8 is build as a superset of ASCII, to make it backwards compatible with ASCII. In order to store more characters, it uses a new approach called “variable-width encoding”. If a sequence of bytes starts with a special “leading byte”, it will be interpreted as 2-, 3- or 4-byte long characters.

This allows for more than enough characters to be stored, while still being storage efficient when using ASCII text. If you are working with a language that uses a lot of special characters, storage performance might increase as all your characters include the leading byte.

To address this UTF-16 was invented - to keep storage relatively efficient. It uses 16 bits or 2 bytes to store a single character while also using variable-width encoding for any special characters which cannot be represented by the 16 bits.

Choosing the wrong character encoding

As long as everyone agrees on one encoding, you can easily share your message by sending the bytes to your recipient. Problems start to arise when you send your message in a different encoding than your recipient expects.

For example: Someone sends me a message in UTF-8, representing “ö” as a 2 byte character. However, I interpret the bytes of the message as some form of extended ASCII, resulting in both bytes being shown as characters, namely “Ô and “¶”.

The same problem can also happen the other way around:

For example: Someone sends me a message in extended ASCII, representing “ö” by their 1 byte character. However, I now interpret the byte as UTF-8 and the original character looks like this: “�”.

How it can be prevented

Today, most services agree on UTF-8 encoding. Furthermore, to avoid misunderstandings of automated systems, almost all HTTP messages come with a special header attached to it which designates the content type and character set:

Content-Type: text/html; charset=UTF-8

To double down, HTML itself can also contain an additional meta tag to specify the character set.

<meta charset="UTF-8">

It’s still ironic that in order to find out the character set, one first has to read a special header string, which itself is encoded in a character set.

Conclusion

In german, as there are only a few special characters, no one really bothers reporting character encoding issues as it is still readable.

Now, coming back to the original message I got: I’m very confident my email client uses UTF-8. Instead what probably happened is that some backend process of the automatic email system assumed a wrong character encoding at some point while processing the message.

If you’ve been on the internet, you’ve seen these character encoding errors. They are very common. If you happen to come across one of these, try to figure out who messed up the encoding and how!