How to exploit a stream cipher and how to prevent it from happening

As part of a resent cyber security competition that I did recently NorthSec 2026 I was able to learn about and take advantage of a type of encryption algorithms called stream ciphers (like RC4 and AES-CTR) which is incredibly fast and simple but have some flaws that we can take advantage of. In this post i’ll explain the challenge, how I exploited it, other ways to exploit it, and the many ways you could prevent attackers from taking advantage of this.

What was the challenge?

If you aren’t interested about the challenge setup and want to go directly to the solution please skip ahead.

The challenge prompt mentioned that we had to find a document called letter.txt that could be access through a website link that was given. When you went on that website you would see a single button that randomly redirected you to one of these urls.

http://docgen.ctf/loadLetter?d=ITI8A2KqR8UciV9IUPeCQRow4hFK2FMjqrNVTe4AH4RSd3A4gFIHlgquMEWHtwYFvMeC%2FzxMq1vw2N7APaovXIWWyV54LHXURRsAnzqCQljv
http://docgen.ctf/loadLetter?d=ITI8A2KqR8UciV9IUPeCQRow4hFK2FMjqrNVTe4AH4RSd3A4UgeWCq4wRYe3BgW8x4L/PEyrW/DY3sA9qi9chZbJXngsddRFGwCfOoJCWO8=
... and others

The first thing I noticed is that the query params look a lot like base64 encoded strings.

However, when you try to decode these as strings you get a bunch of random bytes

from urllib.parse import unquote
import base64

encoded = "http://docgen.ctf/loadLetter?d=ITI8A2KqR8UciV9IUPeCQRow4hFK2FMjqrNVTe4AH4RSd3A4gFIHlgquMEWHtwYFvMeC%2FzxMq1vw2N7APaovXIWWyV54LHXURRsAnzqCQljv"
url_prefix = "http://docgen.ctf/loadLetter?d="
encoded = encoded.removeprefix(url_prefix)
encoded_bytes = base64.b64decode(unquote(encoded))

print(encoded_bytes)

print(encoded_bytes.decode())
$ python3 attack.py
b'!2<\x03b\xaaG\xc5\x1c\x89_HP\xf7\x82A\x1a0\xe2\x11J\xd8S#\xaa\xb3UM\xee\x00\x1f\x84Rwp8\x80R\x07\x96\n\xae0E\x87\xb7\x06\x05\xbc\xc7\x82\xff<L\xab[\xf0\xd8\xde\xc0=\xaa/\\\x85\x96\xc9^x,u\xd4E\x1b\x00\x9f:\x82BX\xef'
Traceback (most recent call last):
  File "/home/mauricio/clones/temp/mailbox/attack.py", line 11, in <module>
    print(encoded_bytes.decode())
          ~~~~~~~~~~~~~~~~~~~~^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaa in position 5: invalid start byte

Well… it looks like its not that simple. They aren’t strings so what are they?

Still, good clues were given when you modified and replaced parts of the url.

When you removed a character
Exception: IllegalArgumentException
Reason: Last unit does not have enough valid bits
When you modified a random character
Exception: JSONException
Reason: Expected a ',' or '}'
When you modified part of the url that wasn’t the common starting characters
Exception: FileNotFoundException
Reason: the file ./docs/fric_ddfcd1c55ea4.md could not be found

If you transformed the URL into bytes, then modified one byte (from f backwards) at a time and transformed it back into base64, you can find the offset of the base64 encoded bytes which modified the starting . (which was offset = 30).

This gave you enough clues to see that:

  1. you had indeed a base64 encoding doing somewhere
  2. It was decoding to a JSON string
  3. it was malleable: which means that one character in the base64 bytes modified one character of the plain text string (you can see on the website that the post was supposed to be done by eric and not fric)

When you search the internet “common ways to encrypt plaint text on website” you’ll see that one encryption type is a stream cipher which has multiple ways of exploiting it.

How to exploit a stream cipher?

Other articles like this one, or this one mention how to get the key of the key stream cipher if you have access to the encryption algorithm output but what if you don’t? Or what if the key is not reused? You can still take advantage of it because the algorithm output is malleable which means we don’t need to really care about getting the key to modify a part of the output plain text if you know the original plain text and if you know where it starts.

How to modify parts of the plain text

As part of the setup for the challenge, we had to modify parts of the plain text JSON to point to letter.txt instead of ./docs/eric_ddfcd1c55ea4.md.

Knowing that a stream cipher is usually done by the XOR operation, we can take advantage of this to replace the current text with the text that we want.

The ciphertext (the base64 encoded bytes) is usually generated like this

 ciphertext = []
 for i in range(0, len(plaintext)):
	 ciphertext[i] = plaintext[i] ^ key[i % len(key)]

But we can take advantage that the XOR operator can cancel out the plain text if you know what it is and replace with with what you know.

In the end you want your new ciphertext to be newciphertext=wantedtext⊕keynewciphertext = wantedtext \oplus key so that when it gets decrypted, your wanted text gets decrypted instead of the current plaint text.

We can do a couple of math tricks to do that:

ciphertext=plaintext⊕keyciphertext = plaintext \oplus key ciphertext⊕plaintext=plaintext⊕plaintext⊕keyciphertext \oplus plaintext = plaintext \oplus plaintext \oplus key # something XOR with itself gives 0 ciphertext⊕plaintext=0⊕keyciphertext \oplus plaintext = 0 \oplus key # XOR with 0 gives you the same thing ciphertext⊕plaintext⊕wantedtext=wantedtext⊕keyciphertext \oplus plaintext \oplus wantedtext = wantedtext \oplus key

So if you XOR the cipher text with the current plain text character with the wanted character in that place, your decrypted ciphertext will containt the wanted plain text

So we can do exactly that to get our wanted filename into this base64 string.

from urllib.parse import quote, unquote
import base64

encoded = "http://docgen.ctf/loadLetter?d=ITI8A2KqR8UciV9IUPeCQRow4hFK2FMjqrNVTe4AH4RSd3A4gFIHlgquMEWHtwYFvMeC%2FzxMq1vw2N7APaovXIWWyV54LHXURRsAnzqCQljv"
url_prefix = "http://docgen.ctf/loadLetter?d="
encoded = encoded.removeprefix(url_prefix)
encoded_bytes = bytearray(base64.b64decode(unquote(encoded)))

starting_filename_offset = 30
current_plaintext = "./docs/eric_ddfcd1c55ea4.md"
wanted_plaintext  = "./letter.txt               "

for i in range(0, len(current_plaintext)):
    encoded_bytes[i + starting_filename_offset] = encoded_bytes[i + starting_filename_offset] ^ ord(current_plaintext[i]) ^ ord(wanted_plaintext[i])

wanted_url = url_prefix + quote(base64.encodebytes(encoded_bytes))

print(wanted_url)

The keen among might have noticed that well, we can replace part of the path that we want but some of it is replaced with spaces, and if we try this URL on the website will get a:

Exception: FileNotFoundException
Reason: the file ./letter.txt could not be found

But, if you noticed that this was JSON and that if you modified the byte after the .md you will see that the JSON parser is expecting a closing ". Which means we can close it early and pad the rest of the JSON with spaces.

from urllib.parse import quote, unquote
import base64

encoded = "http://docgen.ctf/loadLetter?d=ITI8A2KqR8UciV9IUPeCQRow4hFK2FMjqrNVTe4AH4RSd3A4gFIHlgquMEWHtwYFvMeC%2FzxMq1vw2N7APaovXIWWyV54LHXURRsAnzqCQljv"
url_prefix = "http://docgen.ctf/loadLetter?d="
encoded = encoded.removeprefix(url_prefix)
encoded_bytes = bytearray(base64.b64decode(unquote(encoded)))

starting_filename_offset = 30
current_plaintext = "./docs/eric_ddfcd1c55ea4.md\""
wanted_plaintext  = "./letter.txt\"               "

for i in range(0, len(current_plaintext)):
    encoded_bytes[i + starting_filename_offset] = encoded_bytes[i + starting_filename_offset] ^ ord(current_plaintext[i]) ^ ord(wanted_plaintext[i])

wanted_url = url_prefix + quote(base64.encodebytes(encoded_bytes))

print(wanted_url)

And this gives the correct flag!

If you notice that the key is reused

The keen among you may have noticed that the starting URLs all have the same starting base64 encoded characters (I definitely didn’t during the CTF). But if you know that, you can probably guess that the key that is used to encrypt one ciphertext is the same one used to encrypt others. Which means that you can do a little bit more math (and poking around) to figure out entire plain text instead of just the file name and even the entire key.

The math is pretty simple if you understood the previous part.

ciphertext1=plaintext1⊕key1ciphertext_1 = plaintext_1 \oplus key_1 ciphertext2=plaintext2⊕key1ciphertext_2 = plaintext_2 \oplus key_1

If you XOR them together you can get the XOR between the two plain texts.

ciphertext1⊕ciphertext2ciphertext_1 \oplus ciphertext_2 plaintext1⊕key1⊕plaintext2⊕key1plaintext_1 \oplus key_1 \oplus plaintext_2 \oplus key_1 plaintext1⊕plaintext2plaintext_1 \oplus plaintext_2

With this XOR of the two plain texts, you can try guess (or in our case inputting) part of the first plain text to figure out what is the other plain text since XORing with one of them removes it.

plaintext1⊕plaintext2⊕plaintext2≡plaintext1plaintext_1 \oplus plaintext_2 \oplus plaintext_2 \equiv plaintext_1

You can even turn in into a very fun wordle game! (Fully vibe coded but quite fun)

However, this only works if the plain texts are different. If they are the same plain text then the result of the XOR will just be 0 (which means you get no information about what the other key could be).

What you could do to prevent it

1. Not use stream ciphers as an encryption scheme when you don’t need it to be fast

By its nature, stream ciphers are malleable which means that you modify a single bit in the input to modify a single bit in the output. You could use AES-GCM instead which is not malleable (as long as you don’t reuse the same key).

2. Never reuse the same key

Stream ciphers are only cryptographically (semi) secure because they don’t reuse the same key to encrypt the message. If you reuse the key, you can decrypt messages pretty easily by knowing part of the plain text of a message.

3. Use a database for storing the information and not files (prevents path traversal attacks)

if you really want to use a stream cipher for this website, please use it to decrypt something that doesn’t compromise your server like paths to a file. You could just store the data that you want in a SQLite db directly and only store the name of the file. If your encryption gets compromised, the user can only get some data that they could already get. At a user that only has access to the tables that you need and you’re good. But please don’t add SQL injection as another vulnerability when you do this.

4. Use a static site generator (You don’t need all that for a blog)

The best way to secure your server, is to simplify it as much as possible and leave no secrets laying around. If you want a blog (like this one), you can use a static site generator and not have a server at all! This means that the files are directly sent to the user and there is no server to compromise.

5. Restrict read access of the server in the user to only the files that it needs

This must be a given, but almost everyone including myself makes this mistake. We give access to more than we need to users of our server. We could have given the user of this server file access to only a single folder instead of the entire file system.

6. Don’t return error messages (they serve as an oracle to attackers)

When you return error messages to the user, they are great for debugging but they also give more information than necessary to attackers. If you can, just keep your error logs on the server with a message ID to see which error it was when a user reports it.