Reverse Engineering of Xbox Security Method 3

One day I saw a post by Brandon Wilson about Xbox 360 Controller Security and I was curious about how it actually works, so I decided to figure it out myself.

This was my first experience working with the Xbox 360, and I found the Xbox 360 to be much simpler from a software and security perspective than the PS3. Just a few modules, a kernel (xboxkrnl.exe) and a 256kb hypervisor which is mainly used to manage the kv (key vault).

The PS3 has about 400 modules (which share a lot of the same and unused code), a large hypervisor, kernel, and SPU modules.

Microsoft ships XDKs (Xbox Development Kits) with the root encryption key nulled, so the updates provided for it are not encrypted and contain debugging symbols, even for the kernel. And the kernel itself looks like a library and can be debugged on XDK? (Not sure about that, never owned one)

Therefore, finding the XSM3 algorithm was a matter of seconds.

After the Xbox 360 gets all the descriptors from the controller, and the interface descriptor string “Xbox Security Method 3, Version 1.00, 2005 Microsoft Corporation. All rights reserved.” it starts to send auth packets.

Here are my sniffed packets:

UsbdSecXSM3GetIdentificationProtocolData

49 4B 00 00 17 04 E1 11 54 15 ED 88 55 21 01 33 00 00 80 02 5E 04 8E 02 03 00 01 01 C1

UsbdSecXSM3SetChallengeProtocolData

09 40 00 00 1C DE EB 91 87 66 B0 E3 C0 B2 6C 05 6D C8 67 E2 E7 D6 A5 DC 71 6F 21 1F B4 32 28 A0 C2 89

UsbdSecXSM3GetResponseChallengeProtocolData

49 4C 00 00 28 7D 66 8F 48 76 EC EB E9 61 B7 81 82 08 84 B2 7E 73 1F 7A 92 47 83 F8 5E 78 22 11 B9 9B FC AD D6 F4 D9 1E 04 7E C1 C0 0B 9A

UsbdSecXSM3SetVerifyProtocolData1

09 41 00 00 10 60 17 AC 73 E5 0F 10 D4 10 F5 C1 B5 8F 63 56 9B 36

UsbdSecXSM3GetResponseVerifyProtocolData1

49 4C 00 00 10 60 7E 0C 62 AE 46 94 E7 14 BA 3D 70 33 7E 93 DF 09

UsbdSecXSM3SetVerifyProtocolData2

09 41 00 00 10 47 A2 4C A3 97 DF AC CC 29 48 F1 D0 08 11 D1 FD 57

UsbdSecXSM3GetResponseVerifyProtocolData2

49 4C 00 00 10 49 3A E1 F9 8C 91 73 C3 FA A2 07 D3 CC 1E 2F 17 A0

Data format:

• [5 bytes - cmd header]
• [data]
• [1 byte - chksum (xor of data bytes)]

UsbdSecXSM3GetIdentificationProtocolData contains Serial, Category, VendorID and other data.

Category and VendorID are important bytes. Each type of device, such as wireless controller, wired controller, webcam, memory unit, etc. has its own category. VendorID can be of 2 types: Microsoft / 3rd party.

Depending on these bytes, the encryption key is selected.

The full reverse engineered algorithm along with my sniffed USB traffic can be found on my github.

The XSM3 algorithm is quite big, obfuscated, and unfortunately contains custom crypto algorithms. I tried to cheat: checked PC drivers, Xbox SDK, windows debug symbols in hopes that Microsoft used it somewhere else, but none of this worked out, so in the end I had to rewrite this algorithm from PowerPC asm to C.

I did this research back in 2013… who could have guessed then that Hex-Rays would release a PowerPC decompiler ???

The problem is that the Xbox 360 encrypts the hardware serial number with a fixed key and sends it to the USB device. Xbox then uses per-console key from the kv (key vault) for crypto.

The per-console keys are pre-calculated based on the hardware serial number at the factory, and the secret portion to convert the serial number to a console key is placed on authorized devices.

Okay, so what now?

Xbox controller, Skylanders Portal etc. use Infineon TPM chip. But what about unauthorized devices? There are so many of them! Devices like Cronus and XIM require a wired controller to bypass authentication. I looked at some Chinese controllers, but they use real MS chips. It seems Datel were the only ones who really cracked it and were selling their own controllers. In one of them I found 2 MCU’s. One of them is the SiTel SC14450, a DECT phone controller, probably used for the radio interface. The other is an unknown 16-pin microcontroller labeled ‘Raw Science’ (this is a Datel company).

The DFU firmware for the SC14450 can be found in the updater, it is a 16-bit CompactRISC CR16C microcontroller, but the firmware does not contain anything interesting, all commands are sent directly to the Datel ‘Raw Science’ chip.

The Datel chip was decapped. Looks like a smart card or SIM card chip, 6 wires, ROM, flash memory and RAM. But I couldn’t recognize this chip. Wild guess was that MCT = MicroChip Technology, but it is not PIC.

Masked ROM is small and easily recognizable, so getting a bootrom dump was not difficult. It is based on the 8051 architecture, but there is nothing interesting about it or anything that would help us get the secret part.

So, at this point, the Xbox Security Method 3 is reverse engineered, but the secret algorithm for converting the serial number into a per-console key is still a secret.

If you liked this post, you can share it with your followers or follow me on Twitter !